Repository navigation
[rush] Delete stale build-cache-restore staging folders left by interrupted restores - #6213
Conversation
An interrupted build cache restore (Ctrl+C, SIGKILL) left its staging folder in the project's .rush/temp folder, because the folder was only deleted in a finally block. Name each staging folder after the process that made it, and before staging a restore, delete the folders of processes that are no longer running. Fixes microsoft#6145
|
@microsoft-github-policy-service agree |
|
Ssup! this PR fixes #6145. Interrupted restores leave build-cache-restore-* staging folders in .rush/temp. The fix deletes staging folders whose owning process has exited, checked with process.kill(pid, 0), and never touches the current process’s folders. The description covers the design choices and the alternatives I considered. Would either of you have time to take a look? I’m most interested in whether the PID-liveness check is acceptable, or whether you’d prefer a different approach. |
|
Sean Larkin (@TheLarkInn) Is there any slack or discord or any platform where we can discuss & ask doubts. Your guidance will help me a lot |
Summary
Fixes #6145.
When Rush restores an operation from a local build cache entry with tar, it extracts the entry into
<project>/.rush/temp/build-cache-restore-XXXXXX/and then moves the output folders into place. If the process was interrupted during the restore (Ctrl+C on a slowrush build/rush start, or SIGKILL), that staging folder stayed behind indefinitely with a copy of the outputs. Tools that scan the project folder (for example a lint glob) then picked up the stale files.With this change, the next restore of that project deletes the staging folders that interrupted restores left behind.
Details
Root cause: in
libraries/rush-lib/src/logic/buildCache/StagedRestore.tsthe staging folder was deleted only in afinallyblock. Afinallyblock does not run when the process is killed or exits on a signal, and nothing removed such folders later.Fix:
build-cache-restore-<process id>-XXXXXX.deleteStaleStagingFoldersAsyncdeletes thebuild-cache-restore-<pid>-*folders in the project's.rush/tempfolder whose process is no longer running. It checks withprocess.kill(pid, 0): onlyESRCHcounts as not running, andEPERMmeans the process exists.Not handled:
build-cache-restore-XXXXXX, with no process id) are kept, because there's no safe way to tell whether a running older Rush process is using them. They can be deleted by hand once, as before.Alternatives considered: staging under the common temp folder, as the issue suggests, would need a folder on the same volume as every project so that the renames still work. That is a larger change, and it would still leave stale folders, just in a different place.
No backwards-compatibility impact, and the cost is one
readdirof the project's.rush/tempfolder per staged restore.How it was tested
New test
U8insrc/logic/buildCache/test/StagedRestore.test.ts. It sets up three staging folders: one from a process that has exited, one from a running process, and one with no process id. After a restore, only the folder of the exited process is deleted, and the new staging folder's name has the current process id. The test fails without the fix and passes with it.Commands run:
Follow-up issues created: none.