Directory downloads should retry into a fresh staging directory #4805

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 21, 2026
2. O96a commented on Oct 21, 2026

   

   O96a

   on Oct 21, 2026

   More actions

   The directory path needs the same staging treatment as the file path, but the staging dir has to be created in the destination's parent, not in TempDir — rename is only atomic within a filesystem, and /tmp will hand you EXDEV the first time the destination is on a mounted volume.

   The swap is where it actually gets fiddly, since rename(staging, dest) fails with ENOTEMPTY when dest already exists:

   // staging = dest.parent().join(format!(".{}.staging.{}", name, attempt))
   extract_tar(archive, &staging)?;          // full extract, no partial dest
   if dest.exists() {
       fs::rename(&dest, &dest_old)?;        // dest -> dest.old
   }
   fs::rename(&staging, &dest)?;             // staging -> dest
   let _ = fs::remove_dir_all(&dest_old);    // best effort

   Two things that bite if you skip them: the staging dir has to be per-attempt and removed on the failure path, otherwise the retry inherits the previous attempt's partial tree and you've just relocated the bug rather than fixed it. And on startup, sweep any leftover dest.old — if the process died between the two renames you'll otherwise leave a stale tree sitting next to the destination forever.

   Separately, I'd narrow what retry_sandbox_sync considers retryable. A tar extraction failure is usually a truncated stream and worth another attempt; a checksum mismatch or a path-traversal rejection is deterministic and retrying it just burns the deadline.
3. Aliciapet11 commented on Oct 22, 2026

   

   Aliciapet11

   on Oct 22, 2026

   Author

   More actions

   Thanks - agreed. To clarify, the intent is tempfile::TempDir::new_in(destination_parent), matching the file path, so staging and final destination remain on the same filesystem and avoid EXDEV.
   Per-attempt staging plus failure cleanup is also required, and narrowing retryable errors makes sense so deterministic checksum/path failures don’t consume the retry budget.
   Fullsend currently works around this by staging each outer attempt in the destination parent; once OpenShell handles directory retries internally, we can remove that outer retry.