Repository navigation
--append/--inplace: an interrupted transfer can write the same 256 KiB twice #1128
Description
If rsync is stopped while it is writing an --append or --inplace file, the receiver can write the last 256 KiB it received a second time. With --append the damage is silent and permanent: the next run brings the file to the right size, but one 256 KiB region holds a copy of the 256 KiB before it.
It needs the stop to arrive while the receiver is blocked in a write, so slow destinations (FUSE or network filesystems, busy disks) are hit far more often than fast local disks. We ran into it with rsync -a --append syncing growing files onto a JuiceFS (FUSE) mount from a systemd service that is stopped on a schedule; 13 files were damaged this way.
- Affected: every release since 3.1.0. Tested on 3.4.1, 3.5.1 and master at 7998ccf.
- Fix: cleanup: don't flush the write buffer from a signal handler #1129, with a regression test.
- Workarounds: stop rsync with SIGKILL, which leaves the file ending at the last completed write; or use
--append-verify, which repairs the damage on the next transfer of the file.
What goes wrong
- The receiver collects incoming file data in a 256 KiB buffer and writes it out with
flush_write_file(). - A stop reaches the receiver as SIGUSR1, sent by the generator's
_exit_cleanup().sigusr1_handler()runs the exit cleanup inside the handler. - Since 3.1.0 that cleanup flushes the write buffer of an
--inplace/--appendfile, to keep as much received data as possible. - A signal that arrives during a write is handled when the write returns. At that point the bytes are already in the file, but
flush_write_file()has not yet subtracted them from its count. The cleanup's flush therefore sees a full buffer and writes the same 256 KiB again, at the advanced file offset.
With systemctl stop (SIGTERM to every process), the reproduction below damaged the file in every run here. Killing only the client hits the same path when writes are slower still: with 1 s writes it happened in every run, with 200 ms writes in none. The client first sleeps 400 ms in sig_int(), and with faster writes the receiver has drained its input and is no longer writing when the SIGUSR1 arrives.
How to reproduce
The script makes every large write() to a regular file take 200 ms, with signals held until it returns, which is how the kernel treats a signal during a slow FUSE or NFS write. It then stops a running rsync -a --append the way systemd does. It needs cc and GNU coreutils.
repro.sh
#!/usr/bin/env bash
# Interrupt `rsync -a --append` while the receiver is blocked in a slow write,
# then check whether the destination is still a clean prefix of the source.
# Usage: ./repro.sh [path-to-rsync]
set -eu
RSYNC=$(realpath "$(command -v "${1:-rsync}")")
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
cd "$work"
# Every large write() to a regular file takes 200 ms, and signals stay pending
# until it returns: what the kernel does for a signal that arrives during a
# slow write it doesn't abort (FUSE, NFS, a busy disk).
cat > slowwrite.c <<'EOF'
#define _GNU_SOURCE
#include <dlfcn.h>
#include <signal.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
ssize_t write(int fd, const void *buf, size_t count)
{
static ssize_t (*real_write)(int, const void *, size_t);
struct timespec stall = { 0, 200 * 1000 * 1000 };
struct stat st;
sigset_t all, old;
ssize_t ret;
if (!real_write)
real_write = (ssize_t (*)(int, const void *, size_t))dlsym(RTLD_NEXT, "write");
if (count < 65536 || fstat(fd, &st) != 0 || !S_ISREG(st.st_mode))
return real_write(fd, buf, count);
sigfillset(&all);
sigprocmask(SIG_BLOCK, &all, &old);
ret = real_write(fd, buf, count);
nanosleep(&stall, NULL);
sigprocmask(SIG_SETMASK, &old, NULL);
return ret;
}
EOF
cc -shared -fPIC -o slowwrite.so slowwrite.c -ldl
mkdir src dst
head -c 32M /dev/urandom > src/big
head -c 1M src/big > dst/big # a growing file that is already partly synced
# Like `systemctl stop`: SIGTERM to every rsync process at once, mid-transfer.
LD_PRELOAD=$PWD/slowwrite.so setsid "$RSYNC" -a --append src/ dst/ &
pgid=$!
sleep 1.5
kill -TERM -- "-$pgid"
wait "$pgid" || true
while kill -0 -- "-$pgid" 2> /dev/null; do sleep 0.1; done
size=$(stat -c %s dst/big)
if cmp -s -n "$size" src/big dst/big; then
echo "OK: after the interrupted run dst/big is a clean prefix of src/big ($size bytes)"
else
echo "BROKEN: after the interrupted run dst/big ($size bytes) is not a prefix of src/big"
cmp -n "$size" src/big dst/big || true
if cmp -s <(tail -c 262144 dst/big) <(head -c $((size - 262144)) src/big | tail -c 262144); then
echo " its last 256 KiB are a copy of the 256 KiB before them"
fi
fi
"$RSYNC" -a --append src/ dst/
if cmp -s src/big dst/big; then
echo "OK: identical after a normal --append run"
else
echo "BROKEN: still different after a normal --append run, sizes $(stat -c %s src/big) and $(stat -c %s dst/big)"
cmp src/big dst/big || true
fiOutput with master (7998ccf), the same on 3.4.1 and 3.5.1:
BROKEN: after the interrupted run dst/big (3407872 bytes) is not a prefix of src/big
src/big dst/big differ: byte 3145729, line 12062
its last 256 KiB are a copy of the 256 KiB before them
BROKEN: still different after a normal --append run, sizes 33554432 and 33554432
src/big dst/big differ: byte 3145729, line 12062
With --inplace --no-whole-file the interrupted run is damaged the same way, but the next delta transfer repairs it.
Activity