On Windows, core.trustFolderStat=false turns every missing loose object into AccessDeniedException

[ddemidov-issart]

Version

Reproduced against 7.6.0.202603022253-r; observed in production on 7.7.0. Introduced by fed1a54
"Refresh 'objects' dir and retry if a loose object is not found" (11 Jan 2023), first released in
6.5.0. Still present on master and stable-7.7.

Operating System

Windows. (The reproduction below also runs on Linux/Unix and MacOS — see the note in it.)

Bug description

With core.trustFolderStat=false, a lookup for an object that is not present in the repository
fails with java.nio.file.AccessDeniedException on Windows instead of reporting the object as
absent. The exception names the repository's objects directory, not any file.

The cause is the NFS attribute-cache workaround in LooseObjects: on the not-found path it opens
the objects directory as a byte stream to force the client to re-read directory attributes.
Opening a directory that way succeeds on Linux and fails on Windows, so on Windows the workaround
can never complete — it always throws, and the throw escapes to the caller.

We are not proposing a fix; we do not know this codebase well enough to judge the right one. What
follows is what we observed and where.

Where. org.eclipse.jgit.internal.storage.file.LooseObjects#getObjectLoader (current master):

ObjectLoader getObjectLoader(WindowCursor curs, File path, AnyObjectId id)
		throws IOException {
	try {
		return getObjectLoaderWithoutRefresh(curs, path, id);
	} catch (FileNotFoundException e) {
		if (trustLooseObjectStat == TrustStat.ALWAYS) {
			throw e;
		}
		try (InputStream stream = Files
				.newInputStream(directory.toPath())) {      // <-- throws on Windows
			// refresh directory to work around NFS caching issues
		}
		return getObjectLoaderWithoutRefresh(curs, path, id);
	}
}

LooseObjects#open reaches it because, under TrustStat.NEVER, the path.exists() short-circuit
that TrustStat.ALWAYS takes is deliberately skipped:

switch (trustLooseObjectStat) {
case NEVER:
	break;                          // no exists() check: always attempt the read
...
case ALWAYS:
	if (!path.exists()) {
		reload = false;             // absent -> return null, no directory open
	}
	break;
}

and its catch (IOException e) only retries for stale NFS file handles, so an
AccessDeniedException is rethrown:

} catch (IOException e) {
	if (!FileUtils.isStaleFileHandleInCausalChain(e)) {
		throw e;
	}
	...
}

The same workaround appears three times in the file, guarded once.

Call site	Guarded?
has() — line 115	yes: catch (IOException e) { return false; }
getObjectLoader() — line 238	no
getSize() — line 275	no: the enclosing catch takes FileNotFoundException only

We mention this as an observation about the shape of the defect, not as a suggested remedy.

Platform behaviour. Files.newInputStream on a directory succeeds on Linux and fails on
Windows. Measured on Linux:

java.version      = 17.0.19
os.name           = Linux
Files.newInputStream(dir) -> OK, available=4096

On Windows the JDK opens the path through CreateFile without FILE_FLAG_BACKUP_SEMANTICS, and
Windows answers ERROR_ACCESS_DENIED for any directory — the observed frame is
sun.nio.fs.WindowsFileSystemProvider.newByteChannel. It is not a share, protocol or ACL matter;
we see it for UNC paths and expect the same for a local drive.

Why a single failed lookup aborts a whole fetch. FetchProcess#askForIsComplete treats a
missing object as a normal outcome and an IOException as fatal:

} catch (MissingObjectException e) {
	return false;                   // expected: fetch not complete, ask the remote for more
} catch (IOException e) {
	throw new TransportException(JGitText.get().unableToCheckConnectivity, e);
}

AccessDeniedException is an IOException, so the routine "not here yet" answer becomes a hard
TransportException: Unable to check connectivity and the fetch is abandoned.

Steps to reproduce. Runnable on Linux or MacOS against a stock JGit release; no Windows machine
needed. On Windows the chmod step is unnecessary, because opening any directory already fails
there — that step only makes the identical code path observable on POSIX.

import java.io.File;
import java.nio.file.*;
import java.nio.file.attribute.PosixFilePermissions;
import org.eclipse.jgit.lib.*;
import org.eclipse.jgit.storage.file.FileRepositoryBuilder;

public class Repro {

    static void attempt(File gitDir, boolean trustFolderStat) throws Exception {
        try (Repository repo = new FileRepositoryBuilder().setGitDir(gitDir).build()) {
            StoredConfig cfg = repo.getConfig();
            cfg.setBoolean("core", null, "trustFolderStat", trustFolderStat);
            cfg.save();
        }
        try (Repository repo = new FileRepositoryBuilder().setGitDir(gitDir).build();
             ObjectReader reader = repo.newObjectReader()) {
            ObjectId missing = ObjectId
                    .fromString("0123456789abcdef0123456789abcdef01234567");
            try {
                reader.open(missing);
                System.out.println("trustFolderStat=" + trustFolderStat + " -> opened (unexpected)");
            } catch (Exception e) {
                System.out.println("trustFolderStat=" + trustFolderStat + " -> "
                        + e.getClass().getName() + ": " + e.getMessage());
            }
        }
    }

    public static void main(String[] args) throws Exception {
        Path tmp = Files.createTempDirectory("jgit-repro");
        File gitDir = tmp.resolve("repo.git").toFile();
        try (Repository repo = new FileRepositoryBuilder().setGitDir(gitDir).setBare().build()) {
            repo.create(true);
        }
        Path objects = gitDir.toPath().resolve("objects");

        System.out.println("--- objects/ readable (baseline) ---");
        attempt(gitDir, false);
        attempt(gitDir, true);

        System.out.println("--- objects/ not openable ---");
        Files.setPosixFilePermissions(objects, PosixFilePermissions.fromString("---------"));
        try {
            attempt(gitDir, false);
            attempt(gitDir, true);
        } finally {
            Files.setPosixFilePermissions(objects, PosixFilePermissions.fromString("rwxr-xr-x"));
        }
    }
}

Actual behavior

The lookup throws java.nio.file.AccessDeniedException naming the objects directory. When the
lookup happens inside FetchProcess#askForIsComplete, the fetch is abandoned with
TransportException: Unable to check connectivity.

Expected behavior

The lookup reports the object as absent — which is what the same code does with
core.trustFolderStat=true, and what it does on Linux with either value. A caller that handles
MissingObjectException as a normal outcome, as FetchProcess#askForIsComplete does, should
continue rather than fail.

Relevant log output

Reproduction output — JGit 7.6.0.202603022253-r, OpenJDK 17.0.19, Linux:

--- objects/ readable (baseline) ---
trustFolderStat=false -> org.eclipse.jgit.errors.MissingObjectException: Missing unknown 0123456789abcdef0123456789abcdef01234567
trustFolderStat=true  -> org.eclipse.jgit.errors.MissingObjectException: Missing unknown 0123456789abcdef0123456789abcdef01234567

--- objects/ not openable ---
trustFolderStat=false -> java.nio.file.AccessDeniedException: /tmp/jgit-repro.../repo.git/objects
trustFolderStat=true  -> org.eclipse.jgit.errors.MissingObjectException: Missing unknown 0123456789abcdef0123456789abcdef01234567

Frames of the reproduced failure:

at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoader(LooseObjects.java:238)
at org.eclipse.jgit.internal.storage.file.LooseObjects.open(LooseObjects.java:193)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseObject(ObjectDirectory.java:428)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseFromSelfOrAlternate(ObjectDirectory.java:404)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:379)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:359)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:146)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:216)

Production, Windows Data Center, JGit 7.7.0 — identical frames:

Caused by: java.nio.file.AccessDeniedException: \\<server>\<share>\...\<repository>\objects
	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:234)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
	at java.base/java.nio.file.Files.newInputStream(Files.java:160)
	at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoader(LooseObjects.java:238)
	at org.eclipse.jgit.internal.storage.file.LooseObjects.open(LooseObjects.java:193)
	...
	at org.eclipse.jgit.transport.FetchProcess.askForIsComplete(FetchProcess.java:419)

Other information

Git Integration for Jira is a Jira Data Center app; it embeds JGit and keeps bare repositories on
the Jira shared home. On one Windows Data Center instance whose shared home is a UNC path, a single
node produced 432 of these over about 70 minutes across roughly 450 repositories, and every one of
them aborted that repository's fetch. Setting core.trustFolderStat=true on those repositories
stopped it; indexing has since run across 3,786 repositories for over 24 hours without a recurrence.

Related: #288 — LooseObjects#tryMove fails with AccessDeniedException on Windows when several
threads insert the same object. Same file, same exception type, same platform, but the insert path
and a concurrency race; this one is on the read path and needs no concurrency.

Posted by Claude