Repository navigation
feat(gitlab): read opt-in files in batches through GraphQL - #19
Merged
Merged
Conversation
GitLab's reader is now a platform.BatchReader: one GraphQL request per 50 projects, by id (projects(ids:) follows renames), with each project's head commit and the blob at the path (repository.blobs). A blob's mode tells regular files from symlinks, submodules and directories, a size over the limit is ErrTooLarge, and a text blob comes in the answer, checked against its id. A root path that a resolved head lacks is ErrNotFound. Everything else goes to ReadFile, which judges it as before: a project the answer lacks, an empty project, a nested path, a blob stored in LFS, binary, or not hashing to its id. GraphQL takes the token as a bearer token, and a query counts as a read. A rate limit, a refused credential or the end of ctx fails the call; an answer with errors reads its projects file by file, and an instance that turns GraphQL down is read file by file from then on. gitlab.com scored 45 of its 200 for 50 projects (2026-10-08). Tests: unit tests on the fake API, and a conformance case, ReadFiles, that holds every BatchReader to ReadFile's answers across a regular file, an executable, binary content, a file over the limit, a symlink, a submodule, a directory, a missing file and an empty repository; it passes on the GitHub driver against ghfake and runs on live GitLab.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
plananddistributeread every GitLab target's opt-in file one by one, two REST requests each at least. GitLab's reader is now aplatform.BatchReader, as GitHub's is: one GraphQL request per 50 projects.The query.
projects(ids:)by global id, which follows renames, with each project's head commit (repository.tree.lastCommit.sha, which tells that the default branch exists) and the blob at the path (repository.blobs(paths:, first: 1):path mode oid size storedExternally rawTextBlob). The fields were read from gitlab.com's schema, and a request for 50 projects scored 45 of the 200 allowed without a token (250 with one) on 2026-10-08.What the answer settles:
ErrNotRegular.ErrTooLarge.ErrNotFound.What it leaves to
ReadFile, which judges it as before:A missing project or ref is therefore never taken for a missing opt-in file.
Failures.
Tests.
ReadFiles, holds everyBatchReadertoReadFile's answers. It covers a regular file, an executable, binary content, a file over the limit, a symlink, a submodule, a directory, a missing file and an empty repository, plus a path through a symlinked directory. It passes on the GitHub driver against ghfake, and on live GitLab.Checked locally.
go test ./internal/platform/... ./internal/e2e/github/passes. On GitLab CE 18.11.12,scripts/e2e/gitlab.sh --run 'TestConformance|TestDistribute'passes,ReadFilesincluded.