[FR] Unlimited branch support for code search

[brendan-kellam]

Currently, zoekt is limited to 64 branches. This issue tracks making this unlimited so you can search on arbitrary branches, tags, commit SHAs etc.

[KonradStanski]

Is this branches per repo, or branches total?

[brendan-kellam]

This is branches per repo

[Nil2000]

@brendan-kellam Can we do it batch wise may be process at a time 64 branches/tags and do the same until we cover all?

[msukkari]

@Nil2000 this is a limitation with zoekt. Last I looked into this it uses a 64 bit vector flag to track branches which is where the limitation comes from

[abraverm]

sourcegraph/zoekt#1002

[brendan-kellam]

@abraverm this is awesome, thanks for taking the time to implement this into zoekt. I'll take a closer look at the PR soon, although I'll be allot less helpful than the Sourcegraph folks.

Just a FYI: in a ideal world, in addition to searching any branch, you should be able to search any commit in Sourcebot too. In that way, if you are browsing a arbitrary commit (#655), you can then can then search against that commit. We will need to do some benchmarking, but my worry is that the cost of maintaining a index of "all revisions across all repositories" will be too expensive, especially if the repositories have significant history. For example, in UnrealEngine:

$ git rev-list --all | wc -l
460828

My guess is that the p95 of searches are on the default branch[1] anyways. Instead, I wonder if we only index the default branch using zoekt, and then use git grep when searching arbitrary revisions. The tradeoff here is that we don't need to maintain a index for all revisions, at the cost of (likely) slower search performance w/ git grep. IMO this feels like a acceptable tradeoff.

I'll note that your change in zoekt is still valuable since there may be cases where the tradeoff is not acceptable for certain branches that are frequently searched (e.g., release/ branches), so continuing to have the ability to specify specific revisions in the config to be indexed by zoekt makes sense in those cases.

[1] this is just a guess. We don't have any data on this unfortunately - #781 changes that, so will check again in a week or two.

[abraverm]

I have implemented tag and commit search, I want the branch PR to be merged before submitting these.

The cost is really the output size, and then 1 TB vs 10 TB doesn't matter much. Just throw at it more resources 😆

That said, I had to modify the branch patch to make it work with your zoekt fork. Would it be possible for your fork to be more current?

[brendan-kellam]

Yea we can fast forward our zoekt fork to latest, there shouldn't be anything blocking that, assuming no breaking changes haven't been introduced. Happy to do that after the zoekt branch PR has landed.

I have implemented tag and commit search, I want the branch PR to be merged before submitting these.

Could you elaborate on what you mean by tag & commit search?

[abraverm]

Unlimited tags (like with branch) and searching any commit (files and message).
The later shows the commit hash instead of branch/tag, and it needs to be enabled in query "commits:true" kind of thing. The query also includes filtering by time period (before/after).

[brendan-kellam]

To further clarify, is the idea to allow you to search commit messages and have that appear in /search ?

[abraverm]

The idea is to search both commits files content and their messages.