Conversation
LogBucket.getLogBucketRange raised KeyError for any value at or above SHINGLER_LOGBUCKETS, which aborts the whole indexing job. The table covers 0..max_value-1 and FuzzyStatPairShingler buckets max_block_size, num_ins_C, num_ins_S and num_calls through it without bounding any of them - only stack_size is clamped, at its own call site. A single basic block of 108,837 bytes was enough to make a corpus unindexable, and the odds of containing one rise with corpus size. Values outside the table are clamped to its bounds now; nothing inside it moves, so no MinHash that could be computed before changes value. Worker.updateMinHashes raised UnboundLocalError when there was nothing left to hash, because minhashes was only ever bound inside the batch loop. Finishing with an empty backlog therefore failed exactly like a crash, and that is the normal state of a resumed index. The same statement also returned the size of the last batch rather than the total, so any run longer than one workpack under-reported - a 238,991-function backlog across 24 batches reported whatever the final batch held. Every caller reads it as a total, so it accumulates now. Both come with tests that fail against the unfixed code.
Every stage of a 1-vs-N query grows with corpus size, and so does the answer - a result naming thousands of matched samples is not something anybody reads, and bounding the answer turns out to be the only thing that bounds the work. MINHASH_MATCHING_SHORTLIST_SIZE ranks candidate samples cheaply first (one vote per distinct query function, plus weighted PicHash evidence, ranked by vote count and by coverage so a small sample matching entirely is not buried under a large one matching partly) and then matches only those exactly. STORAGE_BAND_DF_CUTOFF skips band hashes whose posting list is longer than the cutoff, decided from a stored df and a (band_hash, df) index so an over-long list is never read rather than read and discarded. MINHASH_PICHASH_MAX_MATCHES bounds the PicHash path the same way, and pairwise scoring now compares each distinct MinHash signature once instead of once per function holding it, which is exact rather than approximate. Measured on four real Malpedia corpus sizes - 2,016 / 2,997 / 5,236 / 7,244 samples, same three query samples throughout, three repeats each - the baseline's median grows as corpus^1.40 while two-stage is corpus^-0.07. At the largest size that is 10.20s down to 1.58s on the median, 25.39s down to 1.93s on the tail, and 766MB down to 253MB peak RSS, with top-10 sample recall still 1.000 and 0.985 of surviving function matches keeping a bit-identical score. All three knobs default to 0, so an instance behaves exactly as before until it opts in.
A posting list lives as a function_ids array inside one document, and MongoDB caps a document at 16 MB. On a 7,244-sample corpus the largest band document held 18,968 postings in 197,606 bytes, about 10.4 bytes each, so roughly 1.6M fit - 84.9x that corpus, which puts the wall somewhere near 615,000 samples. Past it the $push raises "BSONObj size ... is invalid" and indexing stops for any sample holding a function whose band hash is already at the cap. It fails rather than slowing down, and sharding does not help because a document cannot span shards. STORAGE_BAND_BUCKET_SIZE splits a hash across (band_hash, bucket) documents once it would exceed the cap, and defaults to 0, which keeps the current single-document shape byte for byte. Bucket 0 carries the bookkeeping for the whole hash - df as the total across every bucket, plus tail and tail_n for placement - which is what leaves the cutoff filter and its (band_hash, df) index untouched: a hash under the cutoff is nowhere near a bucket's worth so it never spills, and one that did spill has a df that rejects it anyway. Buckets fill in order rather than by hashing the function id, so a short posting list stays in one document instead of scattering across many. Enabling it on an existing database needs rebuild_band_df_index first - documents written earlier have no bucket field, so the upsert would miss them and insert a second document per hash, which nothing would report as an error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #195; only the commits after it are this PR's, and the diff will collapse to them once that one merges.
fork PR with CI history: r0ny123#45
A band posting list lives as a
function_idsarray inside one document, and MongoDB caps a document at 16MB, which puts a hard ceiling on corpus size that I don't think anyone has hit yet but is closer than it looks. On a 7,244-sample corpus the largestband_0document held 18,968 postings in 197,606 bytes — about 10.4 bytes each, so roughly 1.6M fit, which is 84.9x that corpus and puts the wall somewhere near 615,000 samples. I checked rather than extrapolating: pushing 100,000 ids at a time into one document succeeded ten times and failed on the eleventh withBSONObj size: 17588958 is invalid. It fails, it doesn't slow down, so indexing stops outright for any sample holding a function whose band hash is already at the cap. Sharding doesn't help either, since a document can't span shards.STORAGE_BAND_BUCKET_SIZEsplits a hash across(band_hash, bucket)documents once it would exceed the cap, and defaults to0, which keeps the current single-document shape byte for byte. Bucket 0 carries the bookkeeping for the whole hash —dfas the total across every bucket, plustail/tail_nfor placement — which is what leaves the cutoff filter and its(band_hash, df)index untouched: a hash under the cutoff is nowhere near a bucket's worth so it never spills, and one that did spill has adfthat rejects it anyway. Buckets fill in order rather than by hashing the function id, deliberately, since modulo placement would scatter a 200-entry posting list across 200 documents and make the common cheap case expensive. One thing to watch on an existing database: enabling this needsrebuild_band_df_indexrun first, because documents written earlier have nobucketfield and the upsert filter would miss them and insert a second document per hash, splitting the list with nothing reporting an error — the rebuild stampsbucket: 0and is what makes them addressable. The estimate above is also on the optimistic side, since Malpedia is curated and deduplicated; a corpus full of near-duplicate packed variants would concentratedffaster and get there sooner. Stacked on #195.