Conversation
…lplohmann#145) _pichash and _picblockhashes[].hash hold 64 bit values as variable-width hex strings, so a range condition on pichash answered a plausible but wrong set and a pichash-sorted function search could not page (the cursor compares the sort field with a range operator). Write the values 16 digits wide and string order is numeric order. - settings flag pichash_padded: set on fresh instances, reported through getStats/getStatus; the storage reads it once per process - width-aware _encodePichash; the transpiler allows range operators and the storage allows sorting by pichash (mapped to _pichash) only when the instance is padded, and rejects both with a message naming the migration otherwise - on an unpadded instance every lookup by value (search conditions, isPicHash, getMatchesForPicHash, getMatchesForPicBlockHash) matches both widths, so a migration in flight misses nothing - mcrit/migrations/migrate_pichash_padding.py: resumable in-place pad/unpad walks over functions and query_functions with a leftover sweep, a verify mode that flags unpadded values under a set flag - tests: transpiler on both widths, fresh instance, legacy lookups, mixed widths, numeric range answers, pichash-sorted paging forward and backward, migration round trip and CLI Refs danielplohmann#145
…hs in the aggregation, bound the range values - migrate_pichash_padding builds its URI from STORAGE_MONGODB_USERNAME, _PASSWORD and _FLAGS like MongoDbStorage does, or takes --uri - getPicHashMatchesByFunctionId(s) match either spelling of a value while the instance is not padded, so equal pichashes in opposite widths land in one group during a migration - a pichash bound outside 0..0xffffffffffffffff is rejected: it would format wider than 16 digits and not compare
|
Live migration of a running MongoDB 7 instance (4 samples, 1158 functions) that was created before this change: Before (legacy shape): Migration: server and worker stopped, After: |
One conflict, in _ensureIndexAndUnknownFamily, where both sides added to the same fresh-database branch. They are independent and both are kept, at the nesting each belongs to: this branch's `pichash_padded: True` goes into the settings document that is inserted when there is no settings collection yet, and main's picblockhash-index-completeness check stays outside that `if`, so it still runs for an existing database as well as a fresh one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011EAW1DRkBwmjZtGzQ5pgDA
…anielplohmann#69 is about Upstream has 17 open PRs and 15 open issues. One of them is this problem: danielplohmann#69 "Workers consume a lot of ram on query" - open, unlabelled, unassigned, no linked PR. Reported against a 20 million function instance, with workers reaching tens of GB and sometimes over 60 GB, one worker starving the others and crashing multi-worker deployments. That is the same defect as the latency problem seen from the memory side, and this project already measured why: docs/TUNING.md records peak RSS correlating 0.98 with bytes fetched from MongoDB and only 0.62 with the sample's function count. Bytes fetched is candidate volume, and candidate volume grows with the corpus. What exists for danielplohmann#69 today is mitigation rather than a fix - MINHASH_MATCHING_MAX_PAIRS caps pairs held at once, its own comment calling the default "a guard against runaway jobs (danielplohmann#69-class)" - and none of it reduces how many candidates a query produces; it spreads the same work over more batches. The shortlist bounds the candidates themselves, so it should bound the peak, which makes this work a candidate answer to danielplohmann#69 and not only to the latency question. That claim needs a number, so the harness now records peak RSS per query alongside the stage timings, noting in the code that ru_maxrss is a process-wide high-water mark rather than the query's own allocation. docs/scaling/UPSTREAM-REVIEW.md records the rest: every other open upstream PR is authored by this fork's owner and pairs with one of the fork's own, and none touch the matching path's scaling behaviour - client surface, job lifecycle, storage schema, metadata, release. It also records the one file with real overlap (MongoDbStorage, where danielplohmann#181 changes how pichashes are stored and this work changes how they are counted - compatible) so the conflict question is answered rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CkCmcpjEzUwYBroCEhrhVo
Closes #145.
_pichashand_picblockhashes[].hashhold 64 bit values as variable-width hex strings (hex()), sopichash:<0x99answered a plausible but wrong set and a pichash-sorted function search could not page: the search cursor compares the sort field with a range operator, which the transpiler rejected for exactly that reason. Written 16 digits wide, string order is numeric order.What changes
settings.pichash_padded. Fresh instances get it at creation; it is reported throughgetStats()and/status. The storage reads it once per process._encodePichash(..., padded=);_encodeFunctionandrecalculateAllPicHasheswrite in the instance's width. Import goes throughFunctionEntryintegers, so it follows the instance too._pichash); an unpadded one rejects both with a message naming the migration, instead of answering a wrong set.isPicHash,getMatchesForPicHashandgetMatchesForPicBlockHashmatch both encodings ($inon the indexed field), so a migration in flight misses nothing. On a padded instance the point lookups are single-value again.python -m mcrit.migrations.migrate_pichash_padding --mode pad|verify|unpad: resumable keyset walk overfunctionsandquery_functionswith positional$setper changed value, a leftover sweep for documents a still-running server wrote below the cursor, then the flag.verifycounts unpadded values and exits 1 when the flag says padded but some remain (a 16-digit value is legitimate under either encoding, so only that direction is a problem).unpadrolls back.Operationally: stop server and workers, run
pad, restart. Range/sort by pichash on an unmigrated instance keep failing the same way as before, just with a clearer message; everything else behaves as it did.Verification
tests/testPichashPadding.py(MongoDB): fresh instance is padded; lookups by value on both shapes; a half-migrated instance misses nothing and still refuses range/sort; range conditions answer the numeric set (pichash:<0x99is empty); a pichash-sorted function search pages forward and backward in numeric order; migration round trip, idempotent re-run, leftover detection and CLI exit codes.tests/testMongoSearchTranspiler.pycovers both widths and the sort helpers.