Repository navigation
Conversation
… a reported error A collection is saved as one JSON string, and V8 cannot build a string longer than buffer.constants.MAX_STRING_LENGTH (~537 M chars). On a large library the passage index reached ~461 M chars. Past the limit every save threw RangeError, was retried as a transient failure, the retry counter reset in `finally`, and each later write re-ran a multi-second stringify that had to fail, while new passages were lost on restart. - Save only `docs.docs`. Restore has read nothing else since v2.0.0 and rebuilds every index by re-inserting documents. Orama's full export also carried the vector index (a second copy of every embedding) and the text and sort indexes: more than half of each file. Old full-export files still restore, and older builds read the new ones. - A collection whose JSON is too long to build raises `orama.too-large:<name>`, the other collections still save, and it is not stringified again until its document count drops below where it failed. A warning is logged once per launch past 85% of the limit. - A file that decompresses past the limit is no longer quarantined as corrupt. The collection starts empty for the session and that stand-in is never saved, blob or sidecar, so the unread file is not overwritten. A reset or rebuild replaces the stand-in and saving resumes. - `finally` cleared no timer before re-arming, so a write during a save left two timers and a second save started right after the first. - Startup sweeps `*.tmp*` files over an hour old from the vector-db folder (the existing userData sweep is top-level only); interrupted saves left hundreds of MB there.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A search-index collection is saved as one JSON string, and V8 cannot build a string longer than
buffer.constants.MAX_STRING_LENGTH(about 537 million characters). On a large library the passage index reached about 461 million. Past the limit, every save threwRangeErrorand was retried as if it were a transient failure. The retry counter reset infinally, so each later write re-ran a stringify that was certain to fail and took several seconds. Meanwhile new passages were lost at each restart. More than half of that string was Orama's index export, which restore never reads.What changed
docs.docs. Restore has read nothing else since v2.0.0, and it rebuilds every index by inserting the documents again. Dropping the vector index (a second copy of every embedding) and the text and sort indexes should roughly halve each file. Files in the old full-export format still restore, and older builds can read the new files.orama.too-large:<name>(the degraded banner) and keeps saving the other collections. It doesn't stringify that collection again until its document count drops below the count at which it failed. A warning is logged once per launch when a save passes 85% of the limit..corrupt. The collection starts empty for the session, and that empty stand-in is never saved, neither the blob nor the sidecar, so the unread file stays on disk untouched. A reset or rebuild replaces the stand-in, and saving resumes.finallyre-armed its timer without clearing the debounce timer that a write made during the save had armed. Both fired, so a second save started right after the first.*.tmp*files older than an hour are removed from the vector-db folder. The existing userData sweep only covers the top level, and interrupted saves had left hundreds of MB here.Verification
test/OramaVectorService.saveSafety.test.js: 10 tests. 9 fail onmain. The 10th checks that an old-format file still restores, so it passes on both.npm run cion Node 24.22.1 (Windows) is green: 822/822 suites, 12,327 passed, 4 skipped, everyverify:*step, knip and the build.