Repository navigation
Add reproducible Kraken olmOCR-bench run and results - #59
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Hugging Face Jobs reproducer for Kraken's BLLA segmentation and PP-OCRv6 medium recognition, with pinned dependencies, model/data revisions, resumable stages and official olmOCR-bench scoring.
The full run completed 1,403 pages / 8,413 tests, scoring 36.8% overall (95% CI 35.9–37.7) and 74.4% on long/tiny text. Segmentation took 85.5% of measured inference time; estimated compute was $9.65, including smoke checks and retries. The run writeup covers category results, raw-text output limitations and reproduction commands.
The public artifacts include scores, sanitized per-test outcomes, page timings/hashes, costs and source/environment snapshots. Document images, transcriptions, reference text and detailed logs remain in a separate private archive.
Validation: 21-page stock/staged equivalence and resume checks, Ruff, full inference and official scoring ran on Jobs. All pages completed with zero unresolved candidate errors. Published totals reconcile with the run; anonymous download and all artifact hashes were verified.