feat: add byte tracking#745
Conversation
This commit introduced a new feature flag that enables the ability to track byte offsets. When the source-positions feature is enabled, the tokenizer will track the number of UTF-8 bytes consumed from the input so far. This is done by giving BufferQueue a `bytes_consumed` field, which is incremented every time a character is consumed. The changes in cargo.toml were needed to make this project load as a git submodule in the Cadmus project. The xhtml-self-closing feature was needed due to EPUBs using XHTML-compatible self-closing on RCDATA/RAWTEXT elements. Change-Id: 566446e2bca101b7fefdca639c1b4d26 Change-Id-Short: uttvvtlxonpy
a332771 to
0d70705
Compare
Note that servo does not accept AI-generated contributions: https://book.servo.org/contributing/getting-started.html#ai-contributions. Admittedly we should do a better job of communicating that this applies to all repositories under the servo org, not just |
|
I'd be fine with you rewriting the tests and rustdoc.
That's okay.
You don't need to prove that. We rely on the honesty of contributors when rejecting AI generated patches (unless the patch is obviously slop). Thanks for being upfront about it. |

This commit introduced a new feature flag that enables the ability to track byte offsets.
When the
source-positionsfeature is enabled, the tokenizer will track the number of UTF-8 bytes consumed from the input. This is done by giving BufferQueue abytes_consumedfield, which is incremented every time a character is consumed.The changes in cargo.toml were needed to make this project load as a git submodule in the Cadmus project.
The
xhtml-self-closingfeature was needed due to EPUBs using XHTML-compatible self-closing onRCDATA/RAWTEXTelements. (baskerville/plato#426 (comment))Change-Id: 566446e2bca101b7fefdca639c1b4d26
Change-Id-Short: uttvvtlxonpy
🤖 was used to generate some of the rustdocs and tests. Was also used to educate me on the project and brainstorm implementation path.
I have a local branch in Cadmus that added this project as a submodule, works great and seems to not break anything, need to do some more testing by daily driving and asking others to test. (Will cross link once I open PR there, needs a tiny bit of more work)
I'll leave this in draft, so it won't get merged by accident, but I'm ready to get feedback on it. Will mark it as ready once I finish with the changes in Cadmus.
Closes: #734
Ref: