Skip to content

perf: enable gzip/brotli on witness/data HTTP transport#147

Merged
flyq merged 2 commits into
mainfrom
liquan/perf/witness-http-compression
Jun 24, 2026
Merged

perf: enable gzip/brotli on witness/data HTTP transport#147
flyq merged 2 commits into
mainfrom
liquan/perf/witness-http-compression

Conversation

@flyq

@flyq flyq commented Jun 23, 2026

Copy link
Copy Markdown
Member

Summary

The witness/data RPC client uses alloy-provider's HTTP transport, backed by reqwest 0.12, which had no compression features enabled — so the validator fetched responses uncompressed and paid the full base64(zstd) witness size on the wire (mega_getBlockWitness returns "v0:" + base64(zstd), ~33% larger than raw zstd).

This adds a direct reqwest = "0.12" dependency in stateless-common with the gzip and brotli features. Via Cargo feature unification it enables compression on the reqwest 0.12.24 that alloy-provider shares, so reqwest auto-sends Accept-Encoding: gzip, br and transparently decompresses. The crate is intentionally not referenced in code.

Once the client advertises support, Cloudflare serves the witness Worker response Brotli-compressed, recovering the base64 overhead: in testing one witness dropped from 30095 B (identity) to 22672 B (content-encoding: br), ~25% less wire bandwidth (larger witnesses save more).

No code or wire-format change; cargo check -p stateless-common passes and Cargo.lock was regenerated by cargo.

The witness/data RPC client uses alloy-provider's HTTP transport, backed by reqwest 0.12, which had no compression features enabled — so the validator fetched responses uncompressed and paid the full base64(zstd) witness size on the wire (mega_getBlockWitness returns "v0:"+base64(zstd), ~33% larger than raw zstd). Add a direct reqwest 0.12 dependency in stateless-common with the gzip and brotli features; via Cargo feature unification this enables compression on the reqwest 0.12.24 that alloy shares, so reqwest now auto-sends Accept-Encoding: gzip, br and transparently decompresses. Cloudflare serves the witness Worker response Brotli-compressed, recovering the base64 overhead (~25% less witness bandwidth in testing: 30095B -> 22672B). No code or wire-format change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@flyq

flyq commented Jun 23, 2026

Copy link
Copy Markdown
Member Author

验证器栈实测(reqwest 0.12 + PR 开的 gzip/brotli)

对 6906400–6906404 这 5 个块,用 validator 实际的 reqwest 栈各发三种请求:

block before-PR 线上(identity) after-PR 线上(gzip) ratio ce app 看到的 app 的 ce
6906400 4,868,895 3,647,773 0.749 gzip 4,868,895 none(已剥离)
6906401 4,781,587 3,582,283 0.749 gzip 4,781,587 none(已剥离)
6906402 4,842,711 3,628,198 0.749 gzip 4,842,711 none(已剥离)
6906403 4,796,275 3,593,764 0.749 gzip 4,796,275 none(已剥离)
6906404 4,893,919 3,667,023 0.749 gzip 4,893,919 none(已剥离)
合计 24,183,387 18,119,041 0.7492

after-PR 比 before-PR 省 25.1% 线上带宽。

三点结论

  1. PR 效果在 validator 自己的 HTTP 栈上确认:开了 gzip/brotli 后,每个 witness 线上小 ~25%(content-encoding: gzip)。这就是 PR perf: enable gzip/brotli on witness/data HTTP transport #147 的实际收益。
  2. 为什么 validator 加 log 看不到:app 看到的 列 == before_id(完整大小),且 app 的 ce = none(已剥离) —— 证实 reqwest 透明解压(解压后还把 content-encoding/length 头删掉),所以应用层拿到的永远是完整 witness,应用内打 size 日志量不出这个节省。线上(wire)测量才是对的方法,正如这个探针所做。
  3. 交叉验证一致:curl 跑 21 个不同块(6906400–6906420)给出完全相同的 0.7492 比例(98,991,263 → 74,163,987 B,−25.1%),per-block 比例都是 0.749。两种方法、两种工具,结论一致。

一句话

PR #147 对官方端点(它本身就 gzip 压缩响应)带来 ~25% 的 witness 带宽下降,witness 越大绝对节省越多;收益完全在传输层,应用语义/数据零变化。这个探针是临时的,已删除,不进 PR。

@flyq flyq merged commit 7099004 into main Jun 24, 2026
20 of 21 checks passed
@flyq flyq deleted the liquan/perf/witness-http-compression branch June 24, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants