Repository navigation
fix: bound graph rebuilds of the public RPC and release its connections before external calls - #534
Conversation
| let Some(stored) = stored else { | ||
| return Ok(None); | ||
| }; | ||
| Ok(Some(tokio::task::spawn_blocking(move || stored.build()).await??)) |
There was a problem hiding this comment.
[P1] Keep the semaphore permit inside the blocking build
_permit is owned by the HTTP handler future. When the client disconnects, that future is dropped and the permit is released, but an already-running spawn_blocking closure continues executing. Repeating requests followed by disconnects therefore bypasses GRAPH_BUILDS = 2, allowing unauthenticated clients to accumulate expensive graph builds and consume CPU/memory.
I reproduced this with a loopback axum HTTP/1.1 server using this exact load_public_graph body and a gated substitute for stored.build(): start two requests, wait until both builds are running, disconnect both clients, then send a third request. The third build starts while the original two are still running. The regression asserting at most two concurrent builds fails.
Please move the acquired permit into the spawn_blocking closure so it remains held until the actual build finishes, even if the awaiting handler is cancelled. Add a regression covering client disconnects while builds are running.
There was a problem hiding this comment.
Fixed in be1a57d. The permit now moves into the spawn_blocking task, so a request that is dropped keeps it until the build finishes. Added a regression: two builds running, both requests aborted, no permit is available until the builds end.
be1a57d to
4aefc26
Compare
Audit items M26 and M27. Open against
dev.Graph rebuilds (M26).
/v1/graphs/{id}/txand/v1/graphs/{id}/txntake no auth and have no cache or rate limit, and each request did the full graph work twice: once inload_validated_graph_definition, which rebuilds the graph to check its pre-signatures, and once more in the handler'sfrom_simplified. Both ran on the tokio workers while the request held a pool connection.load_validated_graph_definitionis split into a database read (read_graph_definition) and a CPU check (StoredGraphDefinition::build).buildreturns the graph it already rebuilt. Both endpoints now:buildon the blocking pool, at most 2 at a time (GRAPH_BUILDS);Other callers of
load_validated_graph_definitionare unchanged.Connections during external calls (M27). The pool has 10 connections, shared with the engine.
get_instances,get_instance,get_graph,get_graphsand the two endpoints above kept their connection through the esplora and Goat calls, so plain GET requests could take every connection; engine and inbox writes then timed out and/v1/healthreturned 503. Each of them now releases its connection after its last query.Review notes
send-challenge,send-verifier-challenge) still use the old path; they are behindrpc-debug-endpoints.Tests: the public build rejects an unsigned graph with the same error as the validated load, and the read returns nothing for an unknown graph and fails for a mismatched instance. Node tests, workspace clippy with
-D warnings(with and withoutrpc-debug-endpoints) and fmt pass.The test coverage and validation notes above are from the zkm PR notes. Branch scope and diff formatting were checked locally; builds and tests were not rerun during publication.