Stop the metrics feed asking for events it discards, and make allocation profiles usable - #30
Merged
Merged
Conversation
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three harness fixes that came out of investigating why superblocks K6 TTFB averages twice the processing time Nethermind reports over its SSE feed. They are independent; the first is the one with a measurement consequence.
Ask the metrics feed only for the event it reads
SseClientparsesevent: processedand discards everything else, but it subscribes to/data/eventswith no filter, so the client is asked for every event type. One of those isforkChoice, which makes Nethermind serialize the entire head block, every transaction, receipt and log, to JSON on eachforkchoiceUpdated. On a 1 GGas superblock that is roughly 60 MB of JSON plus the writer's growth buffers, allocated on the large object heap, built on a thread-pool thread while the next block is being processed. Nethermind only does this while something is subscribed, so on a node with no dashboard open the cost does not exist. We are creating it by measuring.The effect is not subtle. The same Nethermind image measured with and without that subscription:
processedonlyThat difference is entirely the serialization we asked for and threw away. It also inflates the client's large-object churn by about 125 MB per block, which drives extra background gen2 collections, which is what the TTFB gap is largely made of.
The change is a query string on the feed path. It is backward compatible in both directions: a node that does not understand
events=ignores the query and streams everything as before, and the client filters as it always has. So this can merge independently of any client change.Nethermind side: NethermindEth/nethermind#13334 adds the
events=parameter and the per-type gating. Until this PR lands, PR-triggered benchmark runs usemainand therefore keep measuring the client with the serialization included, which makes that change read as a no-op.Stop the dotnet-trace collector before the client
The collector was stopped during scenario cleanup, after the client container had already been torn down. The runtime emits its method rundown when the trace session ends, so stopping it after the process is gone means the rundown never happens and no managed frame in the
.nettraceresolves. Allocation events arrive with type names and sizes but no attribution.Moving the stop ahead of the client teardown makes stacks resolve, which is what allowed the allocation above to be attributed to the data feed rather than guessed at from type names.
Let the sidecar's CLR event set be overridden
Alongside dotTrace the sidecar is hardcoded to
gc+contention+threading+exceptionatinformational, which does not includeGCAllocationTick, so an allocation profile cannot be captured at all.EXPB_DOTNET_TRACE_CLREVENTSandEXPB_DOTNET_TRACE_CLREVENTLEVELnow override both, defaulting to today's values. Settinggcatverboseproduces the allocation-by-type-and-stack profile used above.Testing
Exercised end to end on the amd64 benchmark runner through
run-expb-reproducible-benchmarkswithexpb_branchpointed at this branch: 5 multi-image superblocks runs, 3 fusaka runs, and 2 profiling runs withEXPB_DOTNET_TRACE_CLREVENTS=gcandEXPB_DOTNET_TRACE_CLREVENTLEVEL=verbose. The profiling runs produced.nettracefiles with resolvable managed stacks, which the previous collector ordering did not.If you would rather take the subscription fix on its own first, it is the last commit and stands alone.