feat(audio): analyse the mixed envelope, not the raw source - #202
Merged
Conversation
volume, volume_keyframes and the fades are what comes out of the speakers, and since #182 the studio plays exactly that — a waveform drawing the raw file's envelope while a fade takes the sound down contradicts what the viewer hears. The gain comes from the encoder's own envelope, factored out of the mixer loop as track_gain_at(track, t_in_track, audible) and expressed in seconds so the mixer (OUTPUT_SAMPLE_RATE, resampled) and the analysis (the file's own rate, decoded source) share one implementation. A second copy would drift, and the whole point is that the picture matches the sound. The cache fingerprint now hashes the serialised track rather than start/end alone: two scenarios can name the same file with different mixes, and adding a field to AudioTrack must not silently leave it out of the key. Not mergeable yet — see #201. The components read scene-local time against a scenario-time analysis, which is harmless while the envelope is flat and turns into a flat trace as soon as it is not.
LeadcodeDev
force-pushed
the
fix/audio-analysis-follows-mix
branch
from
August 12, 2026 17:41
a09db3a to
6268ca8
Compare
LeadcodeDev
marked this pull request as ready for review
August 12, 2026 17:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers the open question on #190 (option A). Unblocked by #203.
What
volume,volume_keyframesand the fades are what comes out of the speakers, and since #182 the studio plays exactly that. Awaveformdrawing the raw file's envelope while a fade takes the sound down contradicts what the viewer hears — the same class of defect as agifthat paints nothing or a halo at the wrong scale.How
The gain is the encoder's own, factored out of the mixer loop:
Expressed in seconds rather than sample indices so the mixer (which works at
OUTPUT_SAMPLE_RATEon resampled audio) and the analysis (which works at the file's own rate on the decoded source) share one implementation. A second copy would drift, and the entire point is that the picture matches the sound. The mixer's 35 tests pass unchanged against the extracted function.The cache fingerprint now hashes the serialised
AudioTrackinstead ofstart/endalone: two scenarios can name the same file with different mixes, and adding a field toAudioTrackmust not silently leave it out of the key.What implementing it uncovered
This surfaced #201 — the audio-reactive painters were reading scene-local time against a scenario-time analysis. While the envelope was flat that was invisible: a periodic track shows peaks at any offset. With a real envelope it was not, and the showcase's waveform went flat. #203 fixed the clock; this can now land.
Verified after the rebase, on the showcase scene whose track is gated to it by
volume_keyframes: the waveform is drawn and follows the envelope across the scene (37 px → 34 px → 27 px at t = 74 / 75.5 / 77 s).Not in scope
The analysis still describes one track at a time, not the summed mix of several. That only matters for a scenario with overlapping tracks feeding one visualisation, and it is a separate question.
Tests
volumere-analyses the same file rather than serving the cached entrycargo test --workspacegreen,cargo fmt --checkandcargo clippy --all-targets -- -D warningsclean.