Skip to content

feat(audio): analyse the mixed envelope, not the raw source - #202

Merged
LeadcodeDev merged 1 commit into
mainfrom
fix/audio-analysis-follows-mix
Aug 12, 2026
Merged

feat(audio): analyse the mixed envelope, not the raw source#202
LeadcodeDev merged 1 commit into
mainfrom
fix/audio-analysis-follows-mix

Conversation

@LeadcodeDev

@LeadcodeDev LeadcodeDev commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Answers the open question on #190 (option A). Unblocked by #203.

What

volume, volume_keyframes and the fades are what comes out of the speakers, and since #182 the studio plays exactly that. A waveform drawing the raw file's envelope while a fade takes the sound down contradicts what the viewer hears — the same class of defect as a gif that paints nothing or a halo at the wrong scale.

How

The gain is the encoder's own, factored out of the mixer loop:

pub(crate) fn track_gain_at(track: &AudioTrack, t_in_track: f64, audible: f64) -> f32

Expressed in seconds rather than sample indices so the mixer (which works at OUTPUT_SAMPLE_RATE on resampled audio) and the analysis (which works at the file's own rate on the decoded source) share one implementation. A second copy would drift, and the entire point is that the picture matches the sound. The mixer's 35 tests pass unchanged against the extracted function.

The cache fingerprint now hashes the serialised AudioTrack instead of start/end alone: two scenarios can name the same file with different mixes, and adding a field to AudioTrack must not silently leave it out of the key.

What implementing it uncovered

This surfaced #201 — the audio-reactive painters were reading scene-local time against a scenario-time analysis. While the envelope was flat that was invisible: a periodic track shows peaks at any offset. With a real envelope it was not, and the showcase's waveform went flat. #203 fixed the clock; this can now land.

Verified after the rebase, on the showcase scene whose track is gated to it by volume_keyframes: the waveform is drawn and follows the envelope across the scene (37 px → 34 px → 27 px at t = 74 / 75.5 / 77 s).

Not in scope

The analysis still describes one track at a time, not the summed mix of several. That only matters for a scenario with overlapping tracks feeding one visualisation, and it is a separate question.

Tests

  • the analysis follows a keyframe envelope that closes at 1 s: audible before, silent after, on an unbroken sine so any variation can only come from the envelope
  • changing volume re-analyses the same file rather than serving the cached entry

cargo test --workspace green, cargo fmt --check and cargo clippy --all-targets -- -D warnings clean.

volume, volume_keyframes and the fades are what comes out of the speakers, and
since #182 the studio plays exactly that — a waveform drawing the raw file's
envelope while a fade takes the sound down contradicts what the viewer hears.

The gain comes from the encoder's own envelope, factored out of the mixer loop
as track_gain_at(track, t_in_track, audible) and expressed in seconds so the
mixer (OUTPUT_SAMPLE_RATE, resampled) and the analysis (the file's own rate,
decoded source) share one implementation. A second copy would drift, and the
whole point is that the picture matches the sound.

The cache fingerprint now hashes the serialised track rather than start/end
alone: two scenarios can name the same file with different mixes, and adding a
field to AudioTrack must not silently leave it out of the key.

Not mergeable yet — see #201. The components read scene-local time against a
scenario-time analysis, which is harmless while the envelope is flat and turns
into a flat trace as soon as it is not.
@LeadcodeDev
LeadcodeDev force-pushed the fix/audio-analysis-follows-mix branch from a09db3a to 6268ca8 Compare August 12, 2026 17:41
@LeadcodeDev
LeadcodeDev marked this pull request as ready for review August 12, 2026 17:41
@LeadcodeDev
LeadcodeDev merged commit b5b3b4d into main Aug 12, 2026
3 checks passed
@LeadcodeDev
LeadcodeDev deleted the fix/audio-analysis-follows-mix branch August 12, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant