diff --git a/CHANGELOG.md b/CHANGELOG.md index 84f69ff1..12d83a9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,8 +10,31 @@ the public-API contract. ## [Unreleased] +_Nothing yet._ + +## [7.40.0] - 2026-10-18 + ### Changed +- **TrueHD Atmos is built from the stream's own channel presentation.** With `objectAudioRendering` + in a `TrueHDObjectAudio` build, every output is now a share of the 7.1 (or 5.1) bed Dolby renders + the same mix into, split per frequency bin between floor and heights by where the objects' energy + lies (`BedSplitter`). No object signal is mixed into an output; the objects are decoded without + their matrix interpolation (LibTrueHD 3.0 no longer compiles the interpolating decode at all) and + taken as points at their positions, without size, snap, zones, screen anchoring, the elevation + switch or metadata ramps. Measured against the previous exact render through Apple's APAC decode on + 72 windows of six films, 0.35 % of the energy lands on another speaker (median of the windows with + heights), Dolby's Unfold demo 3.3 %, the heights within about a dB. A 5.1 bed now covers a 7.1 + layout's rears as well as its sides, and the split's 2048-frame latency is drained at the end of + input and before every encoder restart. The exact render it is measured against is only built with + the measurement-only `TrueHDReferenceRender` trait (#731). +- **DTS:X plays its standard presentation only, into 7.1.4 or 9.1.6.** With LibDTSX 3.0, the heights + are removed from the 7.1 bed by the standard's own constant (each at -3 dB in the speaker under it) + instead of the gains the stream's metadata states, which is no longer read; on twelve DTS:X tracks + every frame stated exactly that fold, so the output is bit-identical. The alternate presentations, + which carry objects (some IMAX-labelled streams), are refused and play as 7.1, and DTS:X is rendered + only into the two layouts that carry the bed and the four heights one to one; any other layout + keeps the channel bridge. The object position mapping and the fold renderer are gone (#744). - **The documentation no longer says TrueHD Atmos objects are always lost.** The Known limitations entry in `docs/formats.md`, the `AudioDelivery.bridged` reference and the channel bridge's notes predate `objectAudioRendering` (#731) and said objects and heights never survive. They now say the @@ -32,6 +55,21 @@ the public-API contract. decode. `aetherctl play` takes `--atmos-bed` as `serve` does and prints them: on a TrueHD Atmos remux, `obj=13/13 up=4 bed=7.1.4`. +### Fixed + +- **The persistent HTTP/1.1 ingest holds the next segment request whenever the buffer is full.** + Backpressure applied only while a segment still had bytes left over, so a segment whose last chunk + was taken just under the 16 MB ceiling sent the next request at once, past the ceiling. The + producer now waits for room between segments either way (#732; a test pins the case, and the + timing test that caught it on CI no longer flakes). +- **A slow origin no longer fails large HEVC-in-TS segments on the persistent HTTP/1.1 ingest.** + Segment bodies stream to the demuxer instead of downloading whole first, the 45 s ceiling covers + only the wait for a response head, and a body ends on its size or on 30 s without a byte (typed + `.stalled` rather than the socket's Code=60). One request per origin at a time, so a seek never + opens a second connection beside a response in flight (#732). +- **The relay admits only http and https origins.** A playlist naming another scheme (`file://` + included) answers 404 instead of reaching URLSession. + ## [7.39.0] - 2026-10-17 **Not built by default any more: TrueHD Atmos and DTS:X object rendering, Profile 7 FEL dual-layer diff --git a/Examples/README.md b/Examples/README.md index ceacda69..36a1954d 100644 --- a/Examples/README.md +++ b/Examples/README.md @@ -18,7 +18,7 @@ Three samples covering different audiences: ``` https://github.com/superuser404notfound/AetherEngine ``` - Dependency Rule: Up to Next Major Version, starting from `7.39.0`. Add the `AetherEngine` library product to your app target. + Dependency Rule: Up to Next Major Version, starting from `7.40.0`. Add the `AetherEngine` library product to your app target. 3. **Drop the file in.** Replace the Xcode template's default `App.swift` (or whatever the generated `@main` file is called) with the contents of [`MinimalPlayerApp.swift`](MinimalPlayer/MinimalPlayerApp.swift). The file is self-contained: it defines both the `@main App` struct and the `ContentView`. diff --git a/Package.resolved b/Package.resolved index fd0a9b6c..533276d5 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,5 +1,5 @@ { - "originHash" : "856dc654c740b018d3c4a31d0d78988b4c57927f83aa3d4d22ee9c48b424e1b6", + "originHash" : "7816b11b113ce741ab17b588aa7b4d9eb9f27d3c190e97a227916509bcc7ba8d", "pins" : [ { "identity" : "ffmpegbuild", @@ -24,8 +24,8 @@ "kind" : "remoteSourceControl", "location" : "https://github.com/superuser404notfound/LibDTSX", "state" : { - "revision" : "1f0dd2daaaeb899b694e8502a5bb6f570046f4f3", - "version" : "2.0.0" + "revision" : "7d68c8bbda348a10e169b9bb9619c8b2b7a0c642", + "version" : "3.0.0" } }, { @@ -33,8 +33,8 @@ "kind" : "remoteSourceControl", "location" : "https://github.com/superuser404notfound/LibTrueHD", "state" : { - "revision" : "3eab2e7c10ff1b3c2078afbab138471f2511859b", - "version" : "2.3.1" + "revision" : "aebd5ffe1046dadbea833ce87a75d00fd8b1edd1", + "version" : "3.0.0" } }, { diff --git a/Package.swift b/Package.swift index 67945ff2..76b34b82 100644 --- a/Package.swift +++ b/Package.swift @@ -37,9 +37,14 @@ let package = Package( name: "TrueHDObjectAudio", description: "Link LibTrueHD and the object renderer, so objectAudioRendering can render TrueHD Atmos objects to APAC." ), + .trait( + name: "TrueHDReferenceRender", + description: "Measurement only: with TrueHDObjectAudio and a LibTrueHD built with matrix interpolation, AETHER_TRUEHD_REFERENCE_RENDER=1 decodes TrueHD Atmos exactly and mixes its objects, the reference the bed split is measured against. Never ship it.", + enabledTraits: ["TrueHDObjectAudio"] + ), .trait( name: "DTSXObjectAudio", - description: "Link LibDTSX and the object renderer, so objectAudioRendering can render DTS:X heights and objects to APAC." + description: "Link LibDTSX and the object renderer, so objectAudioRendering can render DTS:X heights to APAC." ), .trait( name: "DualLayerDolbyVision", @@ -81,13 +86,13 @@ let package = Package( // TrueHD decoder that also decodes the Atmos object presentation (beds, objects, OAMD) for // `objectAudioRendering` (#731). The Rust `truehd` crate (Apache-2.0) behind a C API, prebuilt // as an xcframework like LibDovi so consumers need no Rust. Pinned to the minor for the same reason. - .package(url: "https://github.com/superuser404notfound/LibTrueHD", .upToNextMinor(from: "2.3.1")), // 2.3.1: truehd_atmos_version_string() names the truehd commit actually built in (read from Cargo.lock, LibTrueHD#2), same decoder; 2.3.0: decodes about a third faster again (harletty's truehd speed work: blocks decoded in place, table-driven Huffman, vectorised prediction filters, whole-block matrices; truehdd#42 included), byte-identical output; 2.2.1: rust_eh_personality exported weak, so it coalesces with LibDovi's copy and a host with its own Objective-C and C++ stays within compact unwind's three personality routines (#752); same decoder; 2.2.0: decodes about 30% faster (the truehd crate with its block and PCM buffers sized to the samples they hold, truehdd#42, from a fork of upstream main until that is merged), byte-identical output; 2.1.0: x86_64 in the iOS simulator slice, so a generic iOS Simulator build of a host links; same decoder; 2.0.0: transferred from Silo-Server/SiloObjectAudio and renamed (product TrueHD), rebuilt from source; same decoder as SiloObjectAudio 1.0.0 - // DTS:X decoder for `objectAudioRendering`: lossless DTS-HD MA with the encoder's height and object - // folds removed from the bed, the fixed heights, and the alternate presentations' objects placed in - // the same room coordinates as LibTrueHD's. The Rust `dca` crate behind a C API, a port of FFmpeg's + .package(url: "https://github.com/superuser404notfound/LibTrueHD", .upToNextMinor(from: "3.0.0")), // 3.0.0: API 2, the bed companion (the channel presentation decoded with the object presentation) and a per-handle interpolation switch; the interpolating decode is an opt-in build feature, off in the shipped binary (#731); 2.3.1: truehd_atmos_version_string() names the truehd commit actually built in (read from Cargo.lock, LibTrueHD#2), same decoder; 2.3.0: decodes about a third faster again (harletty's truehd speed work: blocks decoded in place, table-driven Huffman, vectorised prediction filters, whole-block matrices; truehdd#42 included), byte-identical output; 2.2.1: rust_eh_personality exported weak, so it coalesces with LibDovi's copy and a host with its own Objective-C and C++ stays within compact unwind's three personality routines (#752); same decoder; 2.2.0: decodes about 30% faster (the truehd crate with its block and PCM buffers sized to the samples they hold, truehdd#42, from a fork of upstream main until that is merged), byte-identical output; 2.1.0: x86_64 in the iOS simulator slice, so a generic iOS Simulator build of a host links; same decoder; 2.0.0: transferred from Silo-Server/SiloObjectAudio and renamed (product TrueHD), rebuilt from source; same decoder as SiloObjectAudio 1.0.0 + // DTS:X decoder for `objectAudioRendering`: lossless DTS-HD MA in its standard presentation, the + // four fixed heights removed from the 7.1 bed by the standard's constant fold and returned on their + // own channels; presentations with objects are refused. The Rust `dca` crate behind a C API, a port of FFmpeg's // LGPL-2.1-or-later dca decoder, so the built library is LGPL-3.0-or-later and ships as a dynamic // framework the app embeds, like FFmpegBuild's (#798). Pinned to the minor for the same reason. - .package(url: "https://github.com/superuser404notfound/LibDTSX", .upToNextMinor(from: "2.0.0")), // 2.0.0: DTSX.framework, dynamic, exporting only the C API, with dSYMs; NOTICE states the LGPL; same decoder, bit-identical output (#798); 1.0.1: rust_eh_personality weak (#752); dca 0.8.0 (harletty-bridge a717bae); the first two frames after a start or seek only prime the decoder + .package(url: "https://github.com/superuser404notfound/LibDTSX", .upToNextMinor(from: "3.0.0")), // 3.0.0: the standard presentation only, its heights removed by the standard's constant fold without reading the metadata, presentations with objects refused; same output for standard DTS:X (#744); 2.0.0: DTSX.framework, dynamic, exporting only the C API, with dSYMs; NOTICE states the LGPL; same decoder, bit-identical output (#798); 1.0.1: rust_eh_personality weak (#752); dca 0.8.0 (harletty-bridge a717bae); the first two frames after a start or seek only prime the decoder ], targets: [ .target( diff --git a/README.md b/README.md index 22874787..6385f644 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ A scannable summary; the depth for each row lives in **[docs/formats.md](docs/fo | Video (SW) | AV1 (dav1d) without HW, and AV1 High / Professional (4:4:4, 4:2:2, 12-bit) everywhere, VP9 / VP8, MPEG-4 Part 2 / MPEG-2 / VC-1, QuickTime RLE, the Flash tail (Sorenson Spark, On2 VP6) and anything else the FFmpeg build carries a decoder for (software is the default route; only HEVC, H.264 and HW-decodable AV1 go native), H.264 High 4:2:2 / 4:4:4 / 10 and HEVC Rext where VideoToolbox has no HW decoder (Intel Macs, older chips), interlaced H.264 (AVPlayer does not deinterlace; on VOD the declared field order is verified against decoded frames, so progressive-in-interlaced-carriage keeps hardware decode); GPU deinterlace (yadif_videotoolbox, Metal, field-rate by default) with a CPU bwdif fallback | | HDR | HDR10, HDR10+ (per-frame ST 2094-40), Dolby Vision (P5, P7 on the native route as single-layer 8.1 (HDR10 base layer on a display without Dolby Vision), or composed dual-layer on Metal with per-frame DV metadata for FEL when `preferDualLayerDolbyVision` is set, the `DualLayerDolbyVision` trait is on and VideoToolbox decodes HEVC in hardware, P8.1, P8.4, AV1 P10.x), HLG, HDR Vivid (played on its HLG / PQ base, CUVA metadata reported by `probe`) | | Audio | AAC, AC3, EAC3, FLAC, ALAC stream-copy lossless; TrueHD / MLP / DTS / DTS-HD MA / MP3 / MP2 / Opus / Vorbis / LPCM (incl. Blu-ray, G.711) / WMA (Standard, Pro, Lossless, Voice) / Nellymoser / ADPCM-SWF / Speex bridge to EAC3 5.1 (default) or lossless FLAC | -| Dolby Atmos | EAC3+JOC stream-copied on every route (HDMI MAT 2.0, AirPods spatial, BT downmix) except the opt-in Profile 7 FEL dual-layer route (`preferDualLayerDolbyVision`, `DualLayerDolbyVision` trait), which runs on the software host and plays an E-AC-3 JOC track as its 5.1 / 7.1 bed without the objects. No container reliably declares JOC pre-decode, so an honest `TrackInfo.isAtmos` needs a bounded decode: `AetherEngine.probeDetectingAtmos(url:/source:)` answers for a details screen without starting playback, and `LoadOptions.confirmAtmos` has the running session confirm its own tracks in the background and republish `audioTracks`. Both are opt-in and neither sits on the playback-start path. TrueHD Atmos keeps its heights too, opt-in through `LoadOptions.objectAudioRendering` in a build with the `TrueHDObjectAudio` trait ([Optional components](#optional-components)): the engine decodes the Atmos objects, renders them into a speaker bed and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio ([details](docs/formats.md#truehd-atmos-with-heights)). DTS:X takes the same route with the `DTSXObjectAudio` trait, its heights and objects unfolded from the compatible 7.1 ([details](docs/formats.md#dtsx-with-heights)) | +| Dolby Atmos | EAC3+JOC stream-copied on every route (HDMI MAT 2.0, AirPods spatial, BT downmix) except the opt-in Profile 7 FEL dual-layer route (`preferDualLayerDolbyVision`, `DualLayerDolbyVision` trait), which runs on the software host and plays an E-AC-3 JOC track as its 5.1 / 7.1 bed without the objects. No container reliably declares JOC pre-decode, so an honest `TrackInfo.isAtmos` needs a bounded decode: `AetherEngine.probeDetectingAtmos(url:/source:)` answers for a details screen without starting playback, and `LoadOptions.confirmAtmos` has the running session confirm its own tracks in the background and republish `audioTracks`. Both are opt-in and neither sits on the playback-start path. TrueHD Atmos keeps its heights too, opt-in through `LoadOptions.objectAudioRendering` in a build with the `TrueHDObjectAudio` trait ([Optional components](#optional-components)): the engine decodes the Atmos objects, renders them into a speaker bed and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio ([details](docs/formats.md#truehd-atmos-with-heights)). DTS:X takes the same route with the `DTSXObjectAudio` trait, its four heights unfolded from the compatible 7.1 into 7.1.4 or 9.1.6 ([details](docs/formats.md#dtsx-with-heights)) | | Surround | 5.1 / 7.1 with correct `AudioChannelLayout` | | Audio-only | `LoadOptions.audioOnly`: lean pipeline, no video machinery, system Now-Playing on tvOS / iOS | | Background audio | Audio keeps playing when the app backgrounds on iOS: native AVPlayer stays alive, the software path drops video and keeps decoding audio (`backgroundPlaybackEnabled` / `pictureInPictureActive`); a paused session survives quick app switches for a grace window (`backgroundTeardownGraceSeconds`, default 15 s) before the wedge-safe teardown runs. tvOS tears down immediately (wedge-safe) unless a PiP window is active, which keeps the pipeline and loopback server alive; with PiP the software path keeps decoding video too (the window needs frames). Hosts gate corrective actions on the published `isSessionReady` | @@ -387,7 +387,7 @@ Subtitle cues land in raw source PTS; render the overlay against `player.sourceT Install via Swift Package Manager: ```swift -.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.39.0") +.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.40.0") ``` Three samples ship in `Examples/`: @@ -634,10 +634,10 @@ Browse all of this as a searchable site at **[aetherengine.superuser404.de](http AetherEngine uses [Semantic Versioning](https://semver.org). The public API surface, every `public` declaration in `Sources/AetherEngine/`, is the stability contract. **Major** removes / renames public symbols or breaks adopters; **Minor** adds public API or codec / format support; **Patch** fixes bugs with no public API change. `internal` types are not part of the contract. ```swift -.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.39.0") +.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.40.0") ``` -Pin to `.upToNextMinor(from: "7.39.0")` for stricter teams that prefer to opt into minor bumps explicitly. +Pin to `.upToNextMinor(from: "7.40.0")` for stricter teams that prefer to opt into minor bumps explicitly. ## Requirements @@ -672,11 +672,11 @@ These components may be subject to third-party patent licensing. Integrators who To enable them, list the traits on the package dependency: ```swift -.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.39.0", +.package(url: "https://github.com/superuser404notfound/AetherEngine", from: "7.40.0", traits: ["TrueHDObjectAudio", "DTSXObjectAudio", "DualLayerDolbyVision", "DolbyVisionStillReshaping"]) ``` -In an Xcode project, a remote package's traits are set in the package dependency editor (Project › Package Dependencies), and Xcode stores them in the `.pbxproj`. XcodeGen cannot express them yet (2.45.4 drops a `traits:` key on a package without a word), so an XcodeGen project either patches the generated `.pbxproj` after each generate, or depends on a small local package that depends on AetherEngine with the traits set and re-exports it (`@_exported import AetherEngine`). [Probes/AllTraits](Probes/AllTraits/Package.swift) is that package with every trait on, and `Probes/DolbyVisionPulse/Device/PulseEngine` one with `DualLayerDolbyVision` alone. The engine itself builds with them through `swift build --traits TrueHDObjectAudio,DTSXObjectAudio,DualLayerDolbyVision,DolbyVisionStillReshaping` (or `--enable-all-traits`); CI builds and tests both the default and the all-traits configuration, and builds the all-traits engine for tvOS and iOS through `Probes/AllTraits`. +In an Xcode project, a remote package's traits are set in the package dependency editor (Project › Package Dependencies), and Xcode stores them in the `.pbxproj`. XcodeGen cannot express them yet (2.45.4 drops a `traits:` key on a package without a word), so an XcodeGen project either patches the generated `.pbxproj` after each generate (a `traits = ( TrueHDObjectAudio, DTSXObjectAudio, );` list in the package's `XCRemoteSwiftPackageReference`, which Xcode 27 builds with), or depends on a small local package that depends on AetherEngine with the traits set and re-exports it (`@_exported import AetherEngine`). [Probes/AllTraits](Probes/AllTraits/Package.swift) is that package with every trait on, and `Probes/DolbyVisionPulse/Device/PulseEngine` one with `DualLayerDolbyVision` alone. The engine itself builds with them through `swift build --traits TrueHDObjectAudio,DTSXObjectAudio,DualLayerDolbyVision,DolbyVisionStillReshaping` (or `--enable-all-traits`); CI builds and tests both the default and the all-traits configuration, and builds the all-traits engine for tvOS and iOS through `Probes/AllTraits`. ## Support diff --git a/Sources/AetherEngine/AetherEngine+Version.swift b/Sources/AetherEngine/AetherEngine+Version.swift index 313b7595..996db0c1 100644 --- a/Sources/AetherEngine/AetherEngine+Version.swift +++ b/Sources/AetherEngine/AetherEngine+Version.swift @@ -11,5 +11,5 @@ extension AetherEngine { /// /// Between releases, and in a consumer that pins an unreleased commit for a device test, this /// names the last PUBLISHED version the checkout descends from, not every patch in it. - public nonisolated static let version = "7.39.0" + public nonisolated static let version = "7.40.0" } diff --git a/Sources/AetherEngine/Audio/Spatial/BedSplitter.swift b/Sources/AetherEngine/Audio/Spatial/BedSplitter.swift new file mode 100644 index 00000000..d5cba812 --- /dev/null +++ b/Sources/AetherEngine/Audio/Spatial/BedSplitter.swift @@ -0,0 +1,662 @@ +// Only with the TrueHDObjectAudio trait (off by default), which links LibTrueHD. +#if TrueHDObjectAudio +import Accelerate +import Foundation + +/// Builds a `SpatialSpeakerLayout` bed out of a TrueHD stream's own channel-based presentation, the +/// 7.1 (or 5.1) bed Dolby renders the same mix into, using the object presentation only to learn +/// where its energy belongs. +/// +/// Every output is a share of that bed: per STFT bin, each bed channel is divided between the floor +/// speakers it covers and the heights, in proportion to how much of the bin's energy in that channel +/// comes from elements at each place. An element's part in a bed channel is its spectral energy times +/// its ear-level gain into the channel; it goes to the heights in proportion to its layout gains +/// above ear level, the rest to the floor speakers the channel covers. No element signal reaches an +/// output, so the elements only need to be right in where their energy lies, which an object +/// presentation decoded without its matrix interpolation is. +/// +/// Two corrections keep that estimate honest: +/// - Per band, an element can hold no more energy than the bed channels it feeds carry (`boundMargin` +/// times what its ear-level share of them allows). The decode without interpolation leaves a +/// residue in quiet elements; bounded by the bed, it can no longer pull a quiet channel's content +/// up or sideways. +/// - Per output and band, the result is steered towards the energy the elements would give that +/// output in a direct render (smoothed over about 200 ms), which restores the spread a per-bin +/// split loses where several elements share a bin. Raised fully (at most `balanceLimit`), lowered +/// only by a third of the gap in dB (`balanceLowering`): the elements' energies add up to less than +/// a mix of them wherever they carry the same signal, so a target below the split is mostly the +/// estimate's shortfall, and lowering fully cost heights. Not lowering at all let a split that +/// over-allocates stand. +/// +/// The LFE is the bed's own, delayed alongside. +/// +/// Streaming: output frame i is input frame i. The first `latency` frames after a reset produce no +/// output, and `drain` returns the ones still inside. +final class BedSplitter { + static let fftSize = 2048 + static let hop = 512 + static let latency = fftSize + static let boundMargin: Float = 2 + static let boundMinShare: Float = 0.05 + static let balanceSmoothing: Float = 0.95 + static let balanceLimit: Float = 4 + /// A lowering is applied at this fraction of its size in dB. + static let balanceLowering: Float = 1.0 / 3 + + let layout: SpatialSpeakerLayout + let bedRoles: [ObjectAudioRole] + + // MARK: Configuration + + private let n = BedSplitter.fftSize + private let h = BedSplitter.hop + private let bins = BedSplitter.fftSize / 2 + 1 + private let outputs: Int + private let lfeOutput: Int + /// Layout indices of the height speakers, and of the outputs that are split (all but the LFE). + private let heights: [Int] + private let positional: [Int] + /// Bed planes: the LFE, and the ones split onto the layout. + private let bedLFE: Int? + private let splitBeds: [Int] + /// Per split bed channel: its index in `bedPanner`'s layout (nil when the panner has no speaker + /// for it), and the pairs (`pairs` indices) of floor outputs it covers, its primary one first. + private let bedPannerIndex: [Int?] + private let pairsOfBed: [[Int]] + /// (split bed channel, floor output) for every floor output a bed channel covers. + private let pairs: [(bed: Int, output: Int)] + /// Pans a position into the bed's channels (ear level), and into the layout at ear level. + private let bedPanner: AllocentricPanner + private let layoutPanner: AllocentricPanner + private let bandEdges: [Int] + + // MARK: State + + private var elementCount = 0 + private var elementHistory: [Float] = [] + private var bedHistory: [Float] + private var elementHop: [Float] = [] + private var bedHop: [Float] + private var filled = 0 + private var accumulators: [Float] + private var lfeDelay: [Float] + /// The `h` frames being handed out during the current hop. + private var ready: [Float] + private var consumed: Int64 = 0 + private var written: Int64 = 0 + + /// Per element: the position and gain its ear-level and bed gains were computed for. + private var cachedPosition: [SIMD3?] = [] + private var cachedGain: [Float] = [] + private var earGains: [Float] = [] + private var bedGains: [Float] = [] + private var snapshot = ObjectAudioGainSnapshot() + private var hasSnapshot = false + /// Element weights for the last three hops, the oldest used: a frame spans four hops, so its + /// centre lies two hops back from the newest input. + private var weightRing: [[Float]] = [[], [], []] + private var ringNewest = 0 + private var ringCount = 0 + private let weightStride: Int + + private var smoothedTarget: [Float] + private var smoothedSplit: [Float] + + // MARK: Scratch + + private let window: [Float] + private let forward: vDSP_DFT_Setup + private let inverse: vDSP_DFT_Setup + private var frameBuf: [Float] + private var evenBuf: [Float], oddBuf: [Float], specRe: [Float], specIm: [Float] + private var power: [Float], scratch: [Float] + private var bedRe: [Float], bedIm: [Float], bedPower: [Float] + private var bedBandEnergy: [Float] + private var den: [Float], floorNum: [Float], topNum: [Float], target: [Float] + private var outRe: [Float], outIm: [Float], expect: [Float] + + init(layout: SpatialSpeakerLayout, bedRoles: [ObjectAudioRole]) { + self.layout = layout + self.bedRoles = bedRoles + let speakers = layout.speakers + outputs = speakers.count + lfeOutput = layout.lfeIndex + heights = speakers.indices.filter { speakers[$0].isHeight } + positional = speakers.indices.filter { $0 != layout.lfeIndex } + let floor = speakers.indices.filter { !speakers[$0].isHeight && $0 != layout.lfeIndex } + + bedLFE = bedRoles.firstIndex(of: .lfe) + let beds: [(plane: Int, speaker: SpatialSpeaker?, position: SIMD3)] = bedRoles.indices.compactMap { i in + switch bedRoles[i] { + case .bed(let s) where !s.isHeight: return (i, s, s.position) + case .fixed(let p) where p.z == 0: return (i, nil, p) + default: return nil + } + } + splitBeds = beds.map(\.plane) + let bedLayout = SpatialSpeakerLayout.l916 + let bedSpeakers = Set(beds.compactMap(\.speaker)) + bedPanner = AllocentricPanner( + layout: bedLayout, excluded: Set(bedLayout.speakers.filter { !bedSpeakers.contains($0) })) + bedPannerIndex = beds.map { bed in bed.speaker.flatMap { bedLayout.speakers.firstIndex(of: $0) } } + layoutPanner = AllocentricPanner(layout: layout) + + // A floor output takes the bed channels nearest to it; a bed channel no output is nearest to + // goes to the outputs nearest to it. So a 7.1 bed maps one to one onto a 7.1 floor, a 5.1 bed's + // surrounds cover a 7.1 layout's sides and rears, and a 7.1 bed's rears fold onto a 5.1 floor. + func distance(_ a: SIMD3, _ b: SIMD3) -> Float { + let d = SIMD2(a.x - b.x, a.y - b.y) + return (d * d).sum() + } + var covered = [[Int]](repeating: [], count: beds.count) + for o in floor { + let p = speakers[o].position + let distances = beds.map { distance($0.position, p) } + guard let best = distances.min() else { continue } + for (j, d) in distances.enumerated() where d <= best + 1e-6 { covered[j].append(o) } + } + for j in beds.indices where covered[j].isEmpty { + let distances = floor.map { distance(beds[j].position, speakers[$0].position) } + guard let best = distances.min() else { continue } + covered[j] = floor.indices.filter { distances[$0] <= best + 1e-6 }.map { floor[$0] } + } + // The primary output (first) is the nearest one, where a bed channel no element explains goes. + var pairs: [(bed: Int, output: Int)] = [] + var pairsOfBed = [[Int]](repeating: [], count: beds.count) + for j in beds.indices { + let sorted = covered[j].sorted { distance(beds[j].position, speakers[$0].position) + < distance(beds[j].position, speakers[$1].position) } + for o in sorted { + pairsOfBed[j].append(pairs.count) + pairs.append((j, o)) + } + } + self.pairs = pairs + self.pairsOfBed = pairsOfBed + + var edges: [Int] = [] + for i in 0...24 { + let e = Int(pow(Double(n / 2 + 1), Double(i) / 24).rounded()) + if edges.last != e { edges.append(e) } + } + edges[0] = 0 + bandEdges = edges + + // Per element: layout g² (outputs), its total over the positional outputs, its bed shares + // (split beds), floor share per pair, top share per height. + weightStride = outputs + 1 + beds.count + pairs.count + heights.count + + var w = [Float](repeating: 0, count: n) + for i in 0..]?, bed: [UnsafePointer]?, frameCount: Int, + output: [UnsafeMutablePointer]) -> Int { + let count = elements?.count ?? elementCount + if count != elementCount { resizeElements(count) } + var taken = 0 + var out = 0 + while taken < frameCount { + let take = min(h - filled, frameCount - taken) + for c in 0.. skip { + for c in 0..]) -> Int { + let pending = backlog + guard pending > 0 else { reset(); return 0 } + var out = 0 + var remaining = Self.latency + var cursor = output + while remaining > 0, out < pending { + let step = min(remaining, h) + let got = process(elements: nil, bed: nil, frameCount: step, output: cursor) + let keep = min(got, pending - out) + out += keep + cursor = cursor.map { $0 + keep } + remaining -= step + } + reset() + return out + } + + private func resizeElements(_ count: Int) { + elementCount = count + elementHistory = [Float](repeating: 0, count: count * n) + elementHop = [Float](repeating: 0, count: count * h) + cachedPosition = Array(repeating: nil, count: count) + cachedGain = [Float](repeating: 0, count: count) + earGains = [Float](repeating: 0, count: count * outputs) + bedGains = [Float](repeating: 0, count: count * bedPanner.layout.channelCount) + for i in weightRing.indices { weightRing[i] = [Float](repeating: 0, count: count * weightStride) } + ringCount = 0 + } + + // MARK: - Hop + + private func processHop() { + shiftIn(&elementHistory, from: elementHop, channels: elementCount) + shiftIn(&bedHistory, from: bedHop, channels: bedRoles.count) + pushWeights() + let weights = weightRing[(ringNewest + 3 - (ringCount - 1)) % 3] + split(weights: ringCount > 0 ? weights : nil) + emitHop() + } + + private func shiftIn(_ history: inout [Float], from hopBuffer: [Float], channels: Int) { + let n = self.n, h = self.h + history.withUnsafeMutableBufferPointer { hist in + hopBuffer.withUnsafeBufferPointer { hop in + for c in 0.. 0 else { continue } + w[base + outputs] = total + var above: Float = 0 + let tops = base + outputs + 1 + splitBeds.count + pairs.count + for (t, c) in heights.enumerated() { + w[tops + t] = w[base + c] / total + above += w[tops + t] + } + let shares = base + outputs + 1 + for j in splitBeds.indices { + if let i = bedPannerIndex[j] { + let g = bedGains[k * bedChannels + i] + w[shares + j] = g * g + } + } + let floorShares = shares + splitBeds.count + for j in splitBeds.indices { + var sum: Float = 0 + for p in pairsOfBed[j] { let e = earGains[k * outputs + pairs[p].output]; sum += e * e } + for (rank, p) in pairsOfBed[j].enumerated() { + let e = earGains[k * outputs + pairs[p].output] + let share = sum > 0 ? e * e / sum : (rank == 0 ? 1 : 0) + w[floorShares + p] = share * (1 - above) + } + } + } + } + } + + /// Windowed real FFT of `n` frames at `source`, into `specRe`/`specIm` (zrop packing). + private func spectrum(_ source: UnsafePointer) { + frameBuf.withUnsafeMutableBufferPointer { f in + vDSP_vmul(source, 1, window, 1, f.baseAddress!, 1, vDSP_Length(n)) + } + frameBuf.withUnsafeBufferPointer { f in + evenBuf.withUnsafeMutableBufferPointer { e in + oddBuf.withUnsafeMutableBufferPointer { o in + f.baseAddress!.withMemoryRebound(to: DSPComplex.self, capacity: n / 2) { c in + var split = DSPSplitComplex(realp: e.baseAddress!, imagp: o.baseAddress!) + vDSP_ctoz(c, 2, &split, 1, vDSP_Length(n / 2)) + } + } + } + } + vDSP_DFT_Execute(forward, evenBuf, oddBuf, &specRe, &specIm) + } + + /// Unpack `specRe`/`specIm` into `bins` real and imaginary values at `re`/`im`. + private func unpack(re: UnsafeMutablePointer, im: UnsafeMutablePointer) { + let half = n / 2 + specRe.withUnsafeBufferPointer { r in + specIm.withUnsafeBufferPointer { i in + re.update(from: r.baseAddress!, count: half) + im.update(from: i.baseAddress!, count: half) + re[half] = i[0] + im[0] = 0 + im[half] = 0 + } + } + } + + private func split(weights: [Float]?) { + let bins = self.bins, B = splitBeds.count, H = heights.count + let nb = vDSP_Length(bins) + let tiny: Float = 1e-20 + + // Bed spectra and powers. + bedHistory.withUnsafeBufferPointer { hist in + bedRe.withUnsafeMutableBufferPointer { re in + bedIm.withUnsafeMutableBufferPointer { im in + for j in 0.. 0 else { continue } + var shareSum: Float = 0 + for j in 0.. 0 else { continue } + elementHistory.withUnsafeBufferPointer { spectrum($0.baseAddress! + k * n) } + re.withUnsafeMutableBufferPointer { r in + im.withUnsafeMutableBufferPointer { i in unpack(re: r.baseAddress!, im: i.baseAddress!) } + } + vDSP_vmma(re, 1, re, 1, im, 1, im, 1, &power, 1, nb) + // The bed bound, per band, from every bed channel the element puts a real part of + // itself in. Per band rather than per bin: in a single bin the bed can hold less than + // one element, where several cancel in Dolby's mix, and a per-bin bound cut those. + for band in 0.. 0 else { continue } + var limit = Float.infinity + for j in 0.. Self.boundMinShare * shareSum { + limit = min(limit, Self.boundMargin * bedBandEnergy[j * bands + band] / weights[shares + j]) + } + if e > limit { + var f = limit / e + power.withUnsafeMutableBufferPointer { vDSP_vsmul($0.baseAddress! + lo, 1, &f, $0.baseAddress! + lo, 1, vDSP_Length(hi - lo)) } + } + } + target.withUnsafeMutableBufferPointer { t in + for c in positional { + var g = weights[base + c] + if g > 0 { vDSP_vsma(power, 1, &g, t.baseAddress! + c * bins, 1, t.baseAddress! + c * bins, 1, nb) } + } + } + let floorShares = shares + B + let tops = floorShares + pairs.count + for j in 0.. 0 { vDSP_vsma(power, 1, &s, f.baseAddress! + p * bins, 1, f.baseAddress! + p * bins, 1, nb) } + } + } + topNum.withUnsafeMutableBufferPointer { t in + for ti in 0.. 0 { vDSP_vsma(power, 1, &s, t.baseAddress! + row, 1, t.baseAddress! + row, 1, nb) } + } + } + } + } + } + + + // Each bed channel, split by those shares: a bin no element explains stays on the channel's + // primary floor output. + zero(&outRe); zero(&outIm); zero(&expect) + var eps = tiny + var dSafe = [Float](repeating: 0, count: bins), share = dSafe, gain = dSafe + for j in 0..?, primary: Bool) { + if let num { + if primary { + vDSP_vsadd(num, 1, &eps, &share, 1, nb) + vDSP_vdiv(dSafe, 1, share, 1, &share, 1, nb) + } else { + vDSP_vdiv(dSafe, 1, num, 1, &share, 1, nb) + } + } + var count = Int32(bins) + vvsqrtf(&gain, share, &count) + let row = output * bins, bedRow = j * bins + outRe.withUnsafeMutableBufferPointer { o in + bedRe.withUnsafeBufferPointer { vDSP_vma($0.baseAddress! + bedRow, 1, gain, 1, o.baseAddress! + row, 1, o.baseAddress! + row, 1, nb) } + } + outIm.withUnsafeMutableBufferPointer { o in + bedIm.withUnsafeBufferPointer { vDSP_vma($0.baseAddress! + bedRow, 1, gain, 1, o.baseAddress! + row, 1, o.baseAddress! + row, 1, nb) } + } + expect.withUnsafeMutableBufferPointer { e in + bedPower.withUnsafeBufferPointer { vDSP_vma($0.baseAddress! + bedRow, 1, share, 1, e.baseAddress! + row, 1, e.baseAddress! + row, 1, nb) } + } + } + floorNum.withUnsafeBufferPointer { f in + for (rank, p) in pairsOfBed[j].enumerated() { + add(pairs[p].output, num: f.baseAddress! + p * bins, primary: rank == 0) + } + } + topNum.withUnsafeBufferPointer { t in + for (ti, c) in heights.enumerated() { + add(c, num: t.baseAddress! + (j * H + ti) * bins, primary: false) + } + } + } + + // An output fed by several bed channels sums them coherently; set each bin's energy to the + // shares' sum. + for c in positional { + let row = c * bins + for b in 0.. tiny ? (expect[row + b] / got).squareRoot() : 1 + outRe[row + b] *= fix + outIm[row + b] *= fix + } + } + + // Balance per output and band, towards the energy a direct render would give it. + let a = Self.balanceSmoothing + for band in 0.. tiny, se > tiny else { continue } + let r = st / se + let g = (r >= 1 ? min(r, Self.balanceLimit) : pow(r, Self.balanceLowering)).squareRoot() + for b in lo.. ObjectAudioRole { + /// The speaker codes are LibTrueHD's: the standard presentation's 7.1 bed and its top front and + /// top back pairs, each on its own speaker. LibDTSX outputs no other channel. + static func role(forSpeaker code: UInt8) -> ObjectAudioRole? { switch code { - case DTSX_SPEAKER_LFE, DTSX_SPEAKER_LFE2: return .lfe + case DTSX_SPEAKER_LFE: return .lfe case DTSX_SPEAKER_L: return .bed(.left) case DTSX_SPEAKER_R: return .bed(.right) case DTSX_SPEAKER_C: return .bed(.center) @@ -159,27 +163,15 @@ final class DTSXObjectAudioDecoder: ObjectAudioDecoding { case DTSX_SPEAKER_RS: return .bed(.sideRight) case DTSX_SPEAKER_LB: return .bed(.rearLeft) case DTSX_SPEAKER_RB: return .bed(.rearRight) - case DTSX_SPEAKER_LW: return .bed(.wideLeft) - case DTSX_SPEAKER_RW: return .bed(.wideRight) case DTSX_SPEAKER_TFL: return .bed(.topFrontLeft) case DTSX_SPEAKER_TFR: return .bed(.topFrontRight) - case DTSX_SPEAKER_TSL: return .bed(.topMiddleLeft) - case DTSX_SPEAKER_TSR: return .bed(.topMiddleRight) case DTSX_SPEAKER_TBL: return .bed(.topRearLeft) case DTSX_SPEAKER_TBR: return .bed(.topRearRight) - case DTSX_SPEAKER_LSC: return .fixed(SIMD3(0.25, 0, 0)) - case DTSX_SPEAKER_RSC: return .fixed(SIMD3(0.75, 0, 0)) - case DTSX_SPEAKER_CB: return .fixed(SIMD3(0.5, 1, 0)) - case DTSX_SPEAKER_LSD: return .fixed(SpatialSpeaker.sideLeft.position) - case DTSX_SPEAKER_RSD: return .fixed(SpatialSpeaker.sideRight.position) - case DTSX_SPEAKER_TC: return .fixed(SIMD3(0.5, 0.5, 1)) - // On the front-top row, where LibTrueHD's TFC and the ceiling pair since #742 sit. - case DTSX_SPEAKER_TFC: return .fixed(SIMD3(0.5, 0.25, 1)) - default: return .object + default: return nil } } - /// DTS:X states no object gain, size or zone: an element is its room position. + /// A channel's element: its speaker's position, playing at unity. static func state(for element: DTSXElement) -> ObjectAudioElementState { let p = element.position return ObjectAudioElementState( diff --git a/Sources/AetherEngine/Audio/Spatial/ObjectAudioDecoding.swift b/Sources/AetherEngine/Audio/Spatial/ObjectAudioDecoding.swift index bbe3fc29..4351d322 100644 --- a/Sources/AetherEngine/Audio/Spatial/ObjectAudioDecoding.swift +++ b/Sources/AetherEngine/Audio/Spatial/ObjectAudioDecoding.swift @@ -21,6 +21,17 @@ struct ObjectAudioDecodedBlock { var planes: [UnsafePointer] /// Metadata taking effect inside this block, sorted by frame offset. var updates: [ObjectAudioMetadataUpdate] + /// The stream's channel-based presentation of the same frames (Dolby's own 7.1 or 5.1 rendering + /// of the mix), when the decoder carries one: see `BedSplitter`. + var bed: ObjectAudioBed? +} + +/// A channel bed decoded alongside an object presentation, sample-aligned with it. +struct ObjectAudioBed { + /// One per plane: `.bed`, `.fixed` or `.lfe`, never `.object`. + var roles: [ObjectAudioRole] + /// `frameCount` frames each, borrowed like the block's planes. + var planes: [UnsafePointer] } /// A decoder that turns an object-audio bitstream (TrueHD with Atmos, DTS:X) into signals plus the @@ -40,10 +51,14 @@ protocol ObjectAudioDecoding: AnyObject { /// How often the decoder has reset ITSELF after an internal failure. Its input offsets count from /// zero again afterwards, so the caller has to re-anchor its timeline. var selfResetCount: Int { get } + /// The decoder's output is meant to come from its channel bed (`ObjectAudioDecodedBlock.bed`) and + /// nothing else: a block without one plays as silence rather than through the renderer's mix. + var playsBedOnly: Bool { get } } extension ObjectAudioDecoding { var selfResetCount: Int { 0 } + var playsBedOnly: Bool { false } func releaseHeldBlocks() {} } #endif diff --git a/Sources/AetherEngine/Audio/Spatial/ObjectAudioRenderer.swift b/Sources/AetherEngine/Audio/Spatial/ObjectAudioRenderer.swift index 4d1ab170..4d95d0f2 100644 --- a/Sources/AetherEngine/Audio/Spatial/ObjectAudioRenderer.swift +++ b/Sources/AetherEngine/Audio/Spatial/ObjectAudioRenderer.swift @@ -45,6 +45,21 @@ struct ObjectAudioMetadataUpdate: Sendable { var states: [ObjectAudioElementState?] } +/// Element gains and positions as `ObjectAudioRenderer.snapshot` reads them, reused across blocks. +struct ObjectAudioGainSnapshot { + var layoutGains: [Float] = [] + var positions: [SIMD3?] = [] + var gains: [Float] = [] + + mutating func resize(elements: Int, channels: Int) { + if layoutGains.count != elements * channels { layoutGains = [Float](repeating: 0, count: elements * channels) } + if positions.count != elements { + positions = Array(repeating: nil, count: elements) + gains = [Float](repeating: 0, count: elements) + } + } +} + /// Renders object audio (beds + dynamic objects + metadata) into a `SpatialSpeakerLayout` bed. /// /// Gains are computed once per metadata update and interpolated linearly across each update's @@ -68,6 +83,9 @@ final class ObjectAudioRenderer { /// from, and ramping from silence would fade every object in after each seek. private var hasMetadata: [Bool] + /// Each object's latest state, for `snapshot`. + private var objectStates: [ObjectAudioElementState?] + private var scratch: [Float] = [] private var rampScratch: [Float] = [] @@ -80,6 +98,7 @@ final class ObjectAudioRenderer { rampRemaining = Array(repeating: 0, count: roles.count) rampTotal = rampRemaining hasMetadata = Array(repeating: false, count: roles.count) + objectStates = Array(repeating: nil, count: roles.count) // Beds and LFE have gains the moment the renderer exists; objects wait for metadata and are // silent until their first update, rather than guessing at a position. The TrueHD decoder // holds the blocks before that update and gives the first of them its state at frame 0 @@ -96,6 +115,7 @@ final class ObjectAudioRenderer { for index in roles.indices { rampRemaining[index] = 0 hasMetadata[index] = false + objectStates[index] = nil if roles[index] == .object { for c in current[index].indices { current[index][c] = 0 } } else { @@ -154,12 +174,66 @@ final class ObjectAudioRenderer { for update in pending { apply(update) } } + /// Take `frameCount` frames of metadata and ramps forward without mixing any audio: what + /// `render` does to the gains, for a caller that reads them through `snapshot` instead. + func advance(frameCount: Int, updates: [ObjectAudioMetadataUpdate]) { + var cursor = 0 + var pending = updates[...] + while cursor < frameCount { + while let next = pending.first, next.frameOffset <= cursor { + apply(next) + pending = pending.dropFirst() + } + let end = min(frameCount, pending.first?.frameOffset ?? frameCount) + for index in roles.indices where rampRemaining[index] > 0 { + rampRemaining[index] -= min(rampRemaining[index], end - cursor) + if rampRemaining[index] == 0 { current[index] = target[index] } + } + cursor = end + } + for update in pending { apply(update) } + } + + /// Every element's gains towards the layout as they stand now (mid-ramp included), and where + /// it sits: `layoutGains` holds `roles.count` rows of `layout.channelCount`; `positions` is nil + /// for the LFE and for an object without metadata yet; `gains` is the element gain. + func snapshot(into snapshot: inout ObjectAudioGainSnapshot) { + let channels = layout.channelCount + snapshot.resize(elements: roles.count, channels: channels) + for index in roles.indices { + let row = index * channels + if rampRemaining[index] > 0 { + let t = Float(rampTotal[index] - rampRemaining[index]) / Float(rampTotal[index]) + for c in 0..] = [] // Audio the decoder held back for metadata that never came plays as it is. renderHeldBlocks(into: &results) + drainSplitter() encodePending() logLevels(levelMeter.restart()) let finished = pipeline.finish() @@ -303,6 +318,9 @@ final class SpatialAudioBridge: AudioTranscodingBridge, @unchecked Sendable { defer { opLock.unlock() } decoder.reset() renderer?.reset() +#if TrueHDObjectAudio + splitter?.reset() +#endif pipeline.reset() limiter.reset() logLevels(levelMeter.restart()) @@ -384,20 +402,57 @@ final class SpatialAudioBridge: AudioTranscodingBridge, @unchecked Sendable { } else if reanchorPending, let start = encoderStartFrame { // After the decoder's own reset: put this block where the source says it plays. reanchorPending = false - let target = (originFrame ?? start + encoderFramesIn + Int64(pendingFrames)) + block.inputFrameOffset + let target = (originFrame ?? start + encoderFramesIn + Int64(pendingFrames + heldBack)) + + block.inputFrameOffset place(at: target) } else if let expected = nextExpectedInputOffset, block.inputFrameOffset > expected, let start = encoderStartFrame { - let end = start + encoderFramesIn + Int64(pendingFrames) + let end = start + encoderFramesIn + Int64(pendingFrames + heldBack) place(at: end + (block.inputFrameOffset - expected)) } nextExpectedInputOffset = block.inputFrameOffset + Int64(block.frameCount) output.ensureCapacity(pendingFrames + block.frameCount, keeping: pendingFrames) +#if TrueHDObjectAudio + if let bed = block.bed { + if splitter?.bedRoles != bed.roles { + drainSplitter() + output.ensureCapacity(pendingFrames + block.frameCount, keeping: pendingFrames) + splitter = BedSplitter(layout: layout, bedRoles: bed.roles) + EngineLog.emit( + "[SpatialAudioBridge] every output from the stream's \(bed.roles.count)-channel presentation, " + + "split by the objects' energy (latency \(BedSplitter.latency) frames)", + category: .session) + } + renderer!.advance(frameCount: block.frameCount, updates: block.updates) + renderer!.snapshot(into: &gainSnapshot) + splitter!.setGains(gainSnapshot) + pendingFrames += splitter!.process( + elements: block.planes, bed: bed.planes, frameCount: block.frameCount, + output: output.planes.map { $0 + pendingFrames }) + } else if let splitter { + // A block without its bed in a stream that has one: silence on the bed's timeline. + renderer!.advance(frameCount: block.frameCount, updates: block.updates) + pendingFrames += splitter.process( + elements: nil, bed: nil, frameCount: block.frameCount, + output: output.planes.map { $0 + pendingFrames }) + } else if decoder.playsBedOnly { + // No bed yet: silence, never a mix of the object signals. + renderer!.advance(frameCount: block.frameCount, updates: block.updates) + for plane in output.planes { (plane + pendingFrames).update(repeating: 0, count: block.frameCount) } + pendingFrames += block.frameCount + } else { + renderer!.render( + inputs: block.planes, frameCount: block.frameCount, updates: block.updates, + outputs: output.planes.map { $0 + pendingFrames }) + pendingFrames += block.frameCount + } +#else renderer!.render( inputs: block.planes, frameCount: block.frameCount, updates: block.updates, outputs: output.planes.map { $0 + pendingFrames }) pendingFrames += block.frameCount +#endif stats.samplesEnqueued += Int64(block.frameCount) stats.packetsFedSinceLastEnqueue = 0 if pendingFrames >= Self.encodeChunkFrames { encodePending() } @@ -408,13 +463,14 @@ final class SpatialAudioBridge: AudioTranscodingBridge, @unchecked Sendable { /// later packet keeps its source position. private func place(at target: Int64) { guard let start = encoderStartFrame else { return } - let end = start + encoderFramesIn + Int64(pendingFrames) + let end = start + encoderFramesIn + Int64(pendingFrames + heldBack) let gap = target - end if gap == 0 { return } if gap > 0, gap <= Int64(Self.maxSilenceFillFrames) { encodeSilence(frames: Int(gap)) return } + drainSplitter() encodePending() pipeline.restart() limiter.reset() @@ -436,6 +492,15 @@ final class SpatialAudioBridge: AudioTranscodingBridge, @unchecked Sendable { let n = min(remaining, max(Self.encodeChunkFrames - pendingFrames, 0)) if n == 0 { encodePending(); continue } output.ensureCapacity(pendingFrames + n, keeping: pendingFrames) +#if TrueHDObjectAudio + if let splitter { + pendingFrames += splitter.process( + elements: nil, bed: nil, frameCount: n, output: output.planes.map { $0 + pendingFrames }) + remaining -= n + if pendingFrames >= Self.encodeChunkFrames { encodePending() } + continue + } +#endif for plane in output.planes { (plane + pendingFrames).update(repeating: 0, count: n) } pendingFrames += n remaining -= n @@ -443,6 +508,16 @@ final class SpatialAudioBridge: AudioTranscodingBridge, @unchecked Sendable { } } + /// The frames the splitter still holds, into `output`: before the encoder restarts somewhere else + /// or the input ends, so the audio up to there plays in full. + private func drainSplitter() { +#if TrueHDObjectAudio + guard let splitter, splitter.backlog > 0 else { return } + output.ensureCapacity(pendingFrames + splitter.backlog, keeping: pendingFrames) + pendingFrames += splitter.drain(output: output.planes.map { $0 + pendingFrames }) +#endif + } + /// Hand the rendered frames to the encoder's queue. Its packets come back through /// `collectEncoded` on a later call. private func encodePending() { diff --git a/Sources/AetherEngine/Audio/Spatial/SpatialSpeakerLayout.swift b/Sources/AetherEngine/Audio/Spatial/SpatialSpeakerLayout.swift index b2004ab7..4ca86e77 100644 --- a/Sources/AetherEngine/Audio/Spatial/SpatialSpeakerLayout.swift +++ b/Sources/AetherEngine/Audio/Spatial/SpatialSpeakerLayout.swift @@ -37,6 +37,16 @@ public enum SpatialSpeakerLayout: String, Sendable, CaseIterable, Codable { public var channelCount: Int { speakers.count } + /// Whether every channel of DTS:X's standard presentation, the 7.1 bed and the four heights, has + /// a speaker of its own here, so each plays where it was authored and nothing is re-panned or + /// folded: 7.1.4 and 9.1.6. + var carriesDTSXOneToOne: Bool { + let standard: [SpatialSpeaker] = [.left, .right, .center, .lfe, .sideLeft, .sideRight, + .rearLeft, .rearRight, .topFrontLeft, .topFrontRight, + .topRearLeft, .topRearRight] + return standard.allSatisfy(speakers.contains) + } + /// CoreAudio's tag for this bed. The comments in CoreAudioBaseTypes.h spell the order /// `speakers` mirrors. public var channelLayoutTag: AudioChannelLayoutTag { diff --git a/Sources/AetherEngine/Audio/Spatial/TrueHDObjectAudioDecoder.swift b/Sources/AetherEngine/Audio/Spatial/TrueHDObjectAudioDecoder.swift index 1064456b..56dcc48a 100644 --- a/Sources/AetherEngine/Audio/Spatial/TrueHDObjectAudioDecoder.swift +++ b/Sources/AetherEngine/Audio/Spatial/TrueHDObjectAudioDecoder.swift @@ -7,6 +7,12 @@ import TrueHD /// a C API): lossless decode of presentation 3, which for an Atmos stream is the bed plus up to 15 /// dynamic objects, together with the object audio metadata (OAMD) that positions them. /// +/// With `splitsBed` (the engine's TrueHD path, see `BedSplitter`) the decode is a different one: +/// the matrix interpolation values are discarded, so the object signals are approximate, every block +/// also carries the stream's channel-based presentation (Dolby's own 7.1 or 5.1 rendering of the mix) +/// from the same access unit, and elements are described as plain point sources: the bridge then +/// takes every output from that bed and only reads where the objects' energy lies. +/// /// The library's coordinates are OAMD room coordinates (ETSI TS 103 420 4.2.1): x left→right and /// y front→back on 0...1, z −1 floor, 0 ear level, +1 ceiling. That is the renderer's own room /// model above ear level; home Atmos keeps objects at z ≥ 0, and anything below is clamped to the @@ -23,17 +29,31 @@ final class TrueHDObjectAudioDecoder: ObjectAudioDecoding { private var roles: [ObjectAudioRole] = [] private var rolesSerial: UInt32? private var planes: [UnsafePointer] = [] + let splitsBed: Bool + var playsBedOnly: Bool { splitsBed } + private var bedRoles: [ObjectAudioRole] = [] + private var bedRolesSerial: UInt32? + private var bedPlanes: [UnsafePointer] = [] private(set) var selfResetCount = 0 /// Blocks between a reset and the first metadata, so objects play from the first frame (see /// `ObjectMetadataHold`), and the released block last handed out, whose planes it owns. private let hold = ObjectMetadataHold() private var lent: ObjectMetadataHold.Copy? - init() throws { + /// `splitsBed` is false only for the measurement reference (`TrueHDReferenceRender` trait). + init(splitsBed: Bool = true) throws { + #if !TrueHDReferenceRender + precondition(splitsBed, "the exact decode and mix are only built with the TrueHDReferenceRender trait") + #endif guard let handle = truehd_atmos_decoder_create(TRUEHD_ATMOS_PRESENTATION_HIGHEST) else { throw DecoderError(status: TRUEHD_ATMOS_ERR_INVALID_ARGUMENT, message: "create failed") } self.handle = handle + self.splitsBed = splitsBed + if splitsBed { + _ = truehd_atmos_decoder_set_bed_companion(handle, true) + _ = truehd_atmos_decoder_set_ignore_interpolation(handle, true) + } } deinit { truehd_atmos_decoder_destroy(handle) } @@ -88,6 +108,18 @@ final class TrueHDObjectAudioDecoder: ObjectAudioDecoding { planes.removeAll(keepingCapacity: true) for c in 0.. 0, let channels = block.bed_channels, let speakers = block.bed_speakers { + if bedRolesSerial != block.layout_serial || bedRoles.count != bedChannels { + bedRolesSerial = block.layout_serial + bedRoles = (0.. ObjectAudioElementState { + #if TrueHDReferenceRender + splitsBed ? Self.pointSource(for: element) : Self.state(for: element) + #else + Self.pointSource(for: element) + #endif + } + + /// An element as a point at its position: no size, snap, zone exclusion, screen anchoring or + /// elevation switch. ISF elements, whose positions the library does not decode, sit in the room's + /// centre. + static func pointSource(for element: TrueHDAtmosElement) -> ObjectAudioElementState { + let active = element.flags & TRUEHD_ATMOS_ELEMENT_ACTIVE != 0 + let p = element.kind == TRUEHD_ATMOS_ELEMENT_ISF ? (Float(0.5), Float(0.5), Float(0.5)) : element.position + return ObjectAudioElementState(position: SIMD3(p.0, p.1, max(0, p.2)), gain: active ? element.gain : 0) + } + + #if TrueHDReferenceRender + // The reference render's mapping: size, snap, zones, screen anchoring and the elevation switch. static func state(for element: TrueHDAtmosElement) -> ObjectAudioElementState { let active = element.flags & TRUEHD_ATMOS_ELEMENT_ACTIVE != 0 var p = element.position @@ -220,5 +274,6 @@ final class TrueHDObjectAudioDecoder: ObjectAudioDecoding { default: return [] } } + #endif } #endif diff --git a/Sources/AetherEngine/IO/HLSIngest/HLSVODIngestReader.swift b/Sources/AetherEngine/IO/HLSIngest/HLSVODIngestReader.swift index 98956040..160a2db3 100644 --- a/Sources/AetherEngine/IO/HLSIngest/HLSVODIngestReader.swift +++ b/Sources/AetherEngine/IO/HLSIngest/HLSVODIngestReader.swift @@ -542,7 +542,9 @@ final class HLSVODIngestReader: TimeSeekableIOReader, @unchecked Sendable { throw HLSIngestError.playlistInvalid(reason: "segment body over its limit") } guard total > 0 else { throw HLSIngestError.playlistUnreachable(status: body.response.statusCode) } - return pending.isEmpty || append(pending, generation: generation) + // Between segments the ceiling holds even when the last chunk was taken just under it: the + // next request waits for the reader either way, not only when bytes are left over. + return pending.isEmpty ? waitForRoom(generation: generation) : append(pending, generation: generation) } private enum Offer { case taken, full, superseded } @@ -574,6 +576,18 @@ final class HLSVODIngestReader: TimeSeekableIOReader, @unchecked Sendable { return body } + /// Parks until the FIFO is below its ceiling. False when the generation was superseded. + private func waitForRoom(generation: UInt64) -> Bool { + condition.lock() + defer { condition.unlock() } + while self.generation == generation, + !closed, + bufferedBytes >= Self.maximumBufferedBytes { + condition.wait() + } + return self.generation == generation && !closed + } + private func append(_ data: Data, generation: UInt64) -> Bool { condition.lock() defer { condition.unlock() } diff --git a/Sources/AetherEngine/Video/HLSVideoEngine+SpatialAudio.swift b/Sources/AetherEngine/Video/HLSVideoEngine+SpatialAudio.swift index 3ab67b27..9d5a5a08 100644 --- a/Sources/AetherEngine/Video/HLSVideoEngine+SpatialAudio.swift +++ b/Sources/AetherEngine/Video/HLSVideoEngine+SpatialAudio.swift @@ -3,6 +3,20 @@ import AetherLibavformat import AetherLibavcodec import AetherLibavutil +/// Measurement only, in a build with the TrueHDReferenceRender trait: with this environment variable +/// set to 1, TrueHD Atmos is decoded exactly and its objects mixed into the layout, the rendering +/// `BedSplitter` is compared against. Without the trait it is always off and that path is not built. +enum TrueHDReferenceRender { + static let variable = "AETHER_TRUEHD_REFERENCE_RENDER" + static var isOn: Bool { + #if TrueHDReferenceRender + ProcessInfo.processInfo.environment[variable] == "1" + #else + false + #endif + } +} + extension HLSVideoEngine { /// The object-audio format a source would be rendered as, before it is confirmed. @@ -47,7 +61,9 @@ extension HLSVideoEngine { /// markers, so the label misses some DTS:X, and a DTS-HD MA track that turns out not to be DTS:X /// is refused before the bridge is built (`buildSpatialAudioProducerIfEligible`). 48 kHz, or not /// yet known, since TrueHD reports its rate from the first frame and both formats are only ever - /// authored at 48 kHz. Nil as well when the build lacks the source's decoder trait. + /// authored at 48 kHz. Nil as well when the build lacks the source's decoder trait, and for DTS + /// when the layout does not carry DTS:X's 7.1 bed and four heights one to one (see + /// `SpatialSpeakerLayout.carriesDTSXOneToOne`). static func spatialRenderingLayout( rendering: ObjectAudioRendering, codecID: AVCodecID, @@ -58,6 +74,10 @@ extension HLSVideoEngine { objectAudioSource(codecID: codecID, profile: profile)?.isBuiltIn == true, sampleRate == 0 || sampleRate == Int32(ObjectAudioRendering.sampleRate) else { return nil } + if objectAudioSource(codecID: codecID, profile: profile) == .dtsHDMasterAudio, + !layout.carriesDTSXOneToOne { + return nil + } return layout } @@ -149,6 +169,13 @@ extension HLSVideoEngine { category: .session ) } + } else if par.codec_id == AV_CODEC_ID_DTS, let layout = objectAudioRendering.layout, + !layout.carriesDTSXOneToOne { + EngineLog.emit( + "[HLSVideoEngine] DTS:X is rendered into 7.1.4 or 9.1.6 only, not \(layout.rawValue); " + + "keeping the channel bridge", + category: .session + ) } else if par.codec_id == AV_CODEC_ID_DTS, objectAudioRendering != .off { // Lossy DTS has no DTS:X to render; only the lossless profiles are offered to LibDTSX. EngineLog.emit( @@ -203,7 +230,7 @@ extension HLSVideoEngine { switch source { case .trueHDAtmos: #if TrueHDObjectAudio - decoder = try TrueHDObjectAudioDecoder() + decoder = try TrueHDObjectAudioDecoder(splitsBed: !TrueHDReferenceRender.isOn) #else return nil #endif @@ -289,11 +316,17 @@ extension HLSVideoEngine { : "DTS:X → APAC \(layout.rawValue)" audioDelivery = .bridged audioIsAPAC = true + let rendering: String + switch source { + case .trueHDAtmos where TrueHDReferenceRender.isOn: + rendering = "objects rendered into \(layout.rawValue) (reference render, \(TrueHDReferenceRender.variable))" + case .trueHDAtmos: + rendering = "channel presentation split into \(layout.rawValue) by the objects' energy" + case .dtsHDMasterAudio: + rendering = "bed and heights routed one to one into \(layout.rawValue)" + } EngineLog.emit( - "[HLSVideoEngine] \(source.name): " - + (source == .trueHDAtmos ? "objects" : "bed, heights and objects") - + " rendered into \(layout.rawValue), delivered as \(bridge.hlsCodecs) (lossy; the lossless " - + (source == .trueHDAtmos ? "7.1 channel presentation" : "7.1 downmix") + " is not used)", + "[HLSVideoEngine] \(source.name): \(rendering), delivered as \(bridge.hlsCodecs) (lossy)", category: .session ) return producer diff --git a/Tests/AetherEngineTests/BedSplitterTests.swift b/Tests/AetherEngineTests/BedSplitterTests.swift new file mode 100644 index 00000000..ca85d949 --- /dev/null +++ b/Tests/AetherEngineTests/BedSplitterTests.swift @@ -0,0 +1,168 @@ +// Only with the TrueHDObjectAudio trait: the splitter is not built without it. +#if TrueHDObjectAudio +import Testing +import Foundation +@testable import AetherEngine + +/// The TrueHD path takes every output from the stream's own channel presentation and only reads +/// where the objects' energy lies. These pin what that split has to get right: the timeline (output +/// frame i is input frame i, nothing lost at the end), where a bed channel's energy goes for an +/// element on the floor and one on the ceiling, a 5.1 bed covering a 7.1 layout, and the LFE. +@Suite("Bed splitter") +struct BedSplitterTests { + + static let bed71: [ObjectAudioRole] = [.bed(.left), .bed(.right), .bed(.center), .lfe, + .bed(.sideLeft), .bed(.sideRight), .bed(.rearLeft), .bed(.rearRight)] + static let bed51: [ObjectAudioRole] = [.bed(.left), .bed(.right), .bed(.center), .lfe, + .bed(.sideLeft), .bed(.sideRight)] + + /// Deterministic noise, so every bin has energy. + static func noise(_ count: Int, seed: UInt64) -> [Float] { + var state = seed + return (0..> 11) % 2000 - 1000) / 10_000 + } + } + + /// Runs `frames` frames through a splitter in 40-frame blocks with one element at `position` + /// whose signal is `element`, and returns every output channel, drained. + static func run(layout: SpatialSpeakerLayout, bedRoles: [ObjectAudioRole], bed: [[Float]], + element: [Float]?, position: SIMD3, frames: Int) -> [[Float]] { + let splitter = BedSplitter(layout: layout, bedRoles: bedRoles) + var snapshot = ObjectAudioGainSnapshot() + if element != nil { + snapshot.resize(elements: 1, channels: layout.channelCount) + var gains: [Float] = [] + AllocentricPanner(layout: layout).pointGains(position, into: &gains) + snapshot.layoutGains = gains + snapshot.positions = [position] + snapshot.gains = [1] + splitter.setGains(snapshot) + } + let out = (0...allocate(capacity: frames) } + defer { out.forEach { $0.deallocate() } } + var written = 0 + var offset = 0 + while offset < frames { + let n = min(40, frames - offset) + let got = bed.withPointers { bedPlanes in + (element ?? []).withUnsafeBufferPointer { e in + splitter.process( + elements: element == nil ? [] : [e.baseAddress! + offset], + bed: bedPlanes.map { $0 + offset }, frameCount: n, + output: out.map { $0 + written }) + } + } + written += got + offset += n + } + written += splitter.drain(output: out.map { $0 + written }) + #expect(written == frames) + return out.map { Array(UnsafeBufferPointer(start: $0, count: written)) } + } + + static func energy(_ x: [Float]) -> Float { x.reduce(0) { $0 + $1 * $1 } } + + static func silent(_ count: Int) -> [Float] { [Float](repeating: 0, count: count) } + + @Test("output frame i is input frame i, and everything comes out by the end", + arguments: [300, 2047, 2048, 2049, 9_000]) + func timeline(frames: Int) { + let layout = SpatialSpeakerLayout.l714 + var left = Self.silent(frames) + let at = frames / 2 + left[at] = 0.5 + var bed = Self.bed71.map { _ in Self.silent(frames) } + bed[0] = left + bed[3] = left + let out = Self.run(layout: layout, bedRoles: Self.bed71, bed: bed, element: nil, + position: SIMD3(0, 0, 0), frames: frames) + let peak = out[0].indices.max { abs(out[0][$0]) < abs(out[0][$1]) }! + #expect(peak == at) + // The LFE is a pure delay: exactly the bed's. + let lfe = out[layout.lfeIndex] + #expect(lfe == left, "LFE peak at \(lfe.indices.max { abs(lfe[$0]) < abs(lfe[$1]) }!), expected \(at)") + } + + @Test("an element on the floor keeps its bed channel on its own speaker") + func floorElement() { + let layout = SpatialSpeakerLayout.l714, frames = 48_000 + let signal = Self.noise(frames, seed: 1) + var bed = Self.bed71.map { _ in Self.silent(frames) } + bed[0] = signal + let out = Self.run(layout: layout, bedRoles: Self.bed71, bed: bed, element: signal, + position: SIMD3(0, 0, 0), frames: frames) + let total = out.map(Self.energy).reduce(0, +) + #expect(Self.energy(out[0]) / total > 0.99) + #expect(abs(total / Self.energy(signal) - 1) < 0.05) + } + + @Test("an element on the ceiling takes its bed channel's energy up to the heights") + func ceilingElement() { + let layout = SpatialSpeakerLayout.l714, frames = 48_000 + let signal = Self.noise(frames, seed: 2) + var bed = Self.bed71.map { _ in Self.silent(frames) } + bed[0] = signal + let out = Self.run(layout: layout, bedRoles: Self.bed71, bed: bed, element: signal, + position: SIMD3(0, 0, 1), frames: frames) + let topFrontLeft = layout.speakers.firstIndex(of: .topFrontLeft)! + let total = out.map(Self.energy).reduce(0, +) + #expect(Self.energy(out[topFrontLeft]) / total > 0.95) + #expect(Self.energy(out[0]) / total < 0.05) + } + + @Test("a 5.1 bed's surround covers a 7.1 layout's rear as well as its side") + func fiveOneBedOnSevenOne() { + let layout = SpatialSpeakerLayout.l714, frames = 48_000 + let signal = Self.noise(frames, seed: 3) + var bed = Self.bed51.map { _ in Self.silent(frames) } + bed[4] = signal + let rearLeft = layout.speakers.firstIndex(of: .rearLeft)! + let sideLeft = layout.speakers.firstIndex(of: .sideLeft)! + let atRear = Self.run(layout: layout, bedRoles: Self.bed51, bed: bed, element: signal, + position: SIMD3(0, 1, 0), frames: frames) + let total = atRear.map(Self.energy).reduce(0, +) + #expect(Self.energy(atRear[rearLeft]) / total > 0.95) + // The same channel with nothing to say where it belongs stays on the side, its primary output. + let unexplained = Self.run(layout: layout, bedRoles: Self.bed51, bed: bed, element: nil, + position: SIMD3(0, 1, 0), frames: frames) + #expect(Self.energy(unexplained[sideLeft]) / unexplained.map(Self.energy).reduce(0, +) > 0.99) + } + + @Test("a 7.1 bed's rears fold onto a 5.1 layout's surrounds") + func sevenOneBedOnFiveOne() { + let layout = SpatialSpeakerLayout.l514, frames = 24_000 + let signal = Self.noise(frames, seed: 4) + var bed = Self.bed71.map { _ in Self.silent(frames) } + bed[6] = signal + let out = Self.run(layout: layout, bedRoles: Self.bed71, bed: bed, element: nil, + position: SIMD3(0, 1, 0), frames: frames) + let surroundLeft = layout.speakers.firstIndex(of: .surroundLeft)! + #expect(Self.energy(out[surroundLeft]) / out.map(Self.energy).reduce(0, +) > 0.99) + } + + @Test("the element's own signal never reaches an output: a silent bed stays silent") + func elementAloneIsSilent() { + let layout = SpatialSpeakerLayout.l714, frames = 24_000 + let bed = Self.bed71.map { _ in Self.silent(frames) } + let out = Self.run(layout: layout, bedRoles: Self.bed71, bed: bed, element: Self.noise(frames, seed: 5), + position: SIMD3(0.5, 0.5, 1), frames: frames) + #expect(out.allSatisfy { $0.allSatisfy { $0 == 0 } }) + } +} + +private extension Array where Element == [Float] { + func withPointers(_ body: ([UnsafePointer]) -> R) -> R { + var pointers: [UnsafePointer] = [] + func recurse(_ i: Int) -> R { + if i == count { return body(pointers) } + return self[i].withUnsafeBufferPointer { buffer in + pointers.append(buffer.baseAddress!) + return recurse(i + 1) + } + } + return recurse(0) + } +} +#endif diff --git a/Tests/AetherEngineTests/DTSXObjectMetadataMappingTests.swift b/Tests/AetherEngineTests/DTSXObjectMetadataMappingTests.swift index 8670baf8..17b91506 100644 --- a/Tests/AetherEngineTests/DTSXObjectMetadataMappingTests.swift +++ b/Tests/AetherEngineTests/DTSXObjectMetadataMappingTests.swift @@ -5,13 +5,13 @@ import Foundation import DTSX @testable import AetherEngine -/// LibDTSX places DTS:X in the renderer's room itself; `DTSXObjectAudioDecoder` only names the +/// LibDTSX returns DTS:X's standard presentation only, and `DTSXObjectAudioDecoder` names its /// channels. These pin what a listener would hear wrong: a bed or height channel on the wrong -/// speaker, or an object somewhere other than where the library put it. -@Suite("DTS:X metadata maps onto the renderer's room") +/// speaker, or a channel outside the twelve slipping through to be panned. +@Suite("DTS:X channels map onto the renderer's speakers") struct DTSXObjectMetadataMappingTests { - @Test("bed and fixed-height speaker codes route to the matching speaker, LFE to LFE, objects are positioned") + @Test("bed and height speaker codes route to the matching speaker, LFE to LFE, nothing else is taken") func speakerRoles() { // The standard DTS:X presentation: a 7.1 bed and the top front and top back pairs. let standard: [(UInt8, ObjectAudioRole)] = [ @@ -24,49 +24,34 @@ struct DTSXObjectMetadataMappingTests { for (code, role) in standard { #expect(DTSXObjectAudioDecoder.role(forSpeaker: code) == role, "speaker \(code)") } - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_LW) == .bed(.wideLeft)) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_TSR) == .bed(.topMiddleRight)) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_TFC) == .fixed(SIMD3(0.5, 0.25, 1))) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_LSD) == .fixed(SpatialSpeaker.sideLeft.position)) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_RSD) == .fixed(SpatialSpeaker.sideRight.position)) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_OBJECT) == .object) - #expect(DTSXObjectAudioDecoder.role(forSpeaker: DTSX_SPEAKER_UNKNOWN) == .object) + for code in [DTSX_SPEAKER_OBJECT, DTSX_SPEAKER_UNKNOWN, DTSX_SPEAKER_LW, DTSX_SPEAKER_TSR, + DTSX_SPEAKER_TFC, DTSX_SPEAKER_CB, DTSX_SPEAKER_LFE2] { + #expect(DTSXObjectAudioDecoder.role(forSpeaker: code) == nil, "speaker \(code)") + } + } + + @Test("the standard channels play one to one on 7.1.4 and 9.1.6, and only there") + func oneToOneLayouts() { + for layout in SpatialSpeakerLayout.allCases { + #expect(layout.carriesDTSXOneToOne == (layout == .l714 || layout == .l916), "\(layout.rawValue)") + } } - @Test("an object keeps the room position the library gave it, at its gain") - func object() { + @Test("a channel plays at unity while active") + func channelState() { var element = DTSXElement() - element.kind = DTSX_ELEMENT_OBJECT - element.speaker = DTSX_SPEAKER_OBJECT + element.kind = DTSX_ELEMENT_BED + element.speaker = DTSX_SPEAKER_TFL element.flags = DTSX_ELEMENT_ACTIVE - element.position = (1, 0.0375, 0.2667) + element.position = (0, 0, 1) element.gain = 1 let state = DTSXObjectAudioDecoder.state(for: element) - #expect(state.position == SIMD3(1, 0.0375, 0.2667)) + #expect(state.position == SIMD3(0, 0, 1)) #expect(state.gain == 1) - #expect(state.size == 0 && !state.snap && state.excluded.isEmpty) - element.flags = 0 #expect(DTSXObjectAudioDecoder.state(for: element).gain == 0) } - @Test("an IMAX front object at 34.5°, 12° up plays mostly from its front speaker, partly from the top front") - func imaxFrontObjectThroughThePanner() { - // Independence Day's DTS:X carries two static objects at azimuth ±34.5°, elevation 12°. - // LibDTSX places the right one on the right wall 0.0375 of the way back, 12/45 up; panned, - // the floor gets cos and the ceiling sin of 12/45 of a quarter turn, as by angle. - let layout = SpatialSpeakerLayout.l714 - var gains: [Float] = [] - AllocentricPanner(layout: layout).objectGains( - position: SIMD3(1, 0.0375, 12.0 / 45), size: 0, snap: false, into: &gains) - let gain = { (speaker: SpatialSpeaker) in gains[layout.speakers.firstIndex(of: speaker)!] } - let floor = cos(Float(12.0 / 45) * .pi / 2), ceiling = sin(Float(12.0 / 45) * .pi / 2) - #expect(abs(gain(.right) - floor * cos(0.075 * .pi / 2)) < 1e-4) - #expect(abs(gain(.sideRight) - floor * sin(0.075 * .pi / 2)) < 1e-4) - #expect(gain(.topFrontRight) > 0.95 * ceiling) - #expect(gain(.left) == 0 && gain(.topFrontLeft) == 0 && gain(.rearRight) == 0) - } - @Test("the decoder opens, takes input and asks for more") func decoderLifecycle() throws { let decoder = try DTSXObjectAudioDecoder() diff --git a/Tests/AetherEngineTests/DTSXSampleTests.swift b/Tests/AetherEngineTests/DTSXSampleTests.swift index 7acde0cf..3b93ad86 100644 --- a/Tests/AetherEngineTests/DTSXSampleTests.swift +++ b/Tests/AetherEngineTests/DTSXSampleTests.swift @@ -9,7 +9,8 @@ import AetherLibavutil /// End to end on real DTS-HD elementary streams: verdict, decode, render, encode. The samples are /// not in the repository, so each test runs only when its variable names a `.dts` file: /// `AE_DTSX_SAMPLE` the "DTS:X Object Emulator" demo (standard 7.1.4, 95 s), -/// `AE_DTSX_ALTERNATE_SAMPLE` Trinnov's "Experience" trailer (7.1.4 plus five moving objects, 44 s), +/// `AE_DTSX_ALTERNATE_SAMPLE` Trinnov's "Experience" trailer (7.1.4 plus five moving objects, 44 s, +/// refused), /// `AE_DTS_HD_MA_SAMPLE` a DTS-HD Master Audio stream without DTS:X (FFmpeg's FATE /// `dts/master_audio_7.1_24bit.dts`). LibDTSX's README says where to get them. @Suite("DTS:X samples, decode to APAC") @@ -35,13 +36,18 @@ struct DTSXSampleTests { } } - @Test("DTS:X is recognised on its head and plain DTS-HD MA is refused") + @Test("DTS:X is recognised on its head; objects and plain DTS-HD MA are refused") func verdicts() throws { if let data = Self.sample("AE_DTSX_SAMPLE") { #expect(try Self.verdict(data.prefix(1 << 20)) == .renders("7.1.4")) } if let data = Self.sample("AE_DTSX_ALTERNATE_SAMPLE") { - #expect(try Self.verdict(data.prefix(1 << 20)) == .renders("7.1.4+5")) + let verdict = try Self.verdict(data.prefix(1 << 20)) + guard case .refused(let reason) = verdict else { + Issue.record("the alternate presentation was not refused: \(String(describing: verdict))") + return + } + #expect(reason.contains("7.1.4+5"), "\(reason)") } if let data = Self.sample("AE_DTS_HD_MA_SAMPLE") { let verdict = try Self.verdict(data.prefix(1 << 20)) @@ -116,35 +122,6 @@ struct DTSXSampleTests { } } - @Test("alternate-presentation objects move, and render where LibDTSX placed them") - func alternateObjectsMove() throws { - guard let data = Self.sample("AE_DTSX_ALTERNATE_SAMPLE") else { return } - let decoder = try DTSXObjectAudioDecoder() - var objects = 0, ramps = 0, positions = Set>() - try data.withUnsafeBytes { raw in - var offset = 0 - while offset < raw.count { - let n = min(65536, raw.count - offset) - try decoder.push(UnsafeRawBufferPointer(rebasing: raw[offset.. 900, "moving objects ramp over the frame that moves them") - #expect(positions.count > 1000) - #expect(positions.allSatisfy { p in (0...1).contains(p.x) && (0...1).contains(p.y) && (0...1).contains(p.z) }) - } - @Test("the bridge turns the whole standard sample into contiguous APAC packets") func bridgesToAPAC() throws { guard let data = Self.sample("AE_DTSX_SAMPLE") else { return } diff --git a/Tests/AetherEngineTests/Issue732PersistentHTTP1Tests.swift b/Tests/AetherEngineTests/Issue732PersistentHTTP1Tests.swift index 799ba021..5f79be76 100644 --- a/Tests/AetherEngineTests/Issue732PersistentHTTP1Tests.swift +++ b/Tests/AetherEngineTests/Issue732PersistentHTTP1Tests.swift @@ -283,6 +283,33 @@ final class Issue732PersistentHTTP1Tests: XCTestCase { XCTAssertEqual(server.connectionCount, 1) } + func testASegmentTakenWholeJustUnderTheCeilingStillHoldsTheNextRequest() async throws { + let server = try XCTUnwrap(KeepAliveOrigin()) + defer { server.stop() } + let segments = (0..<3).map { "seg\($0).ts" } + // Segment 0 stops 45 KB short of the 16 MB ceiling, so segment 1 (56 KB, one chunk) is taken + // whole while the FIFO is under it and lands it over: full, with nothing left in flight. + server.route = { path in + if path == "/vod.m3u8" { return .init(body: Self.mediaPlaylist(segments)) } + guard let index = segments.firstIndex(where: { "/" + $0 == path }) else { return .init(status: 404) } + return .init(body: Self.transportStream(marker: UInt8(index), packets: index == 0 ? 89_000 : 300)) + } + let made = try await HLSVODIngestReader.makeIfHEVCMPEGTS( + playlistURL: server.url("/vod.m3u8"), httpHeaders: [:], persistentHTTP1: true) + let reader = try XCTUnwrap(made) + defer { reader.close() } + + var head = [UInt8](repeating: 0, count: 188) + XCTAssertEqual(head.withUnsafeMutableBufferPointer { reader.read($0.baseAddress, size: 188) }, 188) + try await waitUntil { server.completed.contains("/seg1.ts") } + try await Task.sleep(nanoseconds: 300_000_000) + XCTAssertFalse(server.requests.contains { $0.path == "/seg2.ts" }, "the next request waits for the reader") + + let rest = Self.drain(reader) + XCTAssertEqual(188 + rest.count, (89_000 + 300 + 300) * 188) + XCTAssertEqual(server.connectionCount, 1) + } + func testStalledBodyFailsWithWhatHadArrived() async throws { let server = try XCTUnwrap(KeepAliveOrigin()) defer { server.stop() } diff --git a/Tests/AetherEngineTests/ItemSwapStillTests.swift b/Tests/AetherEngineTests/ItemSwapStillTests.swift index 4e5d1164..86c5f475 100644 --- a/Tests/AetherEngineTests/ItemSwapStillTests.swift +++ b/Tests/AetherEngineTests/ItemSwapStillTests.swift @@ -61,10 +61,13 @@ struct ItemSwapStillTests { var audioDone = false // A stall deadline, not a total one: under load the writer can be slow and still moving // (`video 4/120 ready=true, audio 24/40 ready=true` after a fixed 20 s on CI), and only a - // writer that takes nothing for 20 s is the deadlock described above. + // writer that takes nothing for 20 s is the deadlock described above. Both tracks reading + // ready at the end of such a run says the loop itself had not run: a starved task sleeps + // through the deadline, so a stall also needs the polls to show it. var lastProgress = ContinuousClock.now + var idlePolls = 0 while !(videoDone && audioDone), writer.status == .writing, - ContinuousClock.now - lastProgress < .seconds(20) { + ContinuousClock.now - lastProgress < .seconds(20) || idlePolls < 1_000 { var fed = false if !videoDone, video.isReadyForMoreMediaData { // Our own buffer, not one from the adaptor's pool: that pool belongs to the @@ -112,7 +115,13 @@ struct ItemSwapStillTests { audioDone = true } } - if fed { lastProgress = .now } else { try await Task.sleep(for: .milliseconds(5)) } + if fed { + lastProgress = .now + idlePolls = 0 + } else { + idlePolls += 1 + try await Task.sleep(for: .milliseconds(5)) + } } guard videoDone, audioDone else { let state = "video \(frame)/120 ready=\(video.isReadyForMoreMediaData), " diff --git a/Tests/AetherEngineTests/SpatialAudioBridgeTests.swift b/Tests/AetherEngineTests/SpatialAudioBridgeTests.swift index 96722c8a..5673847d 100644 --- a/Tests/AetherEngineTests/SpatialAudioBridgeTests.swift +++ b/Tests/AetherEngineTests/SpatialAudioBridgeTests.swift @@ -24,6 +24,9 @@ struct SpatialAudioBridgeTests { let framesPerAU = 40 var skipAfterReset: Int var dropAt: (push: Int, count: Int)? + /// Carry the left bed channel and the LFE as the stream's channel presentation too, as the + /// TrueHD decoder does for the bed splitter. + var withBed = false private var pushes = 0 private var skipped = 0 private var inputFrames: Int64 = 0 @@ -31,9 +34,10 @@ struct SpatialAudioBridgeTests { private let planes: [UnsafeMutablePointer] = (0..<3).map { _ in .allocate(capacity: 1 << 16) } private var t = 0 - init(skipAfterReset: Int, dropAt: (push: Int, count: Int)? = nil) { + init(skipAfterReset: Int, dropAt: (push: Int, count: Int)? = nil, withBed: Bool = false) { self.skipAfterReset = skipAfterReset self.dropAt = dropAt + self.withBed = withBed } deinit { planes.forEach { $0.deallocate() } } @@ -70,7 +74,9 @@ struct SpatialAudioBridgeTests { configurationGeneration: 0, roles: [.bed(.left), .lfe, .object], planes: planes.map { UnsafePointer($0) }, updates: [.init(frameOffset: 0, rampFrames: b.frames, - states: [.init(), .init(), .init(position: SIMD3(x, 0.5, 1))])]) + states: [.init(), .init(), .init(position: SIMD3(x, 0.5, 1))])], + bed: withBed ? ObjectAudioBed(roles: [.bed(.left), .lfe], + planes: [UnsafePointer(planes[0]), UnsafePointer(planes[1])]) : nil) } func reset() { skipped = 0; inputFrames = 0; blocks = []; pushes = 0 } @@ -94,7 +100,8 @@ struct SpatialAudioBridgeTests { private var stateSent = false private let plane = UnsafeMutablePointer.allocate(capacity: 1 << 16) - init(clickFrame: Int64) { self.clickFrame = clickFrame } + let withBed: Bool + init(clickFrame: Int64, withBed: Bool = false) { self.clickFrame = clickFrame; self.withBed = withBed } deinit { plane.deallocate() } func push(_ bytes: UnsafeRawBufferPointer) throws { @@ -113,7 +120,8 @@ struct SpatialAudioBridgeTests { return ObjectAudioDecodedBlock( sampleRate: 48_000, frameCount: b.frames, inputFrameOffset: b.offset, configurationGeneration: 0, roles: [.bed(.left)], planes: [UnsafePointer(plane)], - updates: stateSent ? [] : [.init(frameOffset: 0, rampFrames: 0, states: [.init()])]) + updates: stateSent ? [] : [.init(frameOffset: 0, rampFrames: 0, states: [.init()])], + bed: withBed ? ObjectAudioBed(roles: [.bed(.left)], planes: [UnsafePointer(plane)]) : nil) } func reset() { inputFrames = 0; blocks = []; stateSent = false } @@ -170,6 +178,13 @@ struct SpatialAudioBridgeTests { #expect(HLSVideoEngine.spatialRenderingLayout(rendering: on, codecID: AV_CODEC_ID_DTS, profile: profile, sampleRate: 48_000) == .l714) #expect(HLSVideoEngine.objectAudioSource(codecID: AV_CODEC_ID_DTS, profile: profile) == .dtsHDMasterAudio) } + // Only layouts that carry the 7.1 bed and the four heights one to one. + for layout in SpatialSpeakerLayout.allCases { + let expected: SpatialSpeakerLayout? = layout == .l714 || layout == .l916 ? layout : nil + #expect(HLSVideoEngine.spatialRenderingLayout( + rendering: .apac(layout), codecID: AV_CODEC_ID_DTS, profile: AV_PROFILE_DTS_HD_MA_X, + sampleRate: 48_000) == expected, "\(layout.rawValue)") + } for profile in [AV_PROFILE_DTS, AV_PROFILE_DTS_ES, AV_PROFILE_DTS_HD_HRA, AV_PROFILE_DTS_EXPRESS] { #expect(HLSVideoEngine.spatialRenderingLayout(rendering: on, codecID: AV_CODEC_ID_DTS, profile: profile, sampleRate: 48_000) == nil) } @@ -180,12 +195,23 @@ struct SpatialAudioBridgeTests { #endif #if TrueHDObjectAudio || DTSXObjectAudio - @Test("the first packet sits at the source position of the first decodable frame, and steps by 1024") - func anchorsOnSource() throws { + /// With the stream's bed the output comes from `BedSplitter`, whose latency the bridge has to + /// keep off the timeline: the same timestamps, and the same tail at the end. + static let bedArms: [Bool] = { + #if TrueHDObjectAudio + [false, true] + #else + [false] + #endif + }() + + @Test("the first packet sits at the source position of the first decodable frame, and steps by 1024", + arguments: bedArms) + func anchorsOnSource(withBed: Bool) throws { guard #available(macOS 26.0, iOS 26.0, tvOS 26.0, visionOS 26.0, *) else { return } let bridge = try SpatialAudioBridge( srcTimeBase: AVRational(num: 1, den: 1000), layout: .l714, - decoder: SyntheticDecoder(skipAfterReset: 5)) + decoder: SyntheticDecoder(skipAfterReset: 5, withBed: withBed)) defer { bridge.close() } var packets = try feed(bridge, startMs: 1000, count: 100) packets += bridge.flush().map { fp in @@ -358,12 +384,12 @@ struct SpatialAudioBridgeTests { #expect(a == b) } - @Test("a producer restart re-anchors on the new position") - func restartReanchors() throws { + @Test("a producer restart re-anchors on the new position", arguments: bedArms) + func restartReanchors(withBed: Bool) throws { guard #available(macOS 26.0, iOS 26.0, tvOS 26.0, visionOS 26.0, *) else { return } let bridge = try SpatialAudioBridge( srcTimeBase: AVRational(num: 1, den: 1000), layout: .l916, - decoder: SyntheticDecoder(skipAfterReset: 3)) + decoder: SyntheticDecoder(skipAfterReset: 3, withBed: withBed)) defer { bridge.close() } _ = try feed(bridge, startMs: 0, count: 30) bridge.startSegment() @@ -372,12 +398,12 @@ struct SpatialAudioBridgeTests { #expect(zip(packets, packets.dropFirst()).allSatisfy { $1.pts - $0.pts == 1024 }) } - @Test("a span the decoder drops is filled, so everything after it keeps its timestamp") - func gapKeepsTimeline() throws { + @Test("a span the decoder drops is filled, so everything after it keeps its timestamp", arguments: bedArms) + func gapKeepsTimeline(withBed: Bool) throws { guard #available(macOS 26.0, iOS 26.0, tvOS 26.0, visionOS 26.0, *) else { return } let bridge = try SpatialAudioBridge( srcTimeBase: AVRational(num: 1, den: 1000), layout: .l714, - decoder: SyntheticDecoder(skipAfterReset: 0, dropAt: (push: 10, count: 12))) + decoder: SyntheticDecoder(skipAfterReset: 0, dropAt: (push: 10, count: 12), withBed: withBed)) defer { bridge.close() } var packets = try feed(bridge, startMs: 0, count: 100) packets += bridge.flush().map { fp in @@ -497,13 +523,13 @@ struct SpatialAudioBridgeTests { /// rendered, encoded, muxed and decoded back by AVFoundation, which presents APAC the way /// AVPlayer does, and has to come out where the source put it. With the encoder's priming /// packets dropped it came out 2048 frames (42.7 ms) early, and so did every session's audio. - @Test("a click decodes back at its source position through AVFoundation") - func clickDecodesOnItsSourcePosition() async throws { + @Test("a click decodes back at its source position through AVFoundation", arguments: bedArms) + func clickDecodesOnItsSourcePosition(withBed: Bool) async throws { guard #available(macOS 26.0, iOS 26.0, tvOS 26.0, visionOS 26.0, *) else { return } // The source starts at 0 and the click is 12000 frames in, so it belongs at 0.25 s. let bridge = try SpatialAudioBridge( srcTimeBase: AVRational(num: 1, den: 1000), layout: .l714, - decoder: ClickDecoder(clickFrame: 12_000)) + decoder: ClickDecoder(clickFrame: 12_000, withBed: withBed)) defer { bridge.close() } guard let muxed = try muxWithProbeVideo(bridge, startMs: 0, count: 30) else { return } let sessionDir = muxed.sessionDir diff --git a/Tests/AetherEngineTests/TrueHDObjectMetadataMappingTests.swift b/Tests/AetherEngineTests/TrueHDObjectMetadataMappingTests.swift index 480f1124..3b3cf128 100644 --- a/Tests/AetherEngineTests/TrueHDObjectMetadataMappingTests.swift +++ b/Tests/AetherEngineTests/TrueHDObjectMetadataMappingTests.swift @@ -49,6 +49,19 @@ struct TrueHDObjectMetadataMappingTests { #expect(TrueHDObjectAudioDecoder.role(forSpeaker: TRUEHD_ATMOS_SPEAKER_UNKNOWN) == .object) } + @Test("splitting the bed, an element is a point at its position with its gain, and nothing else") + func pointSource() { + let flags = TRUEHD_ATMOS_ELEMENT_ACTIVE | TRUEHD_ATMOS_ELEMENT_SNAP | TRUEHD_ATMOS_ELEMENT_SCREEN_REF + let state = TrueHDObjectAudioDecoder.pointSource( + for: element(flags: flags, position: (0.8, 0.6, 1), size: (0.4, 0.4, 0.4), zone: TRUEHD_ATMOS_ZONE_NO_BACK)) + #expect(state == ObjectAudioElementState(position: SIMD3(0.8, 0.6, 1), gain: 0.5)) + #expect(TrueHDObjectAudioDecoder.pointSource(for: element(flags: 0)).gain == 0) + #expect(TrueHDObjectAudioDecoder.pointSource(for: element(position: (0.5, 0.5, -0.6))).position.z == 0) + #expect(TrueHDObjectAudioDecoder.pointSource(for: element(kind: TRUEHD_ATMOS_ELEMENT_ISF)).position + == SIMD3(0.5, 0.5, 0.5)) + } + + #if TrueHDReferenceRender @Test("an active elevated object keeps its position, gain and largest size dimension") func plainObject() { let state = TrueHDObjectAudioDecoder.state(for: element(size: (0.1, 0.3, 0.2))) @@ -141,6 +154,8 @@ struct TrueHDObjectMetadataMappingTests { #expect(abs(gains.reduce(0) { $0 + $1 * $1 } - 1) < 1e-5) } + #endif + @Test("the decoder opens, takes input and asks for more") func decoderLifecycle() throws { let decoder = try TrueHDObjectAudioDecoder() diff --git a/docs/api.md b/docs/api.md index 16362779..6c9cd68e 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1299,7 +1299,7 @@ All flags default to safe values; the table is the full set. Depth for the media | `nativeRemoteHLSIngestFallback` | true | The #168 / #293 carriage recovery and the #363 401/403 bypass refusal recovery. Setting it false turns both off. | | `audioOnly` | false | Lean audio pipeline, no video machinery. Also set automatically when the probe finds no video stream. | | `audioBridgeMode` | `.surroundCompat` | Bridge encoder for codecs that cannot stream-copy into fMP4. `.surroundCompat` uses EAC3 for a source with more than two channels and FLAC for one with two or fewer (no surround to carry). `.lossless` uses FLAC up to 7.1 throughout and needs a sink that accepts multichannel LPCM. | -| `objectAudioRendering` | `.off` | TrueHD Atmos and DTS:X with heights (#731), in a build with the `TrueHDObjectAudio` / `DTSXObjectAudio` package traits (off by default, [README › Optional components](../README.md#optional-components)); without a codec's trait the option is accepted and that codec keeps `audioBridgeMode`. `.apac(SpatialSpeakerLayout)` decodes the Atmos objects, or DTS:X's heights and objects, renders them into that bed (`.l512`, `.l514`, `.l712`, `.l714`, `.l916`) and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio. Lossy (320 kbps per bed channel) in place of the lossless 7.1 the bridge delivers, so it is the host's choice. Only TrueHD that FFmpeg marks Atmos, and DTS-HD Master Audio that LibDTSX confirms as DTS:X on its first frames (whatever FFmpeg's label), at 48 kHz, on OS 26+; anything else, or any failure to start, keeps `audioBridgeMode`. With it on, the automatic audio pick prefers a TrueHD Atmos or DTS:X-labelled track over the other tracks of the same language (of the `preferredAudioLanguages` match, or of the default track when no preference decides), never across a language or dub and never over `audioSourceStreamIndexOverride`, and only where the bed can be rendered (a VOD video session off the software path, on OS 26 or later). `SpatialSpeaker` names the positions. See [formats.md](formats.md#truehd-atmos-with-heights) and [DTS:X](formats.md#dtsx-with-heights). Without a codec's trait that codec keeps the channel bridge and the automatic audio pick does not prefer it; `AetherEngine.OptionalComponent` tells a host which codecs a build renders. | +| `objectAudioRendering` | `.off` | TrueHD Atmos and DTS:X with heights (#731), in a build with the `TrueHDObjectAudio` / `DTSXObjectAudio` package traits (off by default, [README › Optional components](../README.md#optional-components)); without a codec's trait the option is accepted and that codec keeps `audioBridgeMode`. `.apac(SpatialSpeakerLayout)` decodes the Atmos objects, or DTS:X's heights, renders them into that bed (`.l512`, `.l514`, `.l712`, `.l714`, `.l916`) and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio. Lossy (320 kbps per bed channel) in place of the lossless 7.1 the bridge delivers, so it is the host's choice. Only TrueHD that FFmpeg marks Atmos, and DTS-HD Master Audio that LibDTSX confirms as DTS:X in its standard presentation on its first frames (whatever FFmpeg's label; DTS:X only into `.l714` or `.l916`), at 48 kHz, on OS 26+; anything else, or any failure to start, keeps `audioBridgeMode`. With it on, the automatic audio pick prefers a TrueHD Atmos or DTS:X-labelled track over the other tracks of the same language (of the `preferredAudioLanguages` match, or of the default track when no preference decides), never across a language or dub and never over `audioSourceStreamIndexOverride`, and only where the bed can be rendered (a VOD video session off the software path, on OS 26 or later). `SpatialSpeaker` names the positions. See [formats.md](formats.md#truehd-atmos-with-heights) and [DTS:X](formats.md#dtsx-with-heights). Without a codec's trait that codec keeps the channel bridge and the automatic audio pick does not prefer it; `AetherEngine.OptionalComponent` tells a host which codecs a build renders. | | `confirmAtmos` | false | Background per-track JOC confirmation, republishing `audioTracks` as tracks confirm. Never on the start path; skipped for live and forward-only readers. | | `preferredAudioLanguages` | empty | First-frame audio pick from the engine's single probe. Ordered BCP-47 / ISO 639 tags; region and script are normalized and rank within one preference (#590). An explicit `audioSourceStreamIndex` still wins. | | `preferredSubtitleLanguages` | empty | Post-load subtitle activation on the host-overlay path. Ranks language specificity (an explicitly opposite script is rejected, not demoted) before the descriptor axis. Pure convenience: no reload and no pre-probe, unlike the audio equivalent. | diff --git a/docs/cli.md b/docs/cli.md index a21fd529..3bb45fbc 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -75,7 +75,7 @@ open 'http://127.0.0.1://master.m3u8' # macOS QuickTime `--start-position S` starts the session at S seconds, the resume anchor a host passes to `load(url:startPosition:)`. Also available on `play`. -`--atmos-bed ` is `LoadOptions.objectAudioRendering = .apac(layout)` (`5.1.2`, `5.1.4`, `7.1.2`, `7.1.4` or `9.1.6`, #731, #744), rendered only by a build with the `TrueHDObjectAudio` / `DTSXObjectAudio` traits: a TrueHD Atmos or DTS:X track is rendered into that bed and served as APAC, so a master reads `CODECS="…,apac.31.03"` (up to 12 channels) or `apac.31.04` (16), and the log carries `TrueHD Atmos: objects rendered into …` (or `DTS:X: bed, heights and objects rendered into …`) and the `[SpatialAudioBridge] bed levels` lines. A DTS-HD MA track that LibDTSX does not confirm as DTS:X logs `DTS-HD MA track is not DTS:X LibDTSX renders (…); keeping the channel bridge` instead. `--audio-index ` serves source stream `n`, as a host's audio picker would. Dolby's Unfold demo, whose TrueHD Atmos track is stream 1: `serve --no-dv --atmos-bed 7.1.4 --audio-index 1 unfold.m2ts`. +`--atmos-bed ` is `LoadOptions.objectAudioRendering = .apac(layout)` (`5.1.2`, `5.1.4`, `7.1.2`, `7.1.4` or `9.1.6`, #731, #744), rendered only by a build with the `TrueHDObjectAudio` / `DTSXObjectAudio` traits: a TrueHD Atmos or DTS:X track is rendered into that bed and served as APAC, so a master reads `CODECS="…,apac.31.03"` (up to 12 channels) or `apac.31.04` (16), and the log carries `TrueHD Atmos: channel presentation split into … by the objects' energy` (or `DTS:X: bed and heights routed one to one into …`) and the `[SpatialAudioBridge] bed levels` lines. A DTS-HD MA track that LibDTSX does not confirm as DTS:X logs `DTS-HD MA track is not DTS:X LibDTSX renders (…); keeping the channel bridge` instead. `--audio-index ` serves source stream `n`, as a host's audio picker would. Dolby's Unfold demo, whose TrueHD Atmos track is stream 1: `serve --no-dv --atmos-bed 7.1.4 --audio-index 1 unfold.m2ts`. In a build with the `TrueHDReferenceRender` trait (and a LibTrueHD built with `matrix-interpolation`), `AETHER_TRUEHD_REFERENCE_RENDER=1` in the environment decodes the TrueHD track exactly and mixes its objects through the renderer instead (`… objects rendered into 7.1.4 (reference render, …)`), the reference the split is measured against: serve the file both ways, decode each APAC track to PCM and compare per channel. `--audio-delay ` parks the server with the AE#464 audio offset already in its muxer, which is how the DELIVERED offset is measured rather than argued about: fetch the media playlist and walk its segments, then read the first audio and video packet timestamps out of `init.mp4` + a segment with `ffprobe -show_entries packet=stream_index,pts_time`. On a 30 fps H.264 + AAC fixture the source's own alignment is +21.8 ms, `--audio-delay 200` reads +221.8 ms and `--audio-delay -150` reads -128.2 ms, i.e. exactly the offset asked for, with the video timestamp unchanged in every arm. diff --git a/docs/formats.md b/docs/formats.md index 7257a04a..f1b79596 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -348,9 +348,11 @@ Matroska CodecPrivate doesn't usually carry the pre-parsed `dec3` / `dac3` box c ### TrueHD Atmos with heights -Apple TV cannot bitstream TrueHD, and no Apple platform hands a third-party app a compressed bitstream output, so the audio bridge decodes a TrueHD track and re-encodes its 7.1 channel presentation (`audioBridgeMode`). For TrueHD Atmos that loses every height and every object position before the audio leaves the engine. `LoadOptions.objectAudioRendering = .apac(layout)` keeps them (AetherEngine#731): the engine decodes the Atmos object presentation (the bed, up to 15 dynamic objects and their object audio metadata, through [LibTrueHD](https://github.com/superuser404notfound/LibTrueHD)), renders it into the `SpatialSpeakerLayout` bed the host names (5.1.2, 5.1.4, 7.1.2, 7.1.4 or 9.1.6, as `SpatialSpeaker` positions in CoreAudio's channel order), and encodes that bed as Apple Positional Audio. tvOS sends APAC to an Atmos receiver or soundbar as Dolby MAT, the same way it delivers its own spatial content, and iOS plays it as Spatial Audio. +Apple TV cannot bitstream TrueHD, and no Apple platform hands a third-party app a compressed bitstream output, so the audio bridge decodes a TrueHD track and re-encodes its 7.1 channel presentation (`audioBridgeMode`). For TrueHD Atmos that loses every height and every object position before the audio leaves the engine. `LoadOptions.objectAudioRendering = .apac(layout)` keeps them (AetherEngine#731): the engine decodes the Atmos object presentation (the bed, up to 15 dynamic objects and their object audio metadata, through [LibTrueHD](https://github.com/superuser404notfound/LibTrueHD)) together with the stream's own channel-based presentation (the 7.1, or 5.1, bed Dolby renders the same mix into), builds the `SpatialSpeakerLayout` bed the host names (5.1.2, 5.1.4, 7.1.2, 7.1.4 or 9.1.6, as `SpatialSpeaker` positions in CoreAudio's channel order) out of that channel presentation, and encodes it as Apple Positional Audio. tvOS sends APAC to an Atmos receiver or soundbar as Dolby MAT, the same way it delivers its own spatial content, and iOS plays it as Spatial Audio. -The renderer is allocentric, the model Atmos metadata is authored in: the room is a unit cube, and an object is balanced constant-power between the two speaker planes that bracket its height, the two rows that bracket its depth and the two speakers that bracket its width, so moving an object never changes its loudness. Ceiling speakers sit where ETSI TS 103 448 puts them, a quarter of the room in from each wall rather than in its corners, so an object on the ceiling a third of the way back plays mostly from the front tops, as Apple's own Atmos render places it. A 5.1 bed's surrounds sit halfway down the side walls, also as TS 103 448 places them, so on 5.1.2 and 5.1.4 an object at the back wall stays on them. Object size spreads it over a grid of points summed by power, snap sends it to the nearest speaker, zone masks keep it off the speakers they exclude, and an object without elevation stays off the ceiling. Bed channels the layout has route discretely, LFE goes to the LFE channel, and metadata updates ramp gains sample-accurately as the stream specifies. After a seek, decoding resumes at the next major sync, and the stream's first object metadata follows it up to 32 ms later; the decoder holds the audio in between and plays it at that metadata's positions, rather than leaving every object silent until it arrives (it waits at most 100 ms). +**How the bed is built.** Every output is a share of the stream's channel presentation (`BedSplitter`). Per STFT bin (2048-frame Hann windows, 512-frame hop), each of its channels is divided between the floor speakers it covers and the heights, in proportion to how much of that bin's energy comes from objects at each place: an object's part is its spectral energy times its ear-level gain into the channel, and it goes up in proportion to its layout gains above ear level. The objects only say where energy lies; no object signal reaches an output, so they are decoded without the stream's matrix interpolation values (LibTrueHD's `set_ignore_interpolation`, the channel presentation comes from the same access units through `set_bed_companion`) and taken as plain points at their positions, without size, snap, zones, screen anchoring or the elevation switch. Two corrections keep the estimate honest. Per band, an object holds no more than twice the energy its bed channels allow it, which keeps the decoding residue in quiet objects from pulling a quiet channel's content up or sideways. And per output and band, an output the split leaves short of the energy a direct render would give it is raised towards it (smoothed over about 200 ms, at most 6 dB), and one the split leaves above it is lowered by only a third of the gap in dB: objects carrying the same signal add up to more in a mix than their energies do, so a lower estimate is mostly the estimate's shortfall. A 7.1 bed maps one to one onto a 7.1 floor, a 5.1 bed's surrounds cover a 7.1 layout's sides and rears, a 7.1 bed's rears fold onto a 5.1 floor, and the LFE is the bed's own. The split runs 2048 frames behind its input: the bridge starts its output that much later after every reset and drains it before the encoder restarts elsewhere and at the end of input, so nothing moves against the picture and nothing is cut off. The exact decode mixed by the renderer, which the split is measured against, is only built with the `TrueHDReferenceRender` package trait and a LibTrueHD built with its `matrix-interpolation` feature (measurement builds, never shipped); `AETHER_TRUEHD_REFERENCE_RENDER=1` in the environment then selects it. A shipped build contains neither the interpolating decode nor the size, snap, zone, screen and elevation mapping for TrueHD. + +The renderer, which gives the split its object gains and routes DTS:X's channels, is allocentric, the model Atmos metadata is authored in: the room is a unit cube, and an object is balanced constant-power between the two speaker planes that bracket its height, the two rows that bracket its depth and the two speakers that bracket its width, so moving an object never changes its loudness. Ceiling speakers sit where ETSI TS 103 448 puts them, a quarter of the room in from each wall rather than in its corners, so an object on the ceiling a third of the way back plays mostly from the front tops, as Apple's own Atmos render places it. A 5.1 bed's surrounds sit halfway down the side walls, also as TS 103 448 places them, so on 5.1.2 and 5.1.4 an object at the back wall stays on them. In the `TrueHDReferenceRender` measurement build, object size spreads an object over a grid of points summed by power, snap sends it to the nearest speaker, zone masks keep it off the speakers they exclude, and an object without elevation stays off the ceiling. Bed channels the layout has route discretely, LFE goes to the LFE channel, and metadata updates ramp gains sample-accurately as the stream specifies. After a seek, decoding resumes at the next major sync, and the stream's first object metadata follows it up to 32 ms later; the decoder holds the audio in between and plays it at that metadata's positions, rather than leaving every object silent until it arrives (it waits at most 100 ms). **Track choice.** With the option on, the automatic audio pick prefers a TrueHD Atmos track over the other tracks of the same language: of the `preferredAudioLanguages` match, or of the session's default track when no preference decides it. It never crosses a language or a dub (a region or script both tags state and disagree on), never picks a commentary for it, applies only where the bed can be rendered (a VOD video session off the software path, on OS 26 or later), and an explicit `audioSourceStreamIndexOverride` still wins. Without that, a film carrying DTS-HD and TrueHD Atmos in one language usually opened on whichever the container listed first. @@ -368,17 +370,17 @@ many objects play and how many overhead without reading the log. ### DTS\:X with heights -DTS:X reaches the bridge the same way TrueHD Atmos does: Apple TV cannot bitstream DTS-HD, so `audioBridgeMode` decodes the lossless 7.1, and on a DTS:X track that 7.1 is a compatible downmix into which the encoder folded the four height channels and any objects. `objectAudioRendering = .apac(layout)` keeps them in a build with the `DTSXObjectAudio` package trait (off by default; without it DTS-HD keeps the channel bridge whatever the option asks, [README › Optional components](../README.md#optional-components)): a DTS-HD Master Audio track goes through [LibDTSX](https://github.com/superuser404notfound/LibDTSX) (the `dca` crate from harletty-bridge behind a C API, packaged like LibTrueHD) into the same renderer, limiter and APAC encoder. LibDTSX does the DTS-specific work, so the engine side is a speaker mapping (`DTSXObjectAudioDecoder`): it subtracts every fold the frame states (or that `dca`'s fold renderer recomputes for an object the encoder panned into the bed) from the bed, so the bed, the heights and the objects play side by side without anything playing twice, and it places DTS:X's angular object positions in LibTrueHD's room coordinates. The reference sphere is projected onto the room's boundary through the speakers' angles (centre 0°, front ±30°, wide ±60°, side ±90°, rear ±145°), the ceiling reached at the 45° of a height speaker and its centre straight overhead, so the dual-balance panner places a sound between two speakers in the same proportion an angular panner would. LibDTSX comes with the `DTSXObjectAudio` package trait, off by default, in the same way. +DTS:X reaches the bridge the same way TrueHD Atmos does: Apple TV cannot bitstream DTS-HD, so `audioBridgeMode` decodes the lossless 7.1, and on a DTS:X track that 7.1 is a compatible downmix into which the encoder folded the four height channels. `objectAudioRendering = .apac(layout)` keeps them in a build with the `DTSXObjectAudio` package trait (off by default; without it DTS-HD keeps the channel bridge whatever the option asks, [README › Optional components](../README.md#optional-components)): a DTS-HD Master Audio track goes through [LibDTSX](https://github.com/superuser404notfound/LibDTSX) (the `dca` crate from harletty-bridge behind a C API, packaged like LibTrueHD) into the same renderer, limiter and APAC encoder, and the engine side is a speaker mapping (`DTSXObjectAudioDecoder`). -The standard presentation, which every DTS:X film in SpatialAudioBridge's library census used in every frame, is a 7.1 bed and four fixed heights (top front and top back pairs), all routed discretely: rendered into 7.1.4, *8 Mile*, *The Bourne Identity* and the DTS:X Object Emulator demo give each speaker exactly its channel, bit for bit what SpatialAudioBridge's DTS:X renderer gives. The alternate presentations add one to five objects; `dca` classifies them from a corpus and calls them experimental. *Independence Day* (FFmpeg: "DTS-HD MA + DTS:X IMAX") is one: two static objects at ±34.5° and 12° up, which pan mostly into the front pair, a little into the sides and the front heights. Against SpatialAudioBridge's renderer, which pans DTS:X by angle over a bed ring and a height ring, the 7.1.4 render of a minute of it points the sound the same way within 0.07° (median of 200 ms windows, energy-weighted; 90th percentile 0.39°), and of the D4 trailer's five moving objects within 0.23° (2.3°); the floor channels correlate at 1.0000, and the ceiling's share of the energy is 0.1 to 0.2 points lower. These figures were taken against the panner before #742 moved its top speakers from the room's corners a quarter in, so the object renders now differ from them slightly; the standard presentation routes discretely and is unaffected. +Only DTS:X's standard presentation is rendered: a 7.1 bed and four fixed heights (the top front and top back pairs), each height folded at -3 dB into the bed speaker under it (top front left into L, top back left into the left rear, and so on) so that a 7.1 decoder plays the whole soundtrack. LibDTSX subtracts exactly that constant from the bed and returns the heights on their own channels; it does not read the stream's private metadata. Every frame of twelve DTS:X tracks measured (eight *Harry Potter* films, two *Shrek* films in two languages, *Mary and the Witch's Flower*) stated this same fold, so the result is bit-identical to subtracting what each frame states. The bed and the heights route discretely, one channel per speaker, so DTS:X is rendered into the two layouts that carry all twelve one to one, 7.1.4 and 9.1.6; any other layout keeps the channel bridge, and so does a track in one of the alternate presentations, which add one to five objects (some IMAX-labelled streams, Trinnov's "Experience" trailer): LibDTSX refuses those on their first frame, and they play as the 7.1 their compatible bed is. -**Which tracks.** FFmpeg names only two of DTS:X's markers, so it labels some DTS:X plain "DTS-HD MA" (Trinnov's "Experience" trailer, with five moving objects, is one), and a track it does label DTS:X can carry a presentation LibDTSX refuses. Every 48 kHz DTS-HD MA track is therefore offered to LibDTSX, which decides on the track's first three frames. The engine shows them to it through the demuxer's peek (`peekPackets`), which holds what it reads for the producer, so deciding costs no extra read and no seek: on a resume the demuxer is put where the first producer starts before the peek, not after it, and the frames there decide. The peek holds at most 512 packets or 24 MiB, every stream's included, and gives up without a verdict past either. A refused track, or one without a verdict, keeps `audioBridgeMode` with its lossless 7.1 as before, and a stream that turns out not to be 48 kHz is refused on its first block. Lossy DTS (core, ES, 96/24, High Resolution) is never offered. The automatic audio pick prefers a track FFmpeg labels DTS:X the way it prefers TrueHD Atmos; a plain-labelled DTS:X track is rendered when it is the track that plays, but cannot be ranked before playback. +**Which tracks.** FFmpeg names only two of DTS:X's markers, so it labels some DTS:X plain "DTS-HD MA" (Trinnov's "Experience" trailer is one), and a track it does label DTS:X can carry a presentation LibDTSX refuses. Every 48 kHz DTS-HD MA track is therefore offered to LibDTSX, which decides on the track's first three frames. The engine shows them to it through the demuxer's peek (`peekPackets`), which holds what it reads for the producer, so deciding costs no extra read and no seek: on a resume the demuxer is put where the first producer starts before the peek, not after it, and the frames there decide. The peek holds at most 512 packets or 24 MiB, every stream's included, and gives up without a verdict past either. A refused track, or one without a verdict, keeps `audioBridgeMode` with its lossless 7.1 as before, and a stream that turns out not to be 48 kHz is refused on its first block. Lossy DTS (core, ES, 96/24, High Resolution) is never offered. The automatic audio pick prefers a track FFmpeg labels DTS:X the way it prefers TrueHD Atmos; a plain-labelled DTS:X track is rendered when it is the track that plays, but cannot be ranked before playback. **Seeks.** A DTS-HD Master Audio frame is lossless only against the core decoder's filter and predictor history, which a stream entered mid-way does not have: the first frame after a seek decodes distorted (over 120 seeks into five DTS:X streams, its error a median 2.7 dB below the signal) and the next can still err by -16 dBFS. LibDTSX decodes the first two frames after a start or seek without output and counts them on its input timeline, so the bridge's first encoded frame sits 21 ms after the segment's start, where those frames end, and nothing fills the gap; the first block then differs from a continuous decode by at most -33 dBFS, and the output is bit-identical again within about ten frames. -**On the receiver.** Over Jellyfin on an Apple TV 4K (tvOS 27.2), *8 Mile*, *The Bourne Identity* and *Independence Day* played as APAC with the soundbar showing Atmos, from the middle of the film and across seeks (back to playing in 0.7 to 2.2 s), with the same `AtmosOutputActivator` TrueHD Atmos uses; *Arrival*, plain DTS-HD MA, was refused on its first frame and showed PCM. +**On the receiver.** Over Jellyfin on an Apple TV 4K (tvOS 27.2), *8 Mile* and *The Bourne Identity* played as APAC with the soundbar showing Atmos, from the middle of the film and across seeks (back to playing in 0.7 to 2.2 s), with the same `AtmosOutputActivator` TrueHD Atmos uses; *Arrival*, plain DTS-HD MA, was refused on its first frame and showed PCM. Those runs predate the standard-only decoder, whose output for these films is the same bit for bit. -**What the session log says.** `[HLSVideoEngine] DTS-HD MA track is DTS:X 7.1.4 (LibDTSX, 3 ms)` names the presentation the verdict found (`7.1.4+2` for *Independence Day*), and a refusal names its reason (`lossless DTS-HD Master Audio without a DTS:X extension`). The `object configuration` and `bed levels` lines are the same as for TrueHD Atmos. The 1200-packet silence report is too: at 512 samples a DTS frame that is 12.8 s of DTS, against a second of TrueHD. +**What the session log says.** `[HLSVideoEngine] DTS-HD MA track is DTS:X 7.1.4 (LibDTSX, 3 ms)` names the presentation the verdict found, and a refusal names its reason (`lossless DTS-HD Master Audio without a DTS:X extension`, or for *Independence Day* `DTS:X in the 7.1.4+2 presentation, with objects: only the standard 7.1.4 presentation is rendered`). A layout other than 7.1.4 or 9.1.6 logs `DTS:X is rendered into 7.1.4 or 9.1.6 only`. The `object configuration` and `bed levels` lines are the same as for TrueHD Atmos. The 1200-packet silence report is too: at 512 samples a DTS frame that is 12.8 s of DTS, against a second of TrueHD. ## Subtitles