Phonon: keep the Mimi decoder state on the GPU for CUDA (#25021) by psiddh · Pull Request #25021 · pytorch/executorch

psiddh · 2026-11-13T05:57:16Z

Summary:

mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph.

StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs.

This diff was written with the help of Claude Code (Claude Opus 5.5).

Reviewed By: shoumikhin, Gasoonjia

Differential Revision: D131675810

pytorch-bot · 2026-11-13T05:57:20Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25021

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 2d35fa1 with merge base fe03d52 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

meta-codesync · 2026-11-13T05:58:03Z

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131675810.

github-actions · 2026-11-13T05:58:56Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Summary: Pull Request resolved: pytorch#25021 mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph. StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs. This diff was written with the help of Claude Code (Claude Opus 5.5). Differential Revision: D131675810

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Gasoonjia

Review automatically exported from Phabricator review in Meta.

Summary: Pull Request resolved: pytorch#25021 mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph. StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: Gasoonjia Differential Revision: D131675810

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Summary: mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph. StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: shoumikhin, Gasoonjia Differential Revision: D131675810

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Summary: mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph. StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: shoumikhin, Gasoonjia Differential Revision: D131675810

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Summary: mimi_decode takes its streaming state (15 tensors, 2.1 MB, mostly the decoder transformer's keys and values) as inputs and returns it as outputs, so on CUDA every 80 ms frame copied it to the GPU and back, with a synchronization each way. That made mimi_decode take 4 ms in the runtime against 1.3 ms for the compiled graph. StatefulMimiDecode wraps MimiDecodeStep and keeps the state in mutable buffers of the program (MimiState): forward(latent, reset) -> pcm, where reset = 1 starts an utterance from the zero state. The CUDA export uses it (mimi_state_in_program in export_phonon.py); the other backends keep the state as inputs and outputs. The program records the choice in a new mimi_state_in_program constant and phonon.json, the C++ and Python runners pass (latent, reset) when it is set, and format_version is now 2, so the runners ask to re-export older programs. This diff was written with the help of Claude Code (Claude Opus 5.5). Reviewed By: shoumikhin, Gasoonjia Differential Revision: D131675810

psiddh requested review from SS-JIA, digantdesai and larryliu0820 as code owners November 13, 2026 05:57

Copilot AI balanced review requested due to automatic review settings November 13, 2026 05:57

Copilot AI reviewed Nov 13, 2026

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 13, 2026

meta-codesync Bot added the meta-exported label Nov 13, 2026

psiddh force-pushed the export-D131675810 branch from 82b31e6 to ae1dbae Compare November 13, 2026 17:29

meta-codesync Bot changed the title ~~Phonon: keep the Mimi decoder state on the GPU for CUDA~~ Phonon: keep the Mimi decoder state on the GPU for CUDA (#25021) Nov 13, 2026

Copilot AI balanced review requested due to automatic review settings November 13, 2026 17:29

Copilot AI reviewed Nov 13, 2026

Gasoonjia approved these changes Nov 13, 2026

View reviewed changes

psiddh force-pushed the export-D131675810 branch from ae1dbae to 14cf793 Compare November 13, 2026 19:08

Copilot AI balanced review requested due to automatic review settings November 13, 2026 19:08

Copilot AI reviewed Nov 13, 2026

psiddh force-pushed the export-D131675810 branch from 14cf793 to df07551 Compare November 14, 2026 03:18

Copilot AI balanced review requested due to automatic review settings November 14, 2026 03:18

Copilot AI reviewed Nov 14, 2026

psiddh force-pushed the export-D131675810 branch from df07551 to dafeee6 Compare November 14, 2026 07:50

Copilot AI balanced review requested due to automatic review settings November 14, 2026 07:50

Copilot AI reviewed Nov 14, 2026

psiddh force-pushed the export-D131675810 branch from dafeee6 to 7447555 Compare November 14, 2026 17:14

psiddh force-pushed the export-D131675810 branch from 7447555 to 2d35fa1 Compare November 14, 2026 18:38

psiddh merged commit a532d10 into pytorch:main Nov 14, 2026
262 checks passed

Navigation Menu

Repository navigation

Phonon: keep the Mimi decoder state on the GPU for CUDA (#25021) - #25021

Phonon: keep the Mimi decoder state on the GPU for CUDA (#25021)#25021
psiddh merged 1 commit into
pytorch:mainfrom
psiddh:export-D131675810

psiddh commented Nov 13, 2026 •

edited by meta-codesync Bot

Loading

Uh oh!

pytorch-bot Bot commented Nov 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

meta-codesync Bot commented Nov 13, 2026

Uh oh!

github-actions Bot commented Nov 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Gasoonjia left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

psiddh commented Nov 13, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25021

⏳ No Failures, 1 Pending

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

meta-codesync Bot commented Nov 13, 2026

Uh oh!

github-actions Bot commented Nov 13, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Gasoonjia left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

psiddh commented Nov 13, 2026 •

edited by meta-codesync Bot

Loading

pytorch-bot Bot commented Nov 13, 2026 •

edited

Loading

This PR needs a `release notes:` label