fix(server): transcode HEIC attachments instead of failing the turn - #5229
fix(server): transcode HEIC attachments instead of failing the turn#5229danigilme wants to merge 1 commit into
Conversation
Attaching or pasting a photo straight from an Apple device sends `image/heic`, which the Claude adapter rejects with "Unsupported Claude image attachment type". HEIC is the default capture format on iOS, so this fires constantly from the mobile composer and from pasting an iPhone screenshot on macOS. The attachment itself is stored fine (`imageMime.ts` already lists HEIC as a safe type) and the mobile composer already labels it `image/heic`, so the only thing missing was converting it before handing it to the provider. `transcodeImageToJpeg` shells out to a tool that ships with the host rather than adding a HEIC decoder dependency: `sips` on macOS, which is always present, and `heif-convert` from libheif elsewhere. If neither is available the conversion fails and the caller reports the attachment as unsupported exactly like before, so the change is strictly additive. The helper lives in its own module with no orchestration imports, so it survives the orchestration-v2 move; the adapter call site is a single `yield*`. Fixes pingdotgg#4579 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:unvouched
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
There was a problem hiding this comment.
One convention finding: the new HEIC transcode path performs subprocess and filesystem work with raw Node APIs behind a Promise, and that imperative helper becomes a dependency of the ClaudeAdapter Effect service. See the inline comments.
Posted via Macroscope — Effect Service Conventions
| if (needsTranscode) { | ||
| const hostPlatform = yield* HostProcessPlatform; | ||
| const converted = yield* Effect.tryPromise({ |
There was a problem hiding this comment.
This is the service boundary where the imperative helper is pulled in. Once transcodeImageToJpeg is an Effect over ProcessRunner/FileSystem, this becomes a plain yield* and the wrapper keeps a structured tagged failure as cause rather than an opaque rejection, with the requirement visible in the adapter's layer types.
Posted via Macroscope — Effect Service Conventions
| function runTranscoder(input: { | ||
| readonly transcoder: Transcoder; | ||
| readonly inputPath: string; | ||
| readonly outputPath: string; | ||
| }): Promise<void> { | ||
| return new Promise((resolve, reject) => { | ||
| NodeChildProcess.execFile( | ||
| input.transcoder.command, | ||
| input.transcoder.args({ | ||
| inputPath: input.inputPath, | ||
| outputPath: input.outputPath, | ||
| }), | ||
| // A photo is a bounded workload; the cap only guards against a wedged | ||
| // helper process holding the turn open forever. | ||
| { timeout: 30_000, maxBuffer: 1024 * 1024 }, | ||
| (error) => (error === null ? resolve() : reject(error)), | ||
| ); | ||
| }); | ||
| } |
There was a problem hiding this comment.
This module spawns a child process and does temp-file I/O with raw Node APIs, and it is then consumed from inside the ClaudeAdapter Effect service via Effect.tryPromise. Per the dependency-acquisition convention, an imperative Promise adapter shouldn't become a dependency of another Effect service — runtime-backed dependencies should be acquired from the environment.
The server already models both of these as services: ProcessRunner (apps/server/src/processRunner.ts, with run, timeouts, and tagged ProcessSpawnError/ProcessTimeoutError/… failures) and FileSystem.FileSystem (makeTempDirectoryScoped, used elsewhere for exactly this scratch-dir pattern, e.g. apps/server/src/atomicWrite.ts).
Suggested shape: make transcodeImageToJpeg an Effect that does yield* ProcessRunner.ProcessRunner and yield* FileSystem.FileSystem, and fail with a Schema.TaggedErrorClass (e.g. ImageTranscodeError carrying the transcoder command and the underlying cause) instead of throw new Error(...)/throw lastError. The platform parameter can stay as-is — that's pure configuration, not service injection. That also lets the nodeBuiltinImport:off suppression at the top of the file be dropped.
Posted via Macroscope — Effect Service Conventions
ApprovabilityVerdict: Needs human review This PR introduces new functionality (HEIC-to-JPEG transcoding via external process calls) rather than a simple bug fix. The change modifies attachment processing behavior in ClaudeAdapter and spawns external processes, warranting human review for this new capability. You can customize Macroscope's approvability policy. Learn more. |
1 participant
What Changed
ClaudeAdapternow converts HEIC/HEIF image attachments to JPEG instead of failing the turn withUnsupported Claude image attachment type 'image/heic'.apps/server/src/imageTranscode.tsexposingisTranscodableImageMimeTypeandtranscodeImageToJpeg.buildUserMessageEffectconverts a transcodable attachment after reading its bytes and pushes the JPEG content block. Everything else is untouched.No new dependencies, and no
package.jsonchange.Why
Fixes #4579.
HEIC is the default capture format on every recent iPhone, so this fires constantly: attaching a photo from the mobile composer, or pasting an iPhone screenshot on desktop. The attachment pipeline already handles HEIC end to end (
imageMime.tslists it inSAFE_IMAGE_FILE_EXTENSIONS, andapps/mobile/src/lib/composerImages.tslabels itimage/heic), so the file is stored correctly and only the provider hand-off rejects it. The issue asks for exactly this: "the image should be converted automagically to a supported format".Why shelling out rather than a decoder dependency
A HEIC decoder (
heic-convert,sharp) would be a new runtime dependency for a narrow edge case. Instead this uses what the host already has:sipson macOS, which is part of the OS and always present, andheif-convertfrom libheif elsewhere. When neither exists the conversion fails and the caller reports the attachment as unsupported exactly as it does today, so the change cannot regress any current behaviour.Note on #2829
#4200 was closed in favour of the orchestration-v2 rewrite because it built on
apps/server/src/orchestration/**andprovider/Layers/*Adapter.ts. This PR keeps all the logic in a standalone module with no orchestration imports, so it survives that move. The adapter call site is a singleyield*that can be dropped into wherever the attachment loop lands in v2.Verification
Run on macOS 15, Node 24:
pnpm exec vp test run src/imageTranscode.test.ts src/provider/Layers/ClaudeAdapter.test.ts— 67 passed, including the HEIC to JPEG round trippnpm exec tsgo --noEmit— cleanpnpm exec vp lint— no new findingspnpm exec vp fmt— appliedThe round-trip test builds a genuine HEIC fixture with
sips, asserts theftypheicbox is present, converts it, and checks the output starts with the JPEG SOI marker. It skips itself on hosts withoutsips.UI Changes
None.
Checklist
Note
Medium Risk
Turn-start path now shells out to host transcoding tools with temp files and a 30s timeout; failure modes are handled but behavior depends on OS tooling (especially Linux without libheif).
Overview
HEIC/HEIF image attachments no longer fail Claude turns with an unsupported MIME error. The adapter now transcodes those formats to JPEG before building the SDK image block.
A new
imageTranscodemodule detects transcodable MIME types (HEIC/HEIF and sequence variants), writes bytes to a temp dir, and runssipson macOS orheif-convertelsewhere—no new npm HEIC decoder. If conversion fails, the turn still errors with a message that includes the original type.ClaudeAdapter.buildUserMessageEffectbranches after reading attachment bytes: transcodable images go throughtranscodeImageToJpegwithHostProcessPlatform; others keep the existing supported-type check. Tests cover MIME detection and a real HEIC→JPEG round trip whensipsis available.Reviewed by Cursor Bugbot for commit b43c309. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Transcode HEIC/HEIF image attachments to JPEG before sending to Claude
transcodeImageToJpeg, which writes the input to a temp file, runssipson macOS orheif-convertelsewhere, and returns the JPEG bytes.isTranscodableImageMimeTypeand transcode before building the image content block; throwsProviderAdapterRequestErrorif transcoding fails.sips(macOS) orheif-convert(Linux) being available on the host; missing tools cause the request to error rather than silently skip the attachment.Macroscope summarized b43c309.