feat: WWDC26 Apple Intelligence APIs by JKobrynski · Pull Request #218 · callstackincubator/ai

JKobrynski · 2026-06-17T14:02:22Z

Related PR - #215

Summary

Builds on the WWDC26 Apple Intelligence API work from #215 and finishes the JS-facing history management API for @react-native-ai/apple.

PR #215 added the broader Apple Intelligence surface, including Apple provider options, model info APIs, iOS 27 Private Cloud Compute support, image prompts, Image Playground generation, Vision built-in tools, and lower-level native helpers. It also introduced context/history utilities internally.

This change exposes those history management utilities as fluent AI SDK model wrappers so consumers can compose Apple history behavior directly from JS:

const summarizerModel = apple()

const model = apple()
  .summarizeHistory(5000, summarizerModel)
  .rollingWindow(10)
  .droppingCompletedToolCalls()

It also adds demo app controls for testing the behavior without replacing the existing basic chat flow.

Changes

Add fluent Apple model wrappers for:
- summarizeHistory(threshold, model)
- rollingWindow(entries)
- droppingCompletedToolCalls()
Apply history transforms before both doGenerate and doStream
Preserve dynamic tool configuration when wrapping Apple language models
Keep transform ordering aligned with Apple-style chained history utilities
Export Apple language model types needed by consumers and the demo app
Add Apple History Demo controls to the Expo example app
Add settings for enabling history management, rolling window size, and summary threshold
Preserve the existing basic chat demo behavior when the history demo is disabled
Document the fluent history management API in the package README and website docs
Include [codex] Add WWDC26 Apple Intelligence APIs #215’s WWDC26 Apple Intelligence API work as the base for this follow-up

Testing

Ran bun run --filter='@react-native-ai/apple' typecheck
Ran bun run --filter='@react-native-ai/example' typecheck
Ran bun lint
Ran git diff --check
Tested manually in the Expo demo app on iOS 26.5 with Apple Intelligence selected
Verified basic chat still works with the Apple History Demo disabled
Verified history transforms run when the Apple History Demo is enabled
Verified rolling window trims older conversation entries while preserving system context
Verified summarization applies when the prompt exceeds a low test threshold
Verified completed tool-call pruning with a synthetic tool-call history path
Removed temporary debug logging after validation

vercel · 2026-06-17T14:02:27Z

@JKobrynski is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

Copilot

Pull request overview

Exposes Apple Foundation Models “history management” utilities (summarization, rolling window, completed tool-call pruning) as fluent JS model wrappers in @react-native-ai/apple, adds native support for model info + Image Playground, and extends the Expo example app to toggle/demo the new behaviors.

Changes:

Add fluent Apple language model wrappers (summarizeHistory, rollingWindow, droppingCompletedToolCalls) and apply transforms before both doGenerate and doStream.
Add native APIs for getModelInfo and generateImages, plus JS exports/types and updated docs/README.
Add “Apple History Demo” toggles + sliders in the Expo example app and preserve existing chat behavior when disabled.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
website/src/docs/apple/generating.md	Updates Apple provider docs (availability, PCC, images, history helpers, error handling).
packages/apple-llm/src/NativeAppleLLM.ts	Extends TurboModule typings to support attachments/providerOptions + new native methods.
packages/apple-llm/src/index.ts	Exports new Apple provider types + `trimAppleMessagesForContext`.
packages/apple-llm/src/AppleFoundationModels.ts	Adds JS fallbacks/wrappers for `getModelInfo` and `generateImages`.
packages/apple-llm/src/ai-sdk.ts	Implements fluent history wrappers, image model, prompt attachment handling, and model-info API.
packages/apple-llm/README.md	Documents new features and requirements + shows fluent history usage.
packages/apple-llm/ios/AppleLLMImpl.swift	Implements `getModelInfo`, Image Playground generation, attachments, PCC, Vision tools plumbing.
packages/apple-llm/ios/AppleLLMError.swift	Refines invalid-message error shape/message.
packages/apple-llm/ios/AppleLLM.mm	Bridges new options + adds `getModelInfo`/`generateImages` TurboModule methods.
apps/expo-example/src/store/chatStore.ts	Adds persisted settings for the Apple History Demo.
apps/expo-example/src/screens/ChatScreen/SettingsSheet.tsx	Adds History Demo toggle + sliders for window/threshold.
apps/expo-example/src/screens/ChatScreen/index.tsx	Applies fluent history wrappers when demo enabled; improves empty state text plumbing.
apps/expo-example/src/screens/ChatScreen/ChatMessages.tsx	Accepts precomputed empty-state subtitle from parent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

artus9033 · 2026-06-20T13:06:24Z

+function isToolRelatedMessage(message: LanguageModelV3Message) {
+  if (message.role === 'tool') {
+    return true
+  }
+
+  if (message.role !== 'assistant') {
+    return false
+  }
+
+  return message.content.some(
+    (part) => part.type === 'tool-call' || part.type === 'tool-result'
+  )
+}


Let's take a look at this

Good catch. This is handled now by checking Array.isArray(message.content) before looking for tool-call / tool-result parts. If the assistant content is a plain string, we now return false, so it is preserved and won’t be treated as a completed tool-call message.

JKobrynski · 2026-06-22T10:00:53Z

+export interface AppleContextOptions {
+  /**
+   * Keep the first system message and only the last N non-system messages.
+   */
+  rollingWindowMessages?: number
+  /**
+   * Remove completed tool-call/tool-result entries from older context.
+   */
+  dropCompletedToolCalls?: boolean
+}


Handled by updating the rollingWindowMessages docs to match the implementation: system messages are preserved, and the window applies to the latest non-system conversation messages.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

grabbou · 2026-06-17T20:39:23Z

Please have a look at outstanding copilot comments and decide whether to address them or not.

Quick comment:

Add fluent Apple model wrappers for:
summarizeHistory(threshold, model)
rollingWindow(entries)
droppingCompletedToolCalls()

Would that be AI SDK native way, or shall we instead use options via apple() object? For example: apple({ rollingWindow: XX })

I am more familiar with the latter.

grabbou · 2026-06-17T20:42:19Z

Please do a follow-up after this one is merged to #217 - I guess partial support for private cloud compute is done here, but we also need to do support for "reasoning" option, some errors and potentially other things (exact diff to be checked).

JKobrynski · 2026-06-19T09:04:19Z

Would that be AI SDK native way, or shall we instead use options via apple() object? For example: apple({ rollingWindow: XX })
I am more familiar with the latter.

@grabbou that makes sense. I originally went with fluent wrappers because the task description called out that shape and it maps closely to Apple’s foundation-models-utilities composition style.

That said, I agree the options-based API is more consistent with the rest of this package and with how Apple-specific behavior is already configured through apple() / providerOptions.apple.

I’ll update this to expose the history controls through options instead.

JKobrynski · 2026-06-19T12:00:44Z

Please do a follow-up after this one is merged to #217 - I guess partial support for private cloud compute is done here, but we also need to do support for "reasoning" option, some errors and potentially other things (exact diff to be checked).

Sure, will do

JKobrynski · 2026-06-19T12:12:19Z

@artus9033 @grabbou PR updated, feel free to take another look

grabbou · 2026-06-19T17:23:13Z

Would that be AI SDK native way, or shall we instead use options via apple() object? For example: apple({ rollingWindow: XX })
I am more familiar with the latter.

@grabbou that makes sense. I originally went with fluent wrappers because the task description called out that shape and it maps closely to Apple’s foundation-models-utilities composition style.

That said, I agree the options-based API is more consistent with the rest of this package and with how Apple-specific behavior is already configured through apple() / providerOptions.apple.

I’ll update this to expose the history controls through options instead.

Yeah, it's likely the AI research went a bit too far on this one!

artus9033

LGTM from my end after a few comments resolved, great job!

artus9033 · 2026-06-20T13:06:24Z

+function isToolRelatedMessage(message: LanguageModelV3Message) {
+  if (message.role === 'tool') {
+    return true
+  }
+
+  if (message.role !== 'assistant') {
+    return false
+  }
+
+  return message.content.some(
+    (part) => part.type === 'tool-call' || part.type === 'tool-result'
+  )
+}


Let's take a look at this

glibenko · 2026-06-21T04:56:19Z

Thanks for the work on this 👍 - noticed it adds getModelInfo() for model availability/capabilities, which feels like the natural neighbor for language support, so raising this here before opening a separate PR.

Use case: Apple's on-device Foundation Models support a fixed set of locales (~16 as of iOS 27), and notably not all languages. For apps that generate localized output ("respond in the user's language"), there's currently no way to know at runtime whether the model can handle a given locale - so we hardcode Apple's list and keep it in sync manually. Apple exposes this natively via SystemLanguageModel.supportedLanguages doc, but the RN wrapper only bridges isAvailable().

Proposal: expose the supported languages through the provider, e.g.: apple.supportedLanguages() // => string[] of BCP-47 codes, e.g. ['en', 'de', 'zh-Hans', ...] backed by SystemLanguageModel.default.supportedLanguages in AppleLLMImpl.swift, returning [] on unsupported OS versions.

Question: would you prefer this as a standalone method, or folded into the getModelInfo() surface you're adding here (e.g. getModelInfo().supportedLanguages)? Happy to open a PR once you confirm the preferred shape - just want to avoid conflicting with this one. Or i can wait when yours @JKobrynski will be merged and then create a patch.

artus9033 · 2026-06-21T13:32:16Z

Hello @glibenko ! First of all, thanks for the interest in this library. Indeed, it is a sensible design to expose this information. I think you could've missed it in the diffs, but I think this PR already has everything you mean - check here & here. Is that what you had on mind, or are we missing something?

JKobrynski · 2026-06-22T09:49:04Z

@glibenko I tested this on device through the demo app and confirmed that getModelInfo().supportedLanguages is working. The native bridge returned isAvailable: true and 23 supported languages for the system model.

While testing, I noticed the values were initially coming through as Swift debug descriptions because we were serializing them with String(describing:). I updated that to return JS-friendly language/locale identifiers instead, e.g. en-US, fr-FR, zh-CN, so the API is easier to consume from JS. Re-tested on device and confirmed the returned values now look correct. Let me know what you think!

JKobrynski · 2026-06-22T10:04:27Z

@artus9033 @grabbou addressed everything up to this point, feel free to take another look at the PR

artus9033 · 2026-06-22T13:33:22Z

LGTM from my end! @grabbou are we good to merge this one? 🚀

…ode 27 beta

grabbou · 2026-06-30T18:15:23Z

Yeah, you guys go ahead!

grabbou · 2026-06-30T18:19:53Z

+              threshold: appleHistorySummarizationThreshold,
+              model: createAppleProvider({
+                model: modelId,
+              }).languageModel(),


I think this is wrong/could be improved. The options should not take LanguageModelV3 and then:

const modelId = model.modelId as AppleLanguageModelId
in order to re-create it here with "createAppleProvider".
Since this is not taking ANY model, but specific Apple models, perhaps it's better to provide strict typing on modelId (worst case scenario: string), and just pass it down as this?
I am not sure what are the options, but I guess it's Apple-specific.

I also wonder, since this is not a built-in Apple feature, shall we rely on it? I think it would be more appropriate to use a separate utility. And then, we can just pass "model" here (and make this work with any model, including Apple and Llama).

I would leave this in user land to be honest and clean-up the code to only rely on Apple features.

grabbou

A few follow-ups from a deeper pass over the non-history surface (model info / image prompts / Image Playground / PCC). The history-management API design is being tracked separately; these are independent of it.

grabbou · 2026-06-30T19:59:01Z

+      .appendingPathComponent(UUID().uuidString)
+      .appendingPathExtension(fileExtension)
+
+    try imageData.write(to: fileURL, options: .atomic)


Temp-file leak. This writes the decoded image bytes to the temp directory but nothing ever deletes the file, so every inline-data image prompt — and every Image Playground source image, since createImagePlaygroundConcepts routes through createImageURL too — leaks a temp file for the lifetime of the app. iOS only purges tmp opportunistically, so a chat that sends images repeatedly will keep growing disk usage.

Suggested path: delete the file once the underlying call has consumed it. Simplest is to collect the URLs created for a request and defer { try? FileManager.default.removeItem(at: url) } after session.respond(...) / creator.images(...) returns. Alternatively write into a per-request subdirectory (temporaryDirectory/<uuid>/) and remove the whole directory at the end.

grabbou · 2026-06-30T19:59:01Z

+#endif
+
+  private func availabilityString(_ availability: Any) -> String {
+    return String(describing: availability)


availability leaks a Swift debug string. String(describing:) of the availability enum reaches JS as something like unavailable(reason: ...) — the same class of issue already fixed for supportedLanguages in this PR. AppleModelInfo.availability: string ends up opaque and unstable across OS versions, so consumers can't reliably switch on it.

Suggested path: map the cases explicitly to stable identifiers, e.g.

switch availability { case .available: return "available" case .unavailable: return "unavailable" }

If the unavailable reason matters, surface it as its own structured field rather than embedding it in a describe-dump.

grabbou · 2026-06-30T19:59:01Z

+      "isAvailable": model.availability == .available,
+      "availability": availabilityString(model.availability),
+      "contextSize": contextSize,
+      "quotaUsage": String(describing: model.quotaUsage),


quotaUsage is a describe-dump. String(describing: model.quotaUsage) reaches JS as the type's debug description (e.g. QuotaUsage(...)) rather than a usable value — same problem as availability and the already-fixed supportedLanguages.

Suggested path: pull the concrete fields off quotaUsage and return a structured object (or a single normalized number/ratio) the JS side can consume, and type quotaUsage accordingly in AppleModelInfo. If the shape isn't stable enough to model yet, I'd drop it from the payload for now rather than ship an opaque string.

grabbou · 2026-06-30T19:59:01Z

+      "quotaUsage": String(describing: model.quotaUsage),
+      "supportsLocale": model.supportsLocale(locale),
+      "supportedLanguages": model.supportedLanguages.map(languageIdentifier).sorted(),
+      "supportsTokenCounting": false,


Confirm intent: supportsTokenCounting is hardcoded false for the PCC model while the system model reports true on 26.4+. Intentional (PCC genuinely can't count tokens), or a copy-paste oversight? Consumers will branch on this. If it's deliberate, a one-line comment here noting why would save the next reader the same double-take.

Good catch. PCC is not actually working/buildable in the current setup, because the SDK we’re compiling against does not expose the required Swift symbols yet. I’m going to leave the PCC implementation unsupported for now rather than add a workaround or pretend the capability is ready.

Once we figure out the broader PCC story and have an SDK surface we can build against reliably, we should revisit this and make the model info shape explicit there. @artus9033 is going to help me with this, I will let you know when we have something!

Yes, this is a problem we are aware of. I'll try experimenting with it. Worst case, we can cut off backwards compat from the new release or require users to install with an env var.

grabbou · 2026-06-30T19:59:01Z

+  return {
+    type: 'image',
+    mediaType,
+    url: value,


Remote-URL handling is inconsistent between the two image paths. This returns any non-data: string as a url attachment, including http(s)://, which then fails later in native with the generic "Image attachment URLs must be local file URLs". The image-generation path (prepareImageModelFiles, lines 196-210) already rejects remote URLs early in JS with a clear message — that was the Copilot fix — but this image-prompt path was missed.

Suggested path: add the same early guard here: reject http:///https:// with a clear "remote URLs not supported, provide a local file or data URL" error, and only pass through data:, file://, and absolute local paths. Better still, factor the URL/data normalization into one shared helper used by both prepareImageAttachment and prepareImageModelFiles so the two paths can't drift again.

grabbou · 2026-06-30T19:59:01Z

+  supportedLanguages: string[]
+  supportsTokenCounting: boolean
+  supportsImagePrompts: boolean
+  supportsPrivateCloudCompute: boolean


Drop supportsPrivateCloudCompute entirely. It's redundant — PCC availability is already answerable via getModelInfo({ model: 'private-cloud-compute' }).isAvailable when the proper model id is used. As implemented it's also misleading: the system model always reports false (native modelInfo(for: SystemLanguageModel...)) even on iOS 27 where PCC is available, because the flag describes the instance, not the OS capability.

Suggested path: remove this field from AppleModelInfo, and remove the supportsPrivateCloudCompute entries from both modelInfo(...) builders in AppleLLMImpl.swift. Consumers determine PCC support by querying the PCC model id directly.

grabbou · 2026-06-30T19:59:01Z

+    return createLanguageModel(options)
  }
  provider.isAvailable = () => NativeAppleLLM.isAvailable()
+  provider.getModelInfo = (options: { locale?: string; model?: string } = {}) =>


Tighten the param type. model?: string is looser than the rest of the surface, which uses AppleLanguageModelId (and AppleModelInfo.model is already AppleLanguageModelId).

Suggested path: type it as { locale?: string; model?: AppleLanguageModelId } so callers get autocomplete and an invalid id is caught at compile time instead of failing in native with Unsupported model '...'.

This is a good suggestion

grabbou and others added 4 commits June 9, 2026 11:25

Add WWDC26 Apple Intelligence APIs

d0beff9

feat: add history management wrappers

e660a07

feat: add apple history demo controls

8de0a24

docs: document history management wrappers

77538f1

JKobrynski marked this pull request as ready for review June 17, 2026 14:07

artus9033 requested review from artus9033, Copilot and grabbou June 17, 2026 14:17

Copilot started reviewing on behalf of artus9033 June 17, 2026 14:17 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

artus9033 assigned JKobrynski Jun 17, 2026

artus9033 changed the title ~~feat: Add WWDC26 Apple Intelligence APIs~~ feat: add WWDC26 Apple Intelligence APIs Jun 17, 2026

Apply suggestions from code review

81ab6e8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

grabbou mentioned this pull request Jun 17, 2026

[codex] Add WWDC26 Apple Intelligence APIs #215

Closed

JKobrynski added 2 commits June 19, 2026 11:44

feat: expose history context options (address comment)

92284f6

fix: guard tool pruning for text messages

492ba46

fix: handle image urls and tool parts safely

ea1916c

artus9033 approved these changes Jun 20, 2026

View reviewed changes

feat: serialize supported languages as identifiers

283c9b2

fix: use jpeg extension for jpeg images

95ea49c

docs: clarify availability guidance

38984e6

artus9033 changed the title ~~feat: add WWDC26 Apple Intelligence APIs~~ feat: WWDC26 Apple Intelligence APIs Jun 24, 2026

artus9033 added 4 commits June 25, 2026 00:24

chore: update bun.lock

0395aea

fix: compilation on iOS - force iOS deployment target to work with XC…

abb477c

…ode 27 beta

fix: make modelInfo async to await model.contextSize

67a8d66

fix: async and throwing problem in modelInfo

0759cd7

grabbou requested changes Jun 30, 2026

View reviewed changes

grabbou reviewed Jun 30, 2026

View reviewed changes

JKobrynski added 3 commits July 1, 2026 10:35

refactor: remove provider-owned history helpers

979ad35

fix: address model info and image handling review comments

b1c40de

feat: add generic history management helpers

4bbaff5

Uh oh!

Conversation

JKobrynski commented Jun 17, 2026

Summary

Changes

Testing

Uh oh!

vercel Bot commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grabbou commented Jun 17, 2026

Uh oh!

grabbou commented Jun 17, 2026

Uh oh!

JKobrynski commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JKobrynski commented Jun 19, 2026

Uh oh!

JKobrynski commented Jun 19, 2026

Uh oh!

grabbou commented Jun 19, 2026

Uh oh!

artus9033 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

glibenko commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

artus9033 commented Jun 21, 2026

Uh oh!

JKobrynski commented Jun 22, 2026

Uh oh!

JKobrynski commented Jun 22, 2026

Uh oh!

artus9033 commented Jun 22, 2026

Uh oh!

grabbou commented Jun 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grabbou Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grabbou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JKobrynski commented Jun 19, 2026 •

edited

Loading

glibenko commented Jun 21, 2026 •

edited

Loading

grabbou Jun 30, 2026 •

edited

Loading