Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
203 changes: 203 additions & 0 deletions generalized-tool-calling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
# WebMCP: A generic tool calling primitive

Authors: Dominic Farolino & Khushal Sagar

This explainer contains three design updates to the WebMCP spec, prompted by Mozilla's feedback
collected in [issue #236](https://github.com/webmachinelearning/webmcp/issues/236) and
[issue #237](https://github.com/webmachinelearning/webmcp/issues/237).

## Generic tool namespace on `WindowProxy`

At TPAC, the group made a few resolutions, recorded in the
[minutes](https://www.w3.org/2026/10/27-webmachinelearning-minutes.html#9330).

- Split the API into two halves:
- Provider (`registerTool()`, `toolactivated`, `toolcancel`): only same-origin contexts can use
it.
- Consumer (`getTools()`, `executeTool()`, `toolchange`): both same-origin and cross-origin
contexts can use it. `exposedTo` further controls which origins can see each tool.
- Expose these APIs to Documents through `WindowProxy`. This means only Documents in the same
browsing context group can reach them.

One question still open is naming. [Issue #236](https://github.com/webmachinelearning/webmcp/issues/236)
asks that the API become a general communication primitive rather than an LLM-specific feature. We
propose the name `tools` instead of `modelContext`, because tools are currently the only primitive
in the spec.

The other question is exactly how the interface is exposed through `WindowProxy`. The options are
below.

1. Two interfaces: window.toolRegistry + window.tools (preferred)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was arguing for (2) at TPAC, but I've been convinced this is right. I'd rename it to toolManager rather than toolRegistry personally (managers having writing power makes sense, whereas I might try go to a registry to read all of the tools). But, I defer naming if you've thought hard on it already.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think toolRegistry happened because were thinking about customElementRegistry when discussing this with jake. i'm good with toolManager as well.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more we looked at this, the more problematic it seemed. @khushalsagar is going to publish a deeper explainer describing why, but the gist is: if we introduce a new object on WindowProxy that is supposed to participate in CrossOrigin access, we end up in a bad situation where:

const promise = iframe.contentWindow.anything.getTools();
// `promise` is constructed in the iframe's realm, so if the iframe navigates
// literally anywhere, `promise` dies.

and similarly, if we introduce anything stateful, like an event handler:

iframe.contentWindow.ontoolchange = e => { ... };
// Or:
iframe.contentWindow.toolRegistry.ontoolchange = e => {};

We similarly run into lifetime problems where, for same-origin iframes, the state is tied to their realm, and kills the parent's event listener without the parent ever explicitly knowing this.

@khushalsagar is going to publish either an explainer, or an amendment to this section, describing our options here.


```js
await window.toolRegistry.registerTool(...);
// works cross-origin.
const theirs = await window.opener.tools.getTools();
// throws "SecurityError"
window.opener.toolRegistry;
```

`toolRegistry` is the provider half. It is an ordinary Window member and same-origin only. tools
is the consumer half which is added to CrossOriginProperties(Window). Every member of Tools is safe
to use cross-origin. This follows the TPAC resolution directly with naming updates. This option
best aligns with HTML's existing cross-origin model. Through a cross-origin WindowProxy, a
caller can reach primitives and functions created in the caller's realm. Every Tools member is
safe to use cross-origin, so the tools getter can work like the cross-origin functions. It
returns a Tools handle that is created in the caller's realm and bound to the target window.
Nothing from the target realm reaches the caller, so Tools needs no exotic behavior, and results
and toolchange events are delivered in the caller's realm. Worker and ServiceWorker objects
already follow this handle model, so one Tools interface works for both windows and workers.

2. Single window.tools interface

```js
await window.tools.registerTool(...);
// works cross-origin
const theirs = await window.opener.tools.getTools();
// throws "SecurityError"
window.opener.tools.registerTool(...);
```

One `Tools` object holds both halves, and only the consumer members work cross-origin. Tools
would become a third exotic object with a per-member split, after WindowProxy and Location. It
would need Location-style overrides of the internal methods, which HTML and WebIDL discourage
and engines support only for Window and Location. Because the object lives in the target realm,
toolchange can only be same-origin. Worker and ServiceWorker objects would still need a
consumer-only interface.

3. Direct members on window
- All methods go directly on Window (`otherWindow.getTools()`), and each method's cross-origin
access depends on which half it belongs to. This adds each consumer method to
`CrossOriginProperties(Window)` one by one, and it mixes both halves on the global.
`toolchange` would fire on Window in the target realm, so it has the same problem as in
option 2. Worker and ServiceWorker would need the methods added to them directly, so there
is no single interface for windows and workers.


## Tool exposure to external agents

To make tool registrations useful for both web callers, we propose a design for separating the tool
parameters required for web callers, from those required for the AI use case and other natural
language consumers.

```js
window.tools.registerTool({
name: "search_flight",
naturalLanguageProperties: {
description: "Provides real-time flight search data with the given search constraints",
inputSchema: <foo>, // Possible required.
outputSchema: <foo>, // Possible required.
annotations: {...} // Optional.
},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that naturalLanguageProperties are just "properties" as they are not specific to "natural language consumers".

For example: annotations are not concern of a model, they are concern of a harness. Similarly the outputSchema.

I also think about cases when we will have debugging tools, for automation, and I need inputSchema and outputSchema mostly and there might be no natural language consumer in picture.

If you agree with above I would generalize "To make tool registrations useful ..." content. In current form it reads to me as naturalLanguageProperties are SPECIFIC to AI Agents and even more - natural language consumers (reads as models (Decision models, LLMs)?)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think about cases when we will have debugging tools, for automation, and I need inputSchema and outputSchema mostly and there might be no natural language consumer in picture.

can you describe how this use-case would work more. I rationalized to myself that its only AI use-cases that can read a natural language description of an API and its parameters at runtime and invoke them. That's why parameters which are necessary for an AI system to make use of an API should be bucketed under a naturalLanguageProperties.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be possible to build a deterministic IFTTT system connecting outputs from one tool to the input of other using schemas provided by the tool developer to design those connections (I haven't seen this in practice, but would theoretically be possible).

My .02 cents is that naturalLanguageProperties describes who the information is for (agents) rather than what the information is (tool metadata), and describing what it is makes usage less prescriptive.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, at the moment we validate input against the inputSchema so I feel like naturalLanguageProperties is slightly misleading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what the ask here is, sorry.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bikeshedding sorry - arguing that inputSchema is a better name than naturalLanguageProperties (or at the very least, that naturalLanguageProperties doesn't feel like a good fit).

@domfarolino domfarolino Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow. This doesn't look right:

registerTool({
  name: 'my-tool',
  inputSchema: { // Used to be `naturalLanguageProperties`
    description: "This is",
    annotations: {readOnlyHint: true},
    actualInputSchema: {type: "object", ...}
  }
});

The idea is that nLP is a bag of properties designed for model consumption. But perhaps you mean flattening the above, and just hoisting properties like description, annotations, etc., into the outer tool definition and just making inputSchema mandatory and not the others. But then what do we do with description, etc.

exposedTo: ["https://origin.com"],
execute: () => {...}
});
```

The inclusion of `naturalLanguageProperties` directs the browser and other web consumers to expose
the tool to native or external agents via whatever native mechanism or protocol, while `exposedTo`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear if exposedTo behavior is changing from what we currently in WebMCP spec.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this states that it does. And I agree with the direction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behaviour of exposedTo remains unchanged. It controls which execution contexts have visibility into a tool.

But your comment makes me think that there are 2 concepts here which are combined and its worth spelling that out, in case there's a better way to approach this in the API:

  1. The author needs to be able to say, "this tool is just an API between my same-origin frames or between me and other.com, its not for any agent. Don't show it to the browser agent".
  2. For a tool to be usable by the browser agent (or any agent), it needs to have these natural language properties.

The API is combining these concepts by saying that the presence of naturalLanguageProperties implies the tool is meant for the browser agent.

An alternative we considered was a separate value in exposedTo called "native-agent", "browser" or "user-agent" to say that this tool should be exposed to consumers outside the web platform. And say that if these consumers are using AI models then they also need the fields in naturalLanguageProperties. So now an author needs to set the bit in exposedTo and naturalLanguageProperties to have the tool be added to the browser's native AI agent.

We didn't go for this because the external consumer use-case is limited to AI consumers but I can see that splitting the 2 is more forward compatible.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One approach to splitting these use cases is to add a private boolean to the tool annotations. This gives authors three options:

  • No naturalLanguageProperties: private API between frames, not useful for any sort of agent.
  • naturalLanguageProperties: Tool exposed to other frames (gated by exposedTo) and browser agents (the default).
  • naturalLanguageProperties + naturalLanguageProperties.annotations.private = true: Tool for use by site agents only (gated by exposedTo).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we've recently resolved to:

  1. Make the permission policy only control exposure "outside" the browser, to native agents / the user agent. This is inline with how permission policy works in general—access to powerful features.
  2. exposedTo (and fromOrigins) will only control the origin rules for tool exposure on the web platform. Tools will be able to be registered in, and called from, any origin that has appropriate WindowProxy access, and is permitted by the origins in these two lists, regardless of the permission policy.
  3. Natural language properties will NOT double as an exposure primitive. It will not control exposure to native/user agent or origins, it's solely an annotation that is useful for tools fed into agents, but not required for "better postMessage() tools that solve the 'comlink' use case".

@khushalsagar @bvandersloot-mozilla, does this sound right to you?

continues to remain for controlling exposure to web consumers, by origin.

## Structured cloneable tool inputs/outputs

WebMCP tool inputs and outputs are currently constrained to JSON. This is reasonable for exclusively
model-facing tools, but it's an unnatural restriction for a generic Web-to-Web tool invocation primitive.
JavaScript and the Web platform already have a richer value model for crossing realm boundaries:
[structured serialization](https://html.spec.whatwg.org/C#serializable-objects). We propose allowing WebMCP
tools to accept as input, and return as output, the structured-cloneable types on the web platform,
such as `Date`, `Map`, `Set`, `Blob`, `File`, `ArrayBuffer`, and `ImageData`. For example, a tool
should eventually be able to express a contract conceptually like:

```js
window.tools.register({
name: "set-profile-picture",
naturalLanguageProperties: {
description: "Set the user's profile picture.",
inputSchema: {
type: "object",
properties: {
image: { /* ImageData */ }
},
required: ["image"]
}
},
execute({ image }) {
// `image` is an official ImageData DOM object.
}
});
```

The exact schema syntax for declaring Web types remains an open question. Possibilities include
extending the existing `type` vocabulary or introducing a WebMCP-specific schema keyword.

### Web callers use Web values

When one Web context invokes a tool exposed by another, values would cross the boundary using
structured-clone semantics rather than being forced through JSON.

A tool accepting a `Date` receives a `Date`; a tool returning an `ImageData` returns an
`ImageData`. Authors should not need to create a second, JSON-shaped representation of their
application's data merely because the same primitive can also be exposed to an agent.

This is especially important if `window.tools` is intended to become a generally useful
cross-realm invocation primitive rather than an API whose semantics are intrinsically tied to LLMs.

### Agents see a standardized JSON lowering

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to flag that it might be feasible to defer this question in the short term without loss of function. Or, to incrementally expand support, at least.

A tool that exposes a JSON-native interface (null, bool, number, dictionary, array) could be made usable without really grappling with this issue. It's only when you want to pass objects that don't have a clean JSON.parse/.serialize round trip that you need to contend with this. Narrower than that really: when the transformation doesn't work in the direction you care about; passing an input object that has a toJSON probably works fine.

That doesn't mean that this isn't worth doing, but I'd treat this as something incremental rather than a core prerequisite. Whatever isn't solved now will mean it is not accessible to agents... for now.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, let me see if I understand what you're saying correctly. Right now, tools don't really distinguish between whether they're built for agents or not. So I think you're recommending one of the following:

  1. Define the subset of all web-native-serializable types that have an obvious JSON representation, and support only those types as tool inputs/outputs, for all agents and web callers. This means, initially, tools can consume and output ImageData DOM objects, but not complex JavaScript objects that contain cycles, or Map JavaScript types, since they don't have an obvious JSON representation for us to define yet.

    • Consider this tool:
    registerTool({
      name: 'external-tool',
      exposedTo: ['user-agent', 'https://parent.origin'],
      executeTool() {
        const obj = {};
        obj.a = obj;
        return obj;
      }
    });
    • BiDi: When a BiDi caller invokes it, it fails because the output has no obvious JSON serialization, which we're deciding is the format we expose tool inputs/outputs over BiDi. (This object is web-serializable, but there's no way to express it to a BiDi caller without leaking the browser's internal serialization representation over BiDi, thus turning it into part of an API contract.)
    • Web: When the parent frame calls this tool, it also fails, because we keep web caller capabilities in sync with BiDi/native caller capabilities. There is no technical reason to do this besides consistency, since obj is otherwise perfectly web-serializable and web-deserializable. Preventing this just keeps authors from writing tools that work for one consumer and not the other.
  2. Tools can use any serializable / structured cloneable type as input and output. For web callers, all types will work fine. But for "native" callers over BiDi (not necessarily LLMs), we'll enforce JSON serialization at the boundary, since that's the format we're choosing to expose things over BiDi. Consider the above tool in (1):

    • BiDi: Calling it would fail, since there is no obvious JSON lowering/seralization
    • Web: When the parent frame calls the tool, it works fine, according to the web's native serialization / deserialization steps.
  3. Require naturalLanguageProperties to exist for tools exposed to BiDi / browser-external callers. In that case, the tool above would never get exposed to BiDi. To get a tool exposed to BiDi, you have to explicitly provide naturalLanguageProperties which opts you into a new, tighter input/output type surface area:

    • naturalLangaugeProperties.{input,output}Schema only supports JSON types + the subset of web-serializable types that we manually decide have an obvious JSON representation
    • Input/output for these tools works the same for web callers and BiDi callers, since the tool author has opted into this tighter in/out type surface area.
    • Downside: it's impossible to expose tools to BiDi without naturalLanguageProperties. Maybe that's OK?

Probably options (3) is the simplest, and closest to what I think we discussed at TPAC. @martinthomson are you OK with us tentatively resolving in that direction?


Agents external to the browser still require JSON-compatible tool schemas and arguments.
For those callers, the user agent would expose a lowered version of such schemas and arguments.

Each supported Web type would have a standardized mapping to:

- a valid model-facing JSON Schema;
- serialization steps from the Web value to that representation; and
- deserialization steps from model-produced JSON back to the Web value.

For example, a `Date` might be exposed as a standardized date-time string, a `Map` might lower
to an array of key/value pairs, and an `ImageData` might be a simple JSON-ified version of its
[serialization record](https://html.spec.whatwg.org/C#the-imagedata-interface:serialization-steps).

The "JSON lowering" would be specified by WebMCP, and apply to all `[Serializable]` types, rather
than being an implementation-defined browser wire format that browsers actually use under the hood
to represent serialization records. This allows third-party agent harnesses to understand the
advertised representation and avoids browser-specific encodings becoming de facto protocols.
Comment on lines +160 to +163

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this framing for this.

A model will want to invoke tools, which means that inputs might need to have fromJSON methods and outputs can use toJSON. That means lowering on the outputs, but lifting (?) on the inputs, right?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. For inputs to a tool, the model will generate JSON which will be converted to a web type when invoking the tool.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A subtlety worth mentioning: this doesn't imply that the serialized JSON is added to a model's context exactly. Its meant to be an interoperable contract between the browser and agent harnesses, the format should err towards reducing the payload size. The agent harness is responsible for optimally adding the JSON to the model's context and ensuring its semantics are preserved.

For example, if the tool output is this:

{
  products: [
    { id: "sku-1", thumbnail: await createImageBitmap(lampImg) },
    { id: "sku-2", thumbnail: await createImageBitmap(chairImg) },
  ],
};

The serialized JSON could be:

{
  "products": [
    {
      "id": "sku-1",
      "thumbnail": { "mimeType": "image/jpeg", "data": "/9j/4AAQSkZJRgABAQ..." }
    },
    {
      "id": "sku-2",
      "thumbnail": { "mimeType": "image/jpeg", "data": "/9j/4AAQSkZJRgABAQ..." }
    }
  ]
}

The agent harness can show this to the model in any way as long as it understands which thumbnail is associated with which product id.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For efficiency reasons, the harness will decode the base64 image data and present it in a form that is most efficient to transport and tokenize before presenting it to the model. Most likely involving multiple transformations into sRGB, then model-specific pixel tensors/embeddings.

So there's no particular reason to assume that the browser/harness interface will be pure JSON, or JSON + some other data attached to the payload. I think that will be sorted out in future standardization efforts around browser/agent interfaces.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there's no particular reason to assume that the browser/harness interface will be pure JSON, or JSON + some other data attached to the payload. I think that will be sorted out in future standardization efforts around browser/agent interfaces.

I'm not sure about this part. I expect the browser will present everything outwards, interoperably through JSON, since basically all models expect this. Who knows, maybe it could be legitimate MCP format... But then from there, harnesses do whatever they want with the data.


### Better error handling

[Issue #282](https://github.com/webmachinelearning/webmcp/issues/282) asks for a better, structured,
MCP-like way of signaling a userland "error" to agents. While we could specify a special
`isError: true`-shaped JSON object, like MCP, that tools can return to the agent, the proposal to
expand the allowable types to all structured cloneable types gives us an opportunity to express
that same information to agents in a more ergonomic, "webby" way.

Specifically, since DOMException and the native JavaScript `Error` types are both "serializable",
their standardized JSON lowering can include all of the relevant information that the error object
contains, packaged up into an appropriate JSON representation derived from the serialization. The
serialization will exclude any internal details (like stack traces).
Comment thread
domfarolino marked this conversation as resolved.
Resolved

> [!WARNING]
> It's an open question as to whether we want to simply return userland errors
> exactly as they are to callers, or wrap all of them in a new error interface
> that's not constructible from JavaScript, like `ToolError` or
> `ToolRuntimeError`. The only benefit of this would be to make it impossible
> for a userland tool error from spoofing one of the errors generated by the
> browser when a tool fails for reasons outside of the tool implementation's
> control.

### Open design questions

The detailed proposal will need to answer several issues, including:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another one that is pretty important: tool coalescing and permissions policy.

Right now, tools can only be registered by frames that have permission and are automatically included in the top's getTools result. This is great for agent-intended tools, but makes much less sense for a message event equivalent.

I think we need to change the meaning of the permission granted from "can register tool" to "tool coalesces upward." This preserves the security property of preventing top level tool-space pollution while allowing generic tools. Maybe rename the policy key from "tools" to "include-tools" to make it a little clearer.

It may also make more sense to have getTools just give the subtree's coalesced tools, moving the current behavior to window.top.tools.getTools.

With both of these together, we can look at this case:

<body> <!-- top -->
 <iframe id="frameDisallowFoo" allow="tools 'none'">
   <!--registers tool foo -->
  </iframe>
 <iframe id="frameAllowBar" allow="tools">
   <!--registers tool bar -->
  </iframe>
  <script>
   <!--registers tool baz -->
  </script>
</body>

Currently in the spec,document.modelContext.getTools() would return bar and baz. And we still get that from top.tools.getTools() here! But now the top frame can also call frameDisallowFoo.tools.getTools() and get foo.

Without this change, there is no way to use this API as an RPC primitive with a frame and also not trust that frame to publish tools on your window.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exact API for reading tools aside (because that's changing with #227), there's 2 fundamental constraints that an embedder can impose on an iframe:

  1. Can the iframe expose its tools to other Documents in the same browsing context group and the browser agent.
  2. Can the iframe read tools from other Documents in the same browsing context group.

Right now the tools policy is a boolean control on both: without the policy the embedded Document can't register any tools (which implicitly means it can't expose tools to any Document) and it also can't read tools from any other Document. We should consider a couple of things for simplifying this:

a. Should tool exposure and tool read be separate permissions? I think yes. Because its a common case to want your tools to only be exposed to a trusted agent (like the browser agent), so you want to prevent the iframe from reading it. But you're ok with the iframe also exposing tools to the browser agent for the functionality its adding to your side.

b. Is the current boolean syntax good enough or should the permission policy mirror the syntax for exposedOrigins? This one I'm not sure about. I think we can start with a bool and add support for exposedOrigins without any compat issues.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's 2 fundamental constraints that an embedder can impose on an iframe:

I disagree. I think exposing tools ban be split into two modes (a) included in a direct getTools on that window or (b) included in a getTools on an ancestor frame.

a. Should tool exposure and tool read be separate permissions?

Yes, and I believe read shouldn't require a permission at all unless we have a concrete use case.

b. Is the current boolean syntax good enough

If you mean permissions policy, they are not boolean. See MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Permissions_Policy#allowlists.


- the syntax used to declare Web types in what is otherwise JSON-only schemas;
- whether containers such as Map and Set declare their key/value/element types, or how
otherwise-unconstrained nested structured-cloneable values are represented;
- how cycles and shared references are handled;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't we talk about the idea that maybe the values returned by getTools() is transferable as well? I realize that this could add a layer of complexity, but it makes things much, much more useful.

Maybe your intention here was to cover that with this comment, but I thought I'd flag it here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, or at least serializable. You should be able to do:

const [tool] = await getTools();
self.parent.postMessage(tool, "*");

This needs some more discussion about what happens if tool is only exposedTo the self's origin, and not self.parent's origin, and how that should work. I do want to make these objects movable in some fashion, however I think it can come some later, and becomes more relevant when design for workers next.

However, this is not what I was trying to capture here. This line in the explainer is mostly trying to capture something that's related to #375 (comment). Basically: if we want to support structured cloneable tool inputs/outputs, how should that work for callers/consumers of tools that only speak JSON? How can we create a JSON representation of objects that are fundamentally not JSON serializable? ImageData might be obvious, but Map and arbitrarily complicated, self-referential JavaScript objects are not obvious without exposing the browser's internal serialization representation over JSON, which seems bad. Hence I like option (3) in my linked response, where we only define serializations for types that do have obvious JSON serializations, or that at least JSON-serialize cleanly.

Given that, do you think we can resolve this comment and move discussion to the one below?

- the exact JSON representation of serializable binary types like files,
images, etc.;
- payload-size limits and whether browsers may transform large values;
- the initial allowlist of supported types; and
- versioning of standardized lowerings.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than say "versioning", I'd prefer that this was about how it might be possible to evolve how things are transformed into a model-compatible form.

We discussed versioning at some length when we met and it seems like there was a bit of a disconnect about expectations. None of that made it into this document, which is good, but I'd like to keep that solid.

One way to manage changes in serialization is the same way you would manage a new, incompatible version of a tool: define a new tool. If the tool takes a different type of object, then that different type can have different transforms to and from JSON.


These details are intentionally deferred from this joint explainer. A separate explainer will specify
the structured-cloneable type system, model-facing lowerings, prior art, security considerations, and
per-type conversion algorithms in detail.