You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds an opt-in mechanism that allows Rill’s AI agents to trigger approved UI actions in the browser.
Adding capability to support ejection of Rill templates to vega lite spec. (#9850)
UI elements can expose themselves to the agent using a stable data-ai-action identifier. Available actions are sent as transient context with each chat request, and the agent can invoke the new click_ui tool to request an action.
Implementation
Added the click_ui backend tool for developer and analyst agents.
Added UIContext and UIAction to the completion API.
Discover visible, enabled, uniquely identified [data-ai-action] elements when sending a chat message.
Revalidate actions immediately before clicking them in the browser.
Added initial actions for:
Editing a canvas
Starting a dashboard pivot
Added frontend and backend tests for action discovery, validation, and execution.
Safety
UI actions are explicitly opt-in. The agent cannot use arbitrary CSS selectors or click arbitrary page elements.
An action is available only when it:
Has a non-empty data-ai-action identifier.
Is visible and enabled.
Has a unique identifier on the current page.
Is outside a data-ai-ignore subtree.
Is included in the current request’s UI context.
The action is checked again in the browser before the click is performed.
Checklist:
Covered by tests
Ran it and it works as intended
Reviewed the diff before requesting a review
Checked for unhandled edge cases
Linked the issues it closes
Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
The browser clicks before the server validates the action ID.Session.Call appends and streams the CALL message (runtime/ai/ai.go:980) before invoking the tool handler (runtime/ai/ai.go:1022), and the frontend runs onCall on every streamed CALL. If ClickUI.Handler then rejects the ID because it was not in the request's UIContext — the page changed after the message was sent, or the ID was a duplicate at send time and is unique now — the click has already happened as long as clickUIAction finds one DOM match. The "included in the current request's UI context" guarantee is only enforced after the fact. Clicking from onResult on a non-error result, or checking membership against the sent snapshot before clicking, would close this. web-common/src/features/chat/core/messages/tools/tool-registry.ts:165
dashboard.start-pivot is not reachable on the default explore view. The tag is on StartPivotButton.svelte, which only DimensionHeader.svelte:163 and TDDHeader.svelte:250 render. The default layout has its own inline Start pivot button with no data-ai-action at web-common/src/features/dashboards/time-series/MetricsTimeSeriesCharts.svelte:357. On a freshly opened explore getAvailableUIActions() returns no pivot action; once a dimension table is expanded, "start a pivot" hits startPivotForDimensionTable rather than the time-series variant.
ClickUI.CheckAccess is untested. The user-agent and empty-UIContext gates are what keep this tool out of MCP clients and out of requests with no browser snapshot, but TestClickUI only covers the path where both hold. runtime/ai/click_ui_test.go:11
Two smaller things: click_ui has no entry in tool-icons.ts:39, so the block renders with the generic Search icon; and the branch now conflicts with main in developer_agent.go, ChatInput.svelte and types.ts — mechanical, but keep beforeFork when re-applying the uiContext change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in mechanism that allows Rill’s AI agents to trigger approved UI actions in the browser.
Adding capability to support ejection of Rill templates to vega lite spec. (#9850)
UI elements can expose themselves to the agent using a stable
data-ai-actionidentifier. Available actions are sent as transient context with each chat request, and the agent can invoke the newclick_uitool to request an action.Implementation
click_uibackend tool for developer and analyst agents.UIContextandUIActionto the completion API.[data-ai-action]elements when sending a chat message.Safety
UI actions are explicitly opt-in. The agent cannot use arbitrary CSS selectors or click arbitrary page elements.
An action is available only when it:
data-ai-actionidentifier.data-ai-ignoresubtree.The action is checked again in the browser before the click is performed.
Checklist: