Skip to content

feat: enable UI actions from AI chat - #9881

Open
djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai
Open

djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai

Conversation

@djbarnwal

Copy link
Copy Markdown
Member

Adds an opt-in mechanism that allows Rill’s AI agents to trigger approved UI actions in the browser.

Adding capability to support ejection of Rill templates to vega lite spec. (#9850)

UI elements can expose themselves to the agent using a stable data-ai-action identifier. Available actions are sent as transient context with each chat request, and the agent can invoke the new click_ui tool to request an action.

Implementation

  • Added the click_ui backend tool for developer and analyst agents.
  • Added UIContext and UIAction to the completion API.
  • Discover visible, enabled, uniquely identified [data-ai-action] elements when sending a chat message.
  • Revalidate actions immediately before clicking them in the browser.
  • Added initial actions for:
    • Editing a canvas
    • Starting a dashboard pivot
  • Added frontend and backend tests for action discovery, validation, and execution.

Safety

UI actions are explicitly opt-in. The agent cannot use arbitrary CSS selectors or click arbitrary page elements.

An action is available only when it:

  • Has a non-empty data-ai-action identifier.
  • Is visible and enabled.
  • Has a unique identifier on the current page.
  • Is outside a data-ai-ignore subtree.
  • Is included in the current request’s UI context.

The action is checked again in the browser before the click is performed.

image

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@djbarnwal
djbarnwal requested review from AdityaHegde and begelundmuller and removed request for begelundmuller September 11, 2026 14:13
@nishantmonu51 nishantmonu51 added Type:Feature New feature request Size:XL Very large change: 2,000+ lines labels Sep 16, 2026
@nishantmonu51

Copy link
Copy Markdown
Collaborator

The browser clicks before the server validates the action ID. Session.Call appends and streams the CALL message (runtime/ai/ai.go:980) before invoking the tool handler (runtime/ai/ai.go:1022), and the frontend runs onCall on every streamed CALL. If ClickUI.Handler then rejects the ID because it was not in the request's UIContext — the page changed after the message was sent, or the ID was a duplicate at send time and is unique now — the click has already happened as long as clickUIAction finds one DOM match. The "included in the current request's UI context" guarantee is only enforced after the fact. Clicking from onResult on a non-error result, or checking membership against the sent snapshot before clicking, would close this. web-common/src/features/chat/core/messages/tools/tool-registry.ts:165

dashboard.start-pivot is not reachable on the default explore view. The tag is on StartPivotButton.svelte, which only DimensionHeader.svelte:163 and TDDHeader.svelte:250 render. The default layout has its own inline Start pivot button with no data-ai-action at web-common/src/features/dashboards/time-series/MetricsTimeSeriesCharts.svelte:357. On a freshly opened explore getAvailableUIActions() returns no pivot action; once a dimension table is expanded, "start a pivot" hits startPivotForDimensionTable rather than the time-series variant.

ClickUI.CheckAccess is untested. The user-agent and empty-UIContext gates are what keep this tool out of MCP clients and out of requests with no browser snapshot, but TestClickUI only covers the path where both hold. runtime/ai/click_ui_test.go:11

Two smaller things: click_ui has no entry in tool-icons.ts:39, so the block renders with the generic Search icon; and the branch now conflicts with main in developer_agent.go, ChatInput.svelte and types.ts — mechanical, but keep beforeFork when re-applying the uiContext change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Size:XL Very large change: 2,000+ lines Type:Feature New feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants