Skip to content

templates: upgrade Yutori templates from n1.5 to n2 #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dprevoznik wants to merge 2 commits into
base: main
Choose a base branch
from
Open

templates: upgrade Yutori templates from n1.5 to n2 #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e2b3ba2
templates: upgrade Yutori templates from n1.5 to n2
dprevoznik Aug 26, 2026
cf32041
templates: fix dollar-token expansion in the n2 edit tool
dprevoznik Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions pkg/create/templates.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ var Templates = map[string]TemplateInfo{
Languages: []string{LanguageTypeScript, LanguagePython},
},
TemplateYutoriComputerUse: {
Name: "Yutori n1.5 Computer Use",
Description: "Implements a Yutori n1.5 computer use agent",
Name: "Yutori n2 Computer Use",
Description: "Implements a Yutori n2 computer use agent",
Languages: []string{LanguageTypeScript, LanguagePython},
},
TemplateTzafonComputerUse: {
Expand Down
90 changes: 42 additions & 48 deletions pkg/templates/python/yutori/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
# Kernel Python Sample App - Yutori n1.5 Computer Use
# Kernel Python Sample App - Yutori n2 Computer Use

This Kernel app implements a prompt loop using Yutori's Navigator n1.5 with Kernel's Computer Controls API.
This Kernel app implements a prompt loop using Yutori's Navigator n2 with Kernel's Computer Controls API.

[Navigator n1.5](https://yutori.com/blog/introducing-n1-5) is Yutori's pixels-to-actions LLM that predicts browser actions from screenshots.

This template runs n1.5 in **computer-use-only mode**. n1.5 also supports a hybrid vision + DOM/JavaScript path (page-state extraction, custom JS, structured JSON output) for multi-field forms and bulk data extraction, but those tools are intentionally disabled here — see [Disabled tools](#disabled-tools).
[Navigator n2](https://docs.yutori.com/reference/n2) is Yutori's computer-use model. It reads a screenshot of the whole screen and answers with a batch of mouse and keyboard actions, and it can run shell commands and edit files on the machine it is driving.

## Setup

Expand All @@ -28,7 +26,7 @@ kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https
Optional payload fields:

- `record_replay` (bool) — capture a video of the session (paid plans only).
- `kiosk` (bool) — launch the browser without address bar / tabs ([see below](#kiosk-mode)).
- `reasoning_effort` (`"none"`, `"low"`, `"medium"`, `"xhigh"`) — n2 reasons at `medium` by default. `xhigh` gives the longest traces and does best on hard multi-step tasks; `none` turns reasoning off.
- `user_timezone` (IANA, e.g. `"America/New_York"`) and `user_location` (free text, e.g. `"New York, NY, US"`) — appended to the task message so the model has accurate temporal/locational grounding.

More involved example (Kanban drag-and-drop):
Expand All @@ -49,66 +47,62 @@ kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https

When enabled, the response will include a `replay_url` field with a link to view the recorded session.

## Kiosk mode

Prefer **non-kiosk mode** by default and when the agent is expected to switch domains via URL. Use **kiosk (`"kiosk": true`)** when: (1) you're recording sessions and want a cleaner UI in the replay, or (2) you're automating on a single website and the combination of the complex site layout and browser chrome (address bar, tabs) may confuse the agent.

Note: In kiosk mode the agent may still try to use the address bar to enter URLs; it's not available, so it will eventually use `goto_url`, but those attempts may result in slowdown of the overall session.
## Screen Configuration

Default (non-kiosk):

```bash
kernel invoke python-yutori-cua cua-task --payload '{"query": "Navigate to https://example.com, then navigate to ign.com and describe the page"}'
```
n2 is a desktop model: it expects a screenshot of the **whole screen**, browser chrome included, and it navigates by clicking the address bar rather than through a dedicated navigation action. A Kernel `viewport` sets the browser window size, and screenshots from Computer Controls capture that window with its tabs and toolbar — which is exactly what n2 wants, so this template does not use kiosk mode.

With kiosk (single-site or recording):
This template runs at **1280x800**. Yutori also lists 1920x1080 and 1280x720 as resolutions in regular use; grounding may degrade at extreme aspect ratios.

```bash
kernel invoke python-yutori-cua cua-task --payload '{"query": "Enter https://example.com in the search box and then describe the page.", "kiosk": true}'
```
> **Note:** n2 outputs coordinates in a 1000x1000 relative space, which are scaled to the actual screen dimensions per action.

## Viewport Configuration
See [Kernel Viewport Documentation](https://www.kernel.sh/docs/browsers/viewport) for all supported configurations.

Yutori n1.5 recommends a **1280×800 (WXGA, 16:10)** viewport for best grounding accuracy.
## Screenshots

> **Note:** n1.5 outputs coordinates in a 1000×1000 relative space, which are automatically scaled to the actual viewport dimensions.
Screenshots are converted to WebP before they are sent. Yutori caps requests at 10 MB, and a full-screen PNG trajectory blows past that on its own.

See [Kernel Viewport Documentation](https://www.kernel.sh/docs/browsers/viewport) for all supported configurations.
The loop also drops screenshots the model will not read: n2 keeps images from only the last 2 image-bearing messages, so older ones are stripped from each request while every message's text is kept.

## Screenshots
## Tools

Screenshots are automatically converted to WebP format for better compression across multi-step trajectories, as recommended by Yutori.
This template pins the `computer_use_tools-20260825` tool set. n2 rejects `disable_tools`, so all five tools are always served and the loop answers all of them.

## n1.5-latest Supported Actions
### `computer_batch`

This template uses the `browser_tools_core-20260403` tool set — coordinate-based browser actions that operate on screenshots only.
n2 returns a whole action list in one call. The loop runs them in order, stops at the first error, and replies with a single tool result carrying one screenshot taken after the last action that ran. Every coordinate in a batch refers to the screenshot from *before* the batch started.

| Action | Description |
|--------|-------------|
| `left_click` | Left mouse click at coordinates (supports `modifier`) |
| `double_click` | Double-click at coordinates (supports `modifier`) |
| `triple_click` | Triple-click at coordinates (supports `modifier`) |
| `middle_click` | Middle mouse click at coordinates |
| `right_click` | Right mouse click at coordinates |
| `mouse_move` | Move mouse to coordinates without clicking |
| `mouse_down` | Press the left mouse button at coordinates |
| `mouse_up` | Release the left mouse button at coordinates |
| `scroll` | Scroll page in a direction |
| `type` | Type text into focused element |
| `key_press` | Send a single key or key combination |
| `hold_key` | Hold a key for a duration |
| `drag` | Click-and-drag operation |
| `wait` | Pause for UI to update |
| `refresh` | Reload current page |
| `go_back` | Navigate back in history |
| `go_forward` | Navigate forward in history |
| `goto_url` | Navigate to a URL |

### Disabled tools

The DOM/Playwright-based "expanded" tools (`extract_elements`, `find`, `set_element_value`, `execute_js`) are intentionally disabled via the `disable_tools` request parameter — this template runs computer-use only and does not expose a Playwright page to the model.
| `middle_click` | Middle mouse click at coordinates (supports `modifier`) |
| `right_click` | Right mouse click at coordinates (supports `modifier`) |
| `scroll` | Scroll up or down at coordinates, in wheel notches |
| `type` | Type text into the focused element |
| `key_press` | Press a key, combination, or sequence |
| `drag` | Drag from `start_coordinates` to `coordinates` |
| `mouse_move` | Move the mouse to coordinates without clicking |
| `mouse_down` | Press and hold the left button (coordinates optional) |
| `mouse_up` | Release the left button (coordinates optional) |
| `hold_key` | Hold a key down for a duration |
| `wait` | Pause without interacting |
| `screenshot` | Look without acting |

Horizontal scrolling is not available, and there are no `goto_url` / `go_back` / `go_forward` / `refresh` actions — n2 drives the address bar and browser buttons like a person would.

### `bash`, `read`, `write`, `edit`

n2 can also work on the browser VM directly. These map onto Kernel's process and filesystem APIs:

| Tool | Backed by |
|------|-----------|
| `bash` | `browsers.process.exec` — each call is a separate process; the working directory carries over between calls, environment variables do not. `run_in_background` detaches the command and reports its pid and log path. |
| `read` | `browsers.fs.read_file` — returns `cat -n` output, with `offset` / `limit` paging |
| `write` | `browsers.fs.write_file` |
| `edit` | read, exact-string replace, write. Refuses to edit a file that has not been read in this session. |

## Resources

- [Yutori n1.5 API Documentation](https://docs.yutori.com/reference/n1-5)
- [Yutori n2 API Documentation](https://docs.yutori.com/reference/n2)
- [Kernel Documentation](https://www.kernel.sh/docs/quickstart)
Loading
Loading