Skip to content

Feature/session based browser - #491

Closed
zfoong wants to merge 73 commits into
mainfrom
feature/session-based-browser
Closed

zfoong wants to merge 73 commits into
mainfrom
feature/session-based-browser

Conversation

@zfoong

@zfoong zfoong commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

What and why?
Combining the generative UI and mini browser into a left panel for each session. Making our built-in browser session-based. Major UI update to make them display in the left panel without being overwhelming.

ahmad-ajmal and others added 30 commits September 30, 2026 11:05
fix: decode and stop silently truncating inbound integration messages
Signed-off-by: ahmad-ajmal <ahmadajmal1514@gmail.com>
fix tunnel failed silently and get error from API rather than log file
Signed-off-by: ahmad-ajmal <ahmadajmal1514@gmail.com>
Signed-off-by: ahmad-ajmal <ahmadajmal1514@gmail.com>
Squash of the original korivi-Exp1 prototype (web agent V1/V2, password
vault, onboarding error wrap) applied on top of current main. Conflicts
resolved against main's newer browser adapter and sidebar; the local-only
slow_mode and profile-interview workarounds were dropped.

The page still targets the old task API; it is adapted in the next commits.
A real Chromium that agents drive and the user watches live, usable by every
kind of agent (main and chat sessions, a dedicated Mini Browser chat,
scheduled runs, Agent App trigger runs and the new browser_agent sub-agent).

Engine (app/mini_browser):
- dedicated "mini-browser" thread + event loop owns Playwright; callers on any
  loop marshal in (fixes sub-agents, which run on their own loops)
- one tab per agent session with per-tab locks, owner labels, take-control,
  follow-view, crash and driver-death recovery, idle shutdown, profile lock
- live view via CDP screencast of the viewed tab, only while a UI tab is
  subscribed, latest-frame-wins delivery
- ad blocking via CDP Network.setBlockedURLs (context.route disabled the HTTP
  cache); CraftBot's own UI origins blocked browser-wide
- human-like mouse paths, typing rhythm and scrolling
- 13 mini_browser_* actions; every result carries a fresh page observation

Password vault: Fernet + Windows DPAPI in .credentials/, atomic writes, no
silent data loss, strict origin matching; autofill only for the current
page's origin and never exposed to the model, logs or UI.

Platform: mini_browser_* WebSocket protocol and lanes, new session type
mini_browser, settings section, error codes, AgentBase run-state / stop /
delete / sub-agent / shutdown hooks, skill, prompt rule and AGENT.md.

Frontend: Mini Browser page (sidebar entry, live view with IME/paste/
shortcuts, tabs, take control, passwords dialog), 7 languages.
ahmad-ajmal and others added 29 commits October 14, 2026 10:23
- Agent input never lands after the user takes control. Waits for an
  element run in 250 ms slices that check control; clicks press through
  page.mouse at a point just checked to hit the element instead of
  Playwright's click, which kept waiting inside the driver. Focus, clear,
  caret, select, upload and whole-value inputs, and the vault login's fill
  and Enter, act on element handles right after the control check.
- Agents cannot close a tab the user controls.
- Opening a tab reserves its slot before the first await, so concurrent
  opens never exceed the browser-wide or per-owner tab limits.
- Agent screenshots cover password and secret fields, including fields
  that were password fields before a "show password" toggle.
- A saved password in a download's suggested name is masked before it
  becomes the file name, a log line or a notice.
- Cut deletes the selection only after its own copy reached the
  clipboard, in the tab it was copied from (copy replies echo requestId).
- One settings normalizer (mini_browser.config.clean_setting) for
  app.config and MiniBrowserSettings.
- Browser tests skip only when Playwright or Chromium is not installed;
  CRAFTBOT_REQUIRE_BROWSER_TESTS=1 makes that a failure too.
Signed-off-by: ahmad-ajmal <ahmadajmal1514@gmail.com>
…ance

fix #475: separate Windows cmd and PowerShell actions
fix: prevent integration settings loader from hanging
…ns-cleanup

Make agent aware of NOTE and make missions obsolete
…replies

fix: enable Telegram replies, owner control, reply context and typing feedback
grep improvement update using ripgrep
Threads (and other sites that link addresses as you type) put agents into
a delete-and-retype loop, with link previews flipping between sites:

- Web and e-mail addresses are inserted whole, like a paste. Typed key by
  key, the site saw every partial address ("https://e", ...,
  "example.co") and previewed the wrong ones.
- After typing into a rich text box (contenteditable), the box is read
  back and the result says whether it holds the whole text; links, link
  previews and styled mentions are named as the site's formatting.
- The same text is typed into the same page at most 3 times in 10
  minutes; the 4th attempt is refused with a "stop and check or ask the
  user" message. Only a hash of the text is kept.
- The mini-browser skill and the type action describe link previews and
  say never to delete and retype a post because it looks different.

Sound: Playwright starts Chromium with --mute-audio, so no page could play
sound. New setting mini_browser.sound (default true) drops that switch; it
applies at the browser's next start.
The compact page observation showed only the first 40 characters of a
post or message box, so a post with a link often appeared as
"https://example.c…": a broken-looking link the agent took for its own
typo, one more trigger of the Threads retype loop.

- Textareas and rich text boxes show up to 300 characters, cut between
  words (also in the tighter fallback format), and say how long the full
  text is when it is cut.
- The read-back after typing covers the whole editor, also when the
  element is one of its paragraphs.
- The retype guard counts a retry that only writes an address
  differently ("https://" added or dropped) as the same text.

Checked against the real Lexical editor with its AutoLinkPlugin: typing
an address key by key changed the link five times (example.c ->
example.co -> example.com -> ...), now it is created once. Sound was
measured in the Windows mixer: Chromium plays (live-view and agent
clicks) with sound on, and is silent with it off.
- Dropdowns: Playwright's select_option keeps waiting in the driver while
  the option is missing (force does not stop that retry), so a choice
  could land after the user took control. Each attempt is now one
  in-page step that finds the option again and selects it only if it is
  there and enabled (input + change, like Playwright); between attempts
  the wait only reads, and every attempt checks control first.
- Closing the last tab: the owner and take-control checks run again after
  the replacement tab has opened, right before the close.
- Cut: the copy is marked as a cut and the backend keeps the text it
  copied (memory only, 30 s). The delete is a "cut" input event in the
  ordered input queue, after whatever the user did meanwhile, and runs
  only while the selection is still exactly that text; otherwise it is
  skipped, so nothing that is not on the clipboard is deleted.
Signed-off-by: ahmad-ajmal <ahmadajmal1514@gmail.com>
- Cut: only a complete copy can be cut. A selection longer than the copy
  limit, or one with a saved password masked out, reaches the clipboard
  only in part, so it is copied but never deleted; the delete compares the
  whole selection. Pending cuts are kept per request, with ids scoped to
  the viewer connection (and unique per view mount), so two viewers' cuts
  never share a record and a mismatched delete leaves the others intact.
- Retype guard: scoped to the same field (data-mb-field) and to text that
  was not sent; Enter or the page moving on resets it. Addresses only
  ignore "https://" and "www.", so different links are different texts.
  Filling several fields with the same value is never refused.
- Navigation: control is checked again right before an operation runs
  (after crash recovery and bringing its window forward), right before
  navigate's goto / back / forward / reload, and before a new tab opened
  with a URL loads it.
Fixed action set accumulation issue causing system prompt to bloat
feat #479: add interactive Generative UI outputs
PR #483 was merged into main by mistake; this merges it into V1.4.4,
its intended base.

V1.4.4 made a session's <actions> a function of its type and turned
add_action_sets into a read that delivers schemas into the event stream
(SessionManager.add/remove/get_action_sets, Session.action_sets and the
create_session action_sets argument are gone). The PR was written
against the old model, so it is ported here:

- SessionType.BASE_ACTION_SETS gives mini_browser sessions the
  mini_browser set; ws.ensure_session no longer passes action_sets.
- load_action_sets keeps the PR's lenient name matching, web_agent
  alias and unknown_sets hint; the PR's remove_action_sets changes are
  dropped with the API.
- _apply_workflow_capabilities keeps the PR's user-skill action sets
  and delivers them through load_action_sets.
- The router keeps the PR's prompt filter, without integration_essentials
  (removed on V1.4.4).
- AGENT.md (both copies) and the heartbeat skill describe the V1.4.4
  model; remove_action_sets references dropped.
- Mini Browser tests updated to the V1.4.4 API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zfoong zfoong closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants