v0.9.43: Sim links from search and file shares, synced org chart, CLI recovery guidance, image and video catalog refresh

.github/CI.md

@@ -2,9 +2,9 @@
 
 The `changes` job runs Turborepo 2.9.14's `affectedTasks` query before installing application dependencies or provisioning services. `turbo.json` records package dependency propagation; `apps/sim/turbo.json` partitions the app into shared runtime, server, workspace UI, and acceptance tasks. `.github/ci-suites.json` turns those tasks into job matrices. The uploaded `ci-plan` artifact names changed files, selected tasks, attribution reasons, and any fallback.
 
-A PR compares its merge base with the checked-out merge commit. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
+A PR compares the checked-out GitHub merge commit with its first parent: the actual target revision tested by that merge. The PR event's base SHA can become stale when the target advances. Scoping requires a matching GitHub merge ref and SHA, exactly two merge parents, and a usable target parent; otherwise every check runs. This works with the existing depth-two checkout without fetching older ancestry. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
 
-Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with their merge base. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Independent SDK and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
+Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with the tested merge's first parent. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Independent SDK and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
 
 Docs production builds run once and publish an artifact to isolated HTTP, corpus, Chromium, and WebKit jobs. Source checks keep their own checkout because generator verification mutates inputs. Content caches use separate lint/source/build namespaces; generation validates every restore, repairs missing declarations, and removes obsolete modules. The Next build cache keeps its supported incremental behavior. Content and Next caches have size limits, and Next cache files expire after seven days; successful content eviction happens after all consumers finish.
 
@@ -36,6 +36,8 @@ Closing a same-repository pull request deletes only its GitHub merge-ref caches
 
 Memory-heavy builds and browser jobs retain appropriately sized Blacksmith runners. Electron uses 4-vCPU Blacksmith: both shards passed within 14 seconds of the 8-vCPU control; the free GitHub rest shard was slower and its lease run stalled.
 
+WebKit acceptance on Blacksmith uses the matching, digest-pinned Playwright browser image for mobile and docs navigation. The app, database fixtures, native modules, and harness stay on the host and retain loopback URLs. Chromium and GitHub fallback keep native browser installation and launch. Update both workspace Playwright dependencies, the image index digest, and the expected client version/revision checks together; verify the installed release's native launch timeout and real WebKit acceptance before changing the pin.
+
 Every HTTP and browser group consumes the same production build in an isolated job/database. Electron also uses that build and skips its development-route warm-up. Local HTTP and Electron development runs retain their dev-server behavior. Build inputs exclude test-only files so a harness edit can reuse an unchanged production build.
 
 Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. Each harness owns its acceptance task, so an edit selects its own job. The inbox protocol task depends on server and realtime code; the browser tasks also depend on workspace UI. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split.
@@ -44,7 +46,7 @@ Live Electron acceptance tracks its live-app suite, fixtures, and Playwright con
 
 Desktop smoke weights also stay outside the desktop runtime task. Editing weights reruns desktop partition coverage without selecting live-app acceptance.
 
-Browser extension acceptance runs native interaction and access-revalidation groups in parallel jobs with isolated databases, apps, browser profiles, and reports. Both retain the same task boundary, so an affected extension harness selects both groups. Native streaming and first-open capture checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.
+Browser extension acceptance runs navigation, composer, and access-revalidation groups in three parallel jobs with isolated databases, apps, browser profiles, and reports. All three retain the same task boundary, so an affected extension harness selects every group. Navigation and composer checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.
 
 Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation.
 

.github/ci-suites.json

@@ -25,7 +25,11 @@
       "task": "ci:e2e-organization-search"
     },
     {
-      "group": "browser-extension-native",
+      "group": "browser-extension-navigation",
+      "task": "ci:e2e-browser-extension"
+    },
+    {
+      "group": "browser-extension-composer",
       "task": "ci:e2e-browser-extension"
     },
     {

.github/scripts/http-e2e.sh

@@ -3,7 +3,7 @@
 # CI groups, which reuse the production build from `e2e-build`.
 # The job extracts it into apps/sim/.next before running these suites.
 #
-# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo>   (run from apps/sim)
+# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access|project-files|mobile|seo>   (run from apps/sim)
 #
 # The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
 # its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
@@ -20,7 +20,7 @@
 # telemetry flush runs detached and still writes .next/dev).
 set -euo pipefail
 
-group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo>}
+group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access|project-files|mobile|seo>}
 report_dir="$RUNNER_TEMP/e2e"
 ready_timeout_seconds=600
 mkdir -p "$report_dir"
@@ -164,7 +164,11 @@ case "$group" in
     export NEXT_PUBLIC_CHAT_DISABLED=true
     export SIM_LINKS=true
     export PEOPLE=true
+    export CRON_SECRET=people-directory-sync-local-e2e-secret
+    start_browser_install people chromium
     start_app scim 3017 SCIM record-http-status
+    wait_for_browsers
+    export PEOPLE_E2E_SCREENSHOT_DIR="$report_dir/people-ui-report.json.artifacts"
     SCIM_E2E_BASE_URL="$local_app_url" \
     SCIM_E2E_DATABASE_URL="$DATABASE_URL" \
     SCIM_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
@@ -233,7 +237,7 @@ case "$group" in
     export COPILOT_TOOL_PERMISSIONS_ENABLED=true
     export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
     export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true NEXT_PUBLIC_ORGANIZATIONS_ENABLED=true
-    if [ "$group" != desktop-inbox ]; then start_browser_install "$group" chromium; fi
+    if [ "$group" != desktop-inbox ]; then start_browser_install "$group" --only-shell chromium; fi
     export INTERNAL_API_SECRET=desktop-inbox-http-ci-local-secret-at-least-32-characters
     export DB_TX_TRIPWIRE=throw
     start_app "$group" 3019 "$group" record-http-status
@@ -264,18 +268,20 @@ case "$group" in
     esac
     ;;
 
-  browser-extension|browser-extension-native|browser-extension-access)
+  browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access)
     extension_group=all
     case "$group" in
       browser-extension-native) extension_group=native ;;
+      browser-extension-navigation) extension_group=navigation ;;
+      browser-extension-composer) extension_group=composer ;;
       browser-extension-access) extension_group=access ;;
     esac
     export REDIS_URL=redis://127.0.0.1:6379
     export NEXT_PUBLIC_FORCE_HOSTED=false
     export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
     export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
     export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
-    start_browser_install "$group" chromium
+    start_browser_install "$group" --no-shell chromium
     extension_identity=$(bun - <<'JS'
       import { generateKeyPairSync, createHash } from 'node:crypto'
       const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
@@ -312,10 +318,19 @@ JS
     export INTERNAL_API_SECRET=mobile-http-ci-local-secret-at-least-32-characters
     export NEXT_PUBLIC_SOCKET_URL=http://127.0.0.1:3025 SOCKET_SERVER_URL=http://127.0.0.1:3025
     read -ra browsers <<< "${MOBILE_E2E_BROWSER:-chromium webkit}"
-    start_browser_install mobile "${browsers[@]}"
+    if [ "${PLAYWRIGHT_WEBKIT_WS_ENDPOINT+x}" = x ]; then
+      if [ "${MOBILE_E2E_BROWSER:-}" != webkit ]; then
+        echo "::error::A remote endpoint requires explicit WebKit selection." >&2
+        exit 1
+      fi
+    else
+      start_browser_install mobile --only-shell "${browsers[@]}"
+    fi
     start_app mobile 3024 'mobile browser' record-http-status
     start_realtime mobile 3025
-    wait_for_browsers
+    if [ "${PLAYWRIGHT_WEBKIT_WS_ENDPOINT+x}" != x ]; then
+      wait_for_browsers
+    fi
     MOBILE_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
     MOBILE_E2E_DATABASE_URL="$DATABASE_URL" \
     MOBILE_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \

.github/scripts/playwright-server.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# Owns one pinned WebKit server; start prints its loopback endpoint, stop preserves its logs.
+set -euo pipefail
+umask 077
+
+action=${1:-}
+state_argument=${2:-}
+if [ "$#" -ne 2 ] || { [ "$action" != start ] && [ "$action" != stop ]; }; then
+  echo 'Usage: playwright-server.sh <start|stop> <state-directory-under-RUNNER_TEMP>' >&2
+  exit 2
+fi
+: "${RUNNER_TEMP:?RUNNER_TEMP must be set}"
+: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY must be set}"
+: "${GITHUB_RUN_ID:?GITHUB_RUN_ID must be set}"
+: "${GITHUB_RUN_ATTEMPT:?GITHUB_RUN_ATTEMPT must be set}"
+: "${GITHUB_JOB:?GITHUB_JOB must be set}"
+if ! [[ "$GITHUB_RUN_ID" =~ ^[1-9][0-9]*$ && "$GITHUB_RUN_ATTEMPT" =~ ^[1-9][0-9]*$ && "$GITHUB_JOB" =~ ^[a-zA-Z0-9_-]+$ ]]; then
+  echo 'Invalid GitHub job ownership metadata.' >&2
+  exit 2
+fi
+state=$(node - "$RUNNER_TEMP" "$state_argument" <<'JS'
+const fs = require('node:fs')
+const path = require('node:path')
+const root = fs.realpathSync(process.argv[2])
+const state = path.resolve(process.argv[3])
+if (path.dirname(state) !== root || (fs.existsSync(state) && fs.realpathSync(state) !== state))
+  throw new Error('State must be a direct, non-symlink child of RUNNER_TEMP')
+process.stdout.write(state)
+JS
+)
+state_key=$(node -e 'process.stdout.write(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex"))' "$state")
+image='mcr.microsoft.com/playwright@sha256:2f05f8a734509a7b3f26a912639488ee08f58a7b292d14d62271bcff9038768f'
+starting=false
+
+stop_container() {
+  local cid inspection remaining status=0
+  [ -f "$state/cid" ] || return 0
+  cid=$(< "$state/cid")
+  if ! [[ "$cid" =~ ^[0-9a-f]{64}$ ]]; then
+    echo 'Refusing to stop an invalid container ID.' >&2
+    return 1
+  fi
+  if ! inspection=$(docker container inspect "$cid" 2> "$state/inspect-error.log"); then
+    remaining=$(docker container ls --all --no-trunc --filter "id=$cid" --format '{{.ID}}') || return 1
+    [ -z "$remaining" ] && return 0
+    cat "$state/inspect-error.log" >&2
+    return 1
+  fi
+  if ! jq -e --arg repository "$GITHUB_REPOSITORY" --arg run "$GITHUB_RUN_ID" \
+    --arg attempt "$GITHUB_RUN_ATTEMPT" --arg job "$GITHUB_JOB" --arg state "$state_key" \
+    '.[0].Config.Labels | .["com.sim.ci.playwright-server"] == "true"
+      and .["com.sim.ci.repository"] == $repository and .["com.sim.ci.run"] == $run
+      and .["com.sim.ci.attempt"] == $attempt and .["com.sim.ci.job"] == $job
+      and .["com.sim.ci.state"] == $state' <<< "$inspection" > /dev/null; then
+    echo 'Refusing to stop a container owned by another job or state directory.' >&2
+    return 1
+  fi
+  docker stop --timeout 10 "$cid" > "$state/stop.log" 2>&1 || status=1
+  docker logs "$cid" > "$state/server.log" 2>&1 || status=1
+  docker rm "$cid" > "$state/remove.log" 2>&1 || status=1
+  return "$status"
+}
+
+trap '
+  status=$?
+  trap - EXIT
+  if [ "$status" -ne 0 ] && [ "$starting" = true ]; then
+    stop_container || echo "Owned Playwright server cleanup failed; preserve its state and logs." >&2
+  fi
+  exit "$status"
+' EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+
+if [ "$action" = stop ]; then
+  stop_container
+  exit 0
+fi
+if [ -e "$state/cid" ]; then
+  echo 'Refusing to reuse an existing Playwright server state.' >&2
+  exit 1
+fi
+mkdir -p -- "$state"
+repo_root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)
+driver=$(node - "$repo_root/apps/sim" <<'JS'
+const fs = require('node:fs')
+const path = require('node:path')
+const packagePath = require.resolve('playwright-core/package.json', { paths: [process.argv[2]] })
+const root = fs.realpathSync(path.dirname(packagePath))
+const manifest = JSON.parse(fs.readFileSync(path.join(root, 'browsers.json'), 'utf8'))
+if (JSON.parse(fs.readFileSync(packagePath, 'utf8')).version !== '1.64.1' ||
+    manifest.browsers.find(browser => browser.name === 'webkit')?.revision !== '2370')
+  throw new Error('The pinned WebKit image requires Playwright 1.64.1, revision 2370')
+process.stdout.write(root)
+JS
+)
+docker pull "$image" > "$state/pull.log" 2>&1
+starting=true
+docker run --detach --init --cidfile "$state/cid" \
+  --name "sim-ci-playwright-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-$GITHUB_JOB-$$" \
+  --label com.sim.ci.playwright-server=true --label "com.sim.ci.repository=$GITHUB_REPOSITORY" \
+  --label "com.sim.ci.run=$GITHUB_RUN_ID" --label "com.sim.ci.attempt=$GITHUB_RUN_ATTEMPT" \
+  --label "com.sim.ci.job=$GITHUB_JOB" --label "com.sim.ci.state=$state_key" \
+  --network host --cpus 8 --memory 4g --memory-swap 4g --pids-limit 512 --shm-size 256m \
+  --mount "type=bind,src=$driver,dst=/opt/playwright-core,readonly" \
+  "$image" node /opt/playwright-core/cli.js run-server --host 127.0.0.1 --port 0 \
+  > "$state/start.log" 2>&1
+cid=$(< "$state/cid")
+if ! [[ "$cid" =~ ^[0-9a-f]{64}$ ]]; then
+  echo 'Docker did not return a complete container ID.' >&2
+  exit 1
+fi
+docker container inspect "$cid" > "$state/metadata.json"
+deadline=$((SECONDS + 30))
+while [ "$SECONDS" -lt "$deadline" ]; do
+  live=$(docker container inspect --format '{{.State.Running}} {{.State.Pid}}' "$cid")
+  if ! [[ "$live" =~ ^true\ [1-9][0-9]*$ ]]; then
+    docker logs "$cid" > "$state/server.log" 2>&1
+    echo 'Playwright server exited before readiness.' >&2
+    exit 1
+  fi
+  docker logs "$cid" > "$state/server.log" 2>&1
+  endpoint=$(sed -n 's@^Listening on \(ws://127\.0\.0\.1:[1-9][0-9]*/\)$@\1@p' "$state/server.log")
+  if [ -n "$endpoint" ]; then
+    node - "$endpoint" <<'JS'
+const endpoint = process.argv[2]
+if (!/^ws:\/\/127\.0\.0\.1:[1-9]\d*\/$/.test(endpoint)) throw new Error('Invalid server endpoint')
+new URL(endpoint)
+process.stdout.write(`${endpoint}\n`)
+JS
+    exit 0
+  fi
+  sleep 0.1
+done
+echo 'Playwright server did not report a live loopback endpoint within 30 seconds.' >&2
+exit 1