Skip to content

Desktop meeting endpoints: AI notes, Ask and cloud transcription - #4322

Open
richiemcilroy wants to merge 115 commits into
mainfrom
feat/meeting-notes-api
Open

richiemcilroy wants to merge 115 commits into
mainfrom
feat/meeting-notes-api

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Server side of the desktop meeting recorder: /api/desktop/meetings/* for capabilities, cloud transcription of meeting clips, AI notes and Ask.

sequenceDiagram
    participant D as Cap desktop
    participant W as /api/desktop/meetings
    participant S as Cap speech service (ASSEMBLY_BASE_URL)
    participant G as AI Gateway
    D->>W: GET /capabilities
    W-->>D: { transcription, notes, notesTrialRemaining?, limits }
    loop during the meeting (Pro, optional)
        D->>W: POST /transcribe (16 kHz mono WAV clip)
        W->>S: upload + transcript (AssemblyAI API shape)
        W-->>D: { text, language, durationMs, words }
    end
    D->>W: POST /notes { transcript, meetingId, ... }
    W->>G: stream (condense long transcripts in chunks first)
    W-->>D: SSE title, delta..., done
    D->>W: POST /ask { question, transcript, notesMarkdown }
    W-->>D: SSE delta..., done
Loading

Endpoints

All behind the existing desktop bearer auth (withAuth, scoped to /meetings only).

Route Gate Body Reply
GET /capabilities signed in none JSON, Cache-Control: no-store
POST /transcribe Pro, cloud transcription configured audio/wav 16 kHz mono s16, max 120 s / 4 MB, keywords in x-cap-keywords JSON words with ms timings
POST /notes Pro, or a free meeting left JSON transcript (max 400k chars) SSE
POST /ask Pro, or a meeting written up free JSON question + meeting material SSE
  • Cloud transcription is only offered when ASSEMBLY_BASE_URL points at Cap's own speech service. Unset, or an assemblyai.com host, means transcription: false in capabilities and 503 not_configured from /transcribe, so the app stays on its local model (apps/web/lib/meetings/speech.ts).
  • Clips go out as ordinary audio with no priority extension fields, so they queue behind live captions on the speech service.
  • Signed-in people without Pro get their first 3 meetings written up free, counted per (userId, meetingId) in the new meeting_notes_trial_uses table (migration 0062). A failed write gives the meeting back; Ask works only on those meetings.
  • Notes and Ask run on the Vercel AI Gateway (OIDC on Vercel, AI_GATEWAY_API_KEY elsewhere), falling back to the existing provider chain on self-hosted installs. Prompts wrap meeting material in a per-request random boundary (Ask uses an HMAC of the material so follow-ups reuse the cached prefix).
  • POST /api/desktop/subscribe now defaults to the monthly plan when no priceId is sent, so the in-app upgrade sheet does not hard-code a price.

Limits

Per user, checked before the body is read or the provider is called (apps/web/lib/meetings/budgets.ts):

Rule Firewall (suggested) In-process fallback
rl_meetings_transcribe 1200 / 1 h 1200 / 1 h
rl_meetings_audio (1 unit per started 10 s) 720 / 1 h 3600 / 24 h
rl_meetings_notes 60 / 1 h 60 / 1 h
rl_meetings_ask 120 / 1 h 120 / 1 h

Deadlines: transcribe 90 s, notes 270 s, ask 120 s.

Kill switches

Layered: defaults < MEETINGS_AI_CONFIG (JSON env) < Edge Config item meetingsAi when EDGE_CONFIG is set (read every 30 s, never blocks a warm instance).

{ "notesEnabled": false, "askEnabled": false, "transcriptionEnabled": false,
  "notesModels": ["openai/gpt-6-luna"], "trialMeetings": 3, "reasoningEffort": "none" }

Existing routes

  • route.ts only mounts /meetings; its auth middleware does not run for other desktop routes.
  • No new queries or awaits on existing routes. The subscribe default is a synchronous plan lookup.

How to test

curl -H "Authorization: Bearer $KEY" "$HOST/api/desktop/meetings/capabilities"
curl -N -H "Authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"transcript":"Alex: ship Friday. Sam: I will write the notes.","meetingId":"m1","title":""}' \
  "$HOST/api/desktop/meetings/notes"

Or point the desktop app's server URL at the deployment, turn on Meeting Mode and end a meeting: notes write themselves and Ask answers from the note.

RetriggerView in GreptileConfidence Score: 2/5

Fix the desktop trial gate and the two notes races before merging.

Findings

  1. P1 Trial exhaustion blocks rewrites ▶
  2. P1 AI reads unsaved notes ▶
  3. P1 Loading erases new edits ▶
Fix with agent prompt
### Issue 1
apps/desktop-gpui/src/meetings/mod.rs:3724-3727
After a free user uses all three meetings, `prepare_notes` rejects every rewrite before contacting the server. `trial_allows()` only checks whether `notesTrialRemaining` is positive, but the server still allows rewrites of covered meetings within their remaining run allowance. Let the server check the requested meeting instead of blocking it based on the account-wide count.

### Issue 2
apps/desktop-gpui/src/meetings_ui/documents/pane.rs:2523-2528
`generate()` queues pending edits with `flush_saves()`, then immediately calls `generate_notes()`. That call reads the notes and previous summary from disk while the save runs on another thread. Clicking Write or Regenerate soon after typing can therefore send the old text to AI and omit the user's latest corrections. Wait for the queued saves before building the request, or build it from the current editor contents.

### Issue 3
apps/desktop-gpui/src/meetings_ui/documents/pane.rs:890-896
The background load replaces the notes whenever the document ID still matches, even if the user has typed since loading started. `open()` makes ordinary notes editable immediately, and `loaded_id` does not guard normal input. On a slow disk, text typed or pasted just after opening a note can disappear when this callback runs. Keep the editor read-only until loading finishes, or reject a load result after the editor changes. `settle_enhanced()` needs the same check.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds desktop meeting recording and notes, calendar connections, speaker labels, and server endpoints for transcription, AI notes, and Ask.

  • Cap desktop records meetings and builds transcripts from microphone and computer audio.
  • Meeting documents can link to events from connected calendars.
  • Meeting notes and questions stream from Cap's AI services into the desktop app.
  • Meeting reliability checks run in CI, and desktop upgrades default to the monthly plan.

Reviews (4) · Last reviewed commit: "test(desktop): let the stand-in cloud he..." · Reviewed by Greptile

One row per (user, meeting) written up free, so signed-in people without
Pro get their first few meeting notes without a separate counter.
checkFirewallRateLimitUnits charges several units against a firewall rule
in one round trip, and consumeRollingBudget keeps an in-process rolling
window that reports when it frees up. Adds the four rl_meetings_* rules.
/api/desktop/meetings/* for the desktop meeting recorder. Notes and Ask
stream over SSE from the AI Gateway (provider chain off Vercel), with
model and kill switches in MEETINGS_AI_CONFIG or an Edge Config item.
Cloud transcription is only offered when ASSEMBLY_BASE_URL points at
Cap's own speech service, never AssemblyAI directly.
The meeting upgrade sheet leaves the plan to the server, so the default
can change without an app release.
@sandchest-com

sandchest-com Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 08a60eb: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 21 credits · Review details

Comment thread apps/web/lib/meetings/trial.ts Resolved

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 5 potential issues (5 medium).

View 2 other findings in Capy.

Open in Capy

Comment thread apps/web/lib/meetings/trial.ts Resolved
Comment thread apps/web/lib/meetings/ai-config.ts Resolved
Each free meeting takes a numbered slot with a unique (userId, slot)
key, so concurrent claims cannot pass the limit. A meeting is marked
written when its notes finish; only a first write-up that failed gives
the meeting back, and a second request for a meeting still being
written waits. Rewrites and questions per free meeting are capped.
Covered against MySQL in the Infra Database job.
Ask checks askEnabled and its own model chain instead of the notes
switch. A free meeting can be rewritten after all of them are used.
Speech calls are abandoned when the deadline passes or the app
disconnects. Off Vercel, long transcripts are condensed before the
single notes pass.
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Nov 21, 2026 5:36am UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

The meeting is marked written before the done event, so Ask works as
soon as notes finish. A run that delivered any notes keeps its free
meeting even if the stream then fails or is dropped; only a run that
showed nothing gives it back. The speech copy is deleted from the
submission itself, so a submission that completes after the request
gave up is still removed.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 17, 2026 19:18 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

# Conflicts:
#	.github/workflows/infra-database.yml

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Ask accepts the desktop vocabulary and adds it as its own delimited block
so names and terms are spelled the same way as in the transcript.

The notes prompt now tells the model to use speaker labels exactly as the
transcript writes them (for example Person 2 or You), so a later rename in
the app maps each label to a name with a precise replace.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

… machine

A feed over plain http could be rewritten in transit to plant fake events
and join links. Upgrade http feeds to https, keeping loopback feeds as they
are.
Meeting audio was created with the default mode, readable by other users
on the machine. Create each file 0600 before ffmpeg opens it and close its
folder to other users.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 00:36 Inactive
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 00:38 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 5 other findings in Capy.

Open in Capy

Listeners on processes and devices that left CoreAudio's list were
forgotten without being removed. An object CoreAudio kept alive could then
call back into the signal after it was freed, and an id that came back was
listened to twice.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 00:46 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

# Conflicts:
#	packages/database/migrations/meta/0065_snapshot.json
#	packages/database/migrations/meta/_journal.json

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

CodeQL reads both prints as logging identifiers. The test names the
missing event by position and the example says whether a device is known.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 01:12 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

When the call is louder than the user and plays through the laptop speakers,
its copy on the microphone can sound like the user. A stretch of the call
was then taken for the user's voice returned by the far end, and its words
were dropped. A stretch of the call whose copy the microphone is following
(the leak test on the microphone over the same stretch) is the call itself,
never an echo of the user.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 01:22 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

…tuned

Checked over a bright, colourful wallpaper in light and dark: the glass
shows through and the sidebar text stays legible at 45% and 50%.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 01:32 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

…their voice

Chatter in a cafe or vocal music can be folded into the user's voice by the
end-of-meeting pass. When Nemotron then hears the two apart after Stop, the
user's own words were cut out as a new voice and You stayed on what was left:
the chatter. When the split voice is the user's, You now moves to the part
that talks while the call is quiet if the part left behind is heard mostly
over the call, the same test the end-of-meeting pass uses to pick the user.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 02:46 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Giving each repaired segment the 2 s before it (6b3a40f) made the local
model go silent for whole long segments it transcribes fully on its own.
Over 63 synthetic meetings (21 scenarios, run plain, with every third
segment lost, and with every third dropped to repair), words lost were
1,226 with that context and 998 without it, with fewer invented words too.

A short tail that starts mid-word now gets no words rather than a guess,
as before that change.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 04:18 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Measured on 27.3 beta 2: both the system audio request and a new process
tap show "would like access to record your system audio", and the answer
reaches the process. Only the comment changes; the rule stays tap-aware.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 21, 2026 05:36 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch was successfully deployed

1 active (outdated) and 1 inactive deployments
Preview – cap-status — 08a60ebf Deployed Nov 21, 2026 by vercel[bot]
Preview – cap-web — 9b319ba4 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants