Repository navigation
Desktop meeting endpoints: AI notes, Ask and cloud transcription - #4322
Open
richiemcilroy wants to merge 115 commits into
Open
richiemcilroy wants to merge 115 commits into
richiemcilroy wants to merge 115 commits into
Conversation
One row per (user, meeting) written up free, so signed-in people without Pro get their first few meeting notes without a separate counter.
checkFirewallRateLimitUnits charges several units against a firewall rule in one round trip, and consumeRollingBudget keeps an in-process rolling window that reports when it frees up. Adds the four rl_meetings_* rules.
/api/desktop/meetings/* for the desktop meeting recorder. Notes and Ask stream over SSE from the AI Gateway (provider chain off Vercel), with model and kill switches in MEETINGS_AI_CONFIG or an Edge Config item. Cloud transcription is only offered when ASSEMBLY_BASE_URL points at Cap's own speech service, never AssemblyAI directly.
The meeting upgrade sheet leaves the plan to the server, so the default can change without an app release.
|
Sandchest reviewed Sandchest reviews every change for vulnerabilities an attacker could use · 21 credits · Review details |
Each free meeting takes a numbered slot with a unique (userId, slot) key, so concurrent claims cannot pass the limit. A meeting is marked written when its notes finish; only a first write-up that failed gives the meeting back, and a second request for a meeting still being written waits. Rewrites and questions per free meeting are capped. Covered against MySQL in the Infra Database job.
Ask checks askEnabled and its own model chain instead of the notes switch. A free meeting can be rewritten after all of them are used. Speech calls are abandoned when the deadline passes or the app disconnects. Off Vercel, long transcripts are condensed before the single notes pass.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Member
Author
|
@greptileai review |
The meeting is marked written before the done event, so Ask works as soon as notes finish. A run that delivered any notes keeps its free meeting even if the stream then fails or is dropped; only a run that showed nothing gives it back. The speech copy is deleted from the submission itself, so a submission that completes after the request gave up is still removed.
Member
Author
|
@greptileai review |
# Conflicts: # .github/workflows/infra-database.yml
Ask accepts the desktop vocabulary and adds it as its own delimited block so names and terms are spelled the same way as in the transcript. The notes prompt now tells the model to use speaker labels exactly as the transcript writes them (for example Person 2 or You), so a later rename in the app maps each label to a name with a precise replace.
… machine A feed over plain http could be rewritten in transit to plant fake events and join links. Upgrade http feeds to https, keeping loopback feeds as they are.
Meeting audio was created with the default mode, readable by other users on the machine. Create each file 0600 before ffmpeg opens it and close its folder to other users.
…nd say why when it can't
Listeners on processes and devices that left CoreAudio's list were forgotten without being removed. An object CoreAudio kept alive could then call back into the signal after it was freed, and an id that came back was listened to twice.
# Conflicts: # packages/database/migrations/meta/0065_snapshot.json # packages/database/migrations/meta/_journal.json
CodeQL reads both prints as logging identifiers. The test names the missing event by position and the example says whether a device is known.
When the call is louder than the user and plays through the laptop speakers, its copy on the microphone can sound like the user. A stretch of the call was then taken for the user's voice returned by the far end, and its words were dropped. A stretch of the call whose copy the microphone is following (the leak test on the microphone over the same stretch) is the call itself, never an echo of the user.
…tuned Checked over a bright, colourful wallpaper in light and dark: the glass shows through and the sidebar text stays legible at 45% and 50%.
…their voice Chatter in a cafe or vocal music can be folded into the user's voice by the end-of-meeting pass. When Nemotron then hears the two apart after Stop, the user's own words were cut out as a new voice and You stayed on what was left: the chatter. When the split voice is the user's, You now moves to the part that talks while the call is quiet if the part left behind is heard mostly over the call, the same test the end-of-meeting pass uses to pick the user.
Giving each repaired segment the 2 s before it (6b3a40f) made the local model go silent for whole long segments it transcribes fully on its own. Over 63 synthetic meetings (21 scenarios, run plain, with every third segment lost, and with every third dropped to repair), words lost were 1,226 with that context and 998 without it, with fewer invented words too. A short tail that starts mid-word now gets no words rather than a guess, as before that change.
Measured on 27.3 beta 2: both the system audio request and a new process tap show "would like access to record your system audio", and the answer reaches the process. Only the comment changes; the rule stays tap-aware.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Server side of the desktop meeting recorder:
/api/desktop/meetings/*for capabilities, cloud transcription of meeting clips, AI notes and Ask.sequenceDiagram participant D as Cap desktop participant W as /api/desktop/meetings participant S as Cap speech service (ASSEMBLY_BASE_URL) participant G as AI Gateway D->>W: GET /capabilities W-->>D: { transcription, notes, notesTrialRemaining?, limits } loop during the meeting (Pro, optional) D->>W: POST /transcribe (16 kHz mono WAV clip) W->>S: upload + transcript (AssemblyAI API shape) W-->>D: { text, language, durationMs, words } end D->>W: POST /notes { transcript, meetingId, ... } W->>G: stream (condense long transcripts in chunks first) W-->>D: SSE title, delta..., done D->>W: POST /ask { question, transcript, notesMarkdown } W-->>D: SSE delta..., doneEndpoints
All behind the existing desktop bearer auth (
withAuth, scoped to/meetingsonly).GET /capabilitiesCache-Control: no-storePOST /transcribeaudio/wav16 kHz mono s16, max 120 s / 4 MB, keywords inx-cap-keywordsPOST /notesPOST /askASSEMBLY_BASE_URLpoints at Cap's own speech service. Unset, or anassemblyai.comhost, meanstranscription: falsein capabilities and503 not_configuredfrom/transcribe, so the app stays on its local model (apps/web/lib/meetings/speech.ts).(userId, meetingId)in the newmeeting_notes_trial_usestable (migration0062). A failed write gives the meeting back; Ask works only on those meetings.AI_GATEWAY_API_KEYelsewhere), falling back to the existing provider chain on self-hosted installs. Prompts wrap meeting material in a per-request random boundary (Ask uses an HMAC of the material so follow-ups reuse the cached prefix).POST /api/desktop/subscribenow defaults to the monthly plan when nopriceIdis sent, so the in-app upgrade sheet does not hard-code a price.Limits
Per user, checked before the body is read or the provider is called (
apps/web/lib/meetings/budgets.ts):rl_meetings_transcriberl_meetings_audio(1 unit per started 10 s)rl_meetings_notesrl_meetings_askDeadlines: transcribe 90 s, notes 270 s, ask 120 s.
Kill switches
Layered: defaults <
MEETINGS_AI_CONFIG(JSON env) < Edge Config itemmeetingsAiwhenEDGE_CONFIGis set (read every 30 s, never blocks a warm instance).{ "notesEnabled": false, "askEnabled": false, "transcriptionEnabled": false, "notesModels": ["openai/gpt-6-luna"], "trialMeetings": 3, "reasoningEffort": "none" }Existing routes
route.tsonly mounts/meetings; its auth middleware does not run for other desktop routes.How to test
Or point the desktop app's server URL at the deployment, turn on Meeting Mode and end a meeting: notes write themselves and Ask answers from the note.
Fix the desktop trial gate and the two notes races before merging.
Findings
Fix with agent prompt
Summary
Adds desktop meeting recording and notes, calendar connections, speaker labels, and server endpoints for transcription, AI notes, and Ask.
Reviews (4) · Last reviewed commit: "test(desktop): let the stand-in cloud he..." · Reviewed by Greptile