Mercurial

comparison infinite_canvas/docs/agent-sessions.md @ 280:49e9e591c9bb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
author MrJuneJune <me@mrjunejune.com>
date Tue, 18 Aug 2026 19:14:53 -0700
parents
children c57149ad216e
comparison
equal deleted inserted replaced
279:b3b547563ec7 280:49e9e591c9bb
1 # Dictation and Spatial Agent Sessions
2
3 ## Prototype flow
4
5 The first orchestration slice connects three retained systems:
6
7 1. `M` reuses the single retained `Dictation` text-area scratchpad and toggles
8 a hidden browser transport for `http://127.0.0.1:8090/?canvas=1`.
9 2. `Canvas_Web_Surface_Set_Dictation_Active()` deterministically starts or
10 stops capture on an already-negotiated WebRTC client. The hidden transport
11 acquires its stream, completes signaling/ICE, and opens the transcript data
12 channel during startup with its audio track disabled.
13 CEF grants microphone access only to the loopback dictation origin.
14 3. Status, partial, and final transcript events cross the platform boundary:
15 - native CEF encodes each event in the document title and
16 `OnTitleChange()` queues it for the caption;
17 - WebAssembly uses `window.postMessage()` from a hidden iframe.
18 4. `main.c` accumulates speech in the scratchpad and submits it to
19 `Canvas_Agent_Service` only when the user presses `Enter`. A selected text
20 area can enter the same path with `Ctrl/Cmd+Enter`.
21 Enter sends a `commit` command through the live transcript data channel, so
22 the current utterance finalizes without closing the microphone session.
23 5. `agent_service_copilot.c` sends the new thought plus
24 `Canvas_Scene_Build_Visible_Context()` through the shared asynchronous
25 `Inference_Bridge`.
26 6. The Bazel-managed Copilot SDK sidecar uses the `canvas_orchestrator` profile
27 and returns JSON choosing `create` or `append` and a conversation entity ID.
28 The canvas materializes that decision as a retained rich conversation card.
29
30 The scratchpad is excluded from serialized camera context because the submitted
31 thought is already sent separately. Only conversation entities currently in
32 view are valid append targets; moving a conversation out of view removes it
33 from the orchestrator's routing choices.
34
35 ## Service boundary
36
37 `agent_service_copilot.c` submits work to `Inference_Bridge`, whose worker
38 thread owns the JSON-line Python sidecar process so SDK latency never blocks
39 Raylib. The sidecar maintains the persistent
40 `infinite-canvas-orchestrator` SDK session and uses the loopback LiteLLM
41 gateway for authenticated GitHub Copilot inference. Only one orchestration
42 request is admitted at a time in this prototype.
43
44 `agent_service_copilot.c` warms that orchestration session plus three reserved
45 worker sessions during initialization. They share the same
46 `canvas_orchestrator` profile and move Copilot SDK client/session setup ahead of
47 the first user turn. The persistent orchestrator is resumed when available;
48 reserved workers are created fresh because they have no conversation history
49 to recover, avoiding expected `session.resume` errors during startup.
50
51 ## Conversation entities
52
53 `CANVAS_ENTITY_CONVERSATION` is intentionally more than an editable text area.
54 It retains:
55
56 - a stable entity/session ID used by Copilot routing;
57 - a title;
58 - user and agent transcript content;
59 - asynchronous working state;
60 - independent scroll, selection, lifecycle, z-order, and pinning state.
61 - auto-follow to the latest appended turn and a collapsed header-only state;
62 - a recoverable parked state that is excluded from orchestration context.
63
64 Images, browser views, and other components already appear in visible canvas
65 context as typed neighboring entities. A future resource list can attach those
66 entity IDs directly to a conversation without replacing the conversation card
67 or flattening rich content into one string.
68
69 Archiving a conversation records its prior position and moves it to the
70 off-canvas parking lot. The Developer Controls parking-lot button fits the
71 camera to parked sessions. Restoring a parked card returns it to its prior
72 position and makes it eligible for routing again when visible.
73
74 ## Current constraints
75
76 - `//infinite_canvas:orchestration_dev` owns the dictation server, LiteLLM
77 gateway, and canvas lifecycle. `//infinite_canvas:agent_dev` is an alias.
78 - One persistent Dictation scratchpad is reused and cleared after submission.
79 Submitted turns are retained in conversation entities after orchestration.
80 - One Copilot request runs at a time.
81 - Conversation transcript storage is currently bounded by
82 `CANVAS_ENTITY_TEXT_CAPACITY`.
83 - Linux uses the Copilot SDK bridge. Other platforms currently select an
84 explicit unavailable stub.