# Dictation and Spatial Agent Sessions

## Prototype flow

The first orchestration slice connects three retained systems:

1. `M` reuses the single retained `Dictation` text-area scratchpad and toggles
   a hidden browser transport for `http://127.0.0.1:8090/?canvas=1`.
2. `Canvas_Web_Surface_Set_Dictation_Active()` deterministically starts or
   stops capture on an already-negotiated WebRTC client. The hidden transport
   acquires its stream, completes signaling/ICE, and opens the transcript data
   channel during startup with its audio track disabled.
   CEF grants microphone access only to the loopback dictation origin.
3. Status, partial, and final transcript events cross the platform boundary:
   - native CEF encodes each event in the document title and
     `OnTitleChange()` queues it for the caption;
   - WebAssembly uses `window.postMessage()` from a hidden iframe.
4. `main.c` accumulates speech in the scratchpad and submits it to
   `Canvas_Agent_Service` only when the user presses `Enter`. A selected text
   area can enter the same path with `Ctrl/Cmd+Enter`.
   Enter sends a `commit` command through the live transcript data channel, so
   the current utterance finalizes without closing the microphone session.
5. `agent_service_copilot.c` sends the new thought plus
   `Canvas_Scene_Build_Visible_Context()` through the shared asynchronous
   `Inference_Bridge`.
6. The Bazel-managed Copilot SDK sidecar uses the `canvas_orchestrator` profile
   and returns JSON choosing `create` or `append` and a conversation entity ID.
   The canvas materializes that decision as a retained rich conversation card.

The scratchpad is excluded from serialized camera context because the submitted
thought is already sent separately. Only conversation entities currently in
view are valid append targets; moving a conversation out of view removes it
from the orchestrator's routing choices.

## Service boundary

`agent_service_copilot.c` submits work to `Inference_Bridge`, whose worker
thread owns the JSON-line Python sidecar process so SDK latency never blocks
Raylib. The sidecar maintains the persistent
`infinite-canvas-orchestrator` SDK session and uses the loopback LiteLLM
gateway for authenticated GitHub Copilot inference. Only one orchestration
request is admitted at a time in this prototype.

`agent_service_copilot.c` warms that orchestration session plus three reserved
worker sessions during initialization. They share the same
`canvas_orchestrator` profile and move Copilot SDK client/session setup ahead of
the first user turn. The persistent orchestrator is resumed when available;
reserved workers are created fresh because they have no conversation history
to recover, avoiding expected `session.resume` errors during startup.

## Conversation entities

`CANVAS_ENTITY_CONVERSATION` is intentionally more than an editable text area.
It retains:

- a stable entity/session ID used by Copilot routing;
- a title;
- user and agent transcript content;
- asynchronous working state;
- independent scroll, selection, lifecycle, z-order, and pinning state.
- auto-follow to the latest appended turn and a collapsed header-only state;
- a recoverable parked state that is excluded from orchestration context.

Images, browser views, and other components already appear in visible canvas
context as typed neighboring entities. A future resource list can attach those
entity IDs directly to a conversation without replacing the conversation card
or flattening rich content into one string.

Archiving a conversation records its prior position and moves it to the
off-canvas parking lot. The Developer Controls parking-lot button fits the
camera to parked sessions. Restoring a parked card returns it to its prior
position and makes it eligible for routing again when visible.

## Current constraints

- `//infinite_canvas:orchestration_dev` owns the dictation server, LiteLLM
  gateway, and canvas lifecycle. `//infinite_canvas:agent_dev` is an alias.
- One persistent Dictation scratchpad is reused and cleared after submission.
  Submitted turns are retained in conversation entities after orchestration.
- One Copilot request runs at a time.
- Conversation transcript storage is currently bounded by
  `CANVAS_ENTITY_TEXT_CAPACITY`.
- Linux uses the Copilot SDK bridge. Other platforms currently select an
  explicit unavailable stub.