# Dictation and Spatial Agent Sessions

## Prototype flow

The first orchestration slice connects three retained systems:

1. `M` reuses the single retained `Dictation` text-area scratchpad and toggles
   a hidden browser transport for `http://127.0.0.1:8090/?canvas=1`.
2. `Canvas_Web_Surface_Set_Dictation_Active()` deterministically starts or
   stops capture on an already-negotiated WebRTC client. The hidden transport
   acquires its stream, completes signaling/ICE, and opens the transcript data
   channel during startup with its audio track disabled.
   CEF grants microphone access only to the loopback dictation origin.
3. Status, partial, and final transcript events cross the platform boundary:
   - native CEF encodes each event in the document title and
     `OnTitleChange()` queues it for the caption;
   - WebAssembly uses `window.postMessage()` from a hidden iframe.
4. `main.c` accumulates speech in the scratchpad and submits it to
   `Canvas_Agent_Service` only when the user presses `Enter`. A selected text
   area can enter the same path with `Ctrl/Cmd+Enter`.
   Enter sends a `commit` command through the live transcript data channel, so
   the current utterance finalizes without closing the microphone session.
5. `agent_service_copilot.c` sends the new thought plus
   `Canvas_Scene_Build_Visible_Context()` through the shared asynchronous
   `Inference_Bridge`.
6. The Bazel-managed Copilot SDK sidecar uses the `canvas_orchestrator` profile
   and returns JSON choosing `action` or `conversation` presentation,
   `create` or `append` routing, a conversation entity ID, and optional typed
   showcase entities. The canvas materializes only the requested surface.

All submissions begin as one small pulsing Raylib/Lucide indicator anchored to
the board. Hovering reveals its working label, and dragging it changes the
world position where an eventual result or conversation is materialized. A
one-off action replaces it with the resulting entity or compact notification
and does not create another agent/session. Complex work creates a retained
conversation card after orchestration completes; continuations append to an
already-visible conversation.

The scratchpad is excluded from serialized camera context because the submitted
thought is already sent separately. Only conversation entities currently in
view are valid append targets; moving a conversation out of view removes it
from the orchestrator's routing choices.

## Service boundary

`agent_service_copilot.c` submits work to `Inference_Bridge`, whose worker
thread owns the JSON-line Python sidecar process so SDK latency never blocks
Raylib. The sidecar maintains the persistent
`infinite-canvas-orchestrator` SDK session and uses the loopback LiteLLM
gateway for authenticated GitHub Copilot inference. Only one orchestration
request is admitted at a time in this prototype.

`agent_service_copilot.c` warms only the persistent orchestration session
during initialization. This moves Copilot SDK client/session setup ahead of the
first user turn without creating unused worker sessions. The orchestrator is
resumed when available.

## Conversation entities

`CANVAS_ENTITY_CONVERSATION` is intentionally more than an editable text area.
It retains:

- a stable entity/session ID used by Copilot routing;
- a title;
- user and agent transcript content;
- asynchronous working state;
- independent scroll, selection, lifecycle, z-order, and pinning state.
- auto-follow to the latest appended turn and a collapsed header-only state;
- a recoverable parked state that is excluded from orchestration context.

Images, browser views, and other components already appear in visible canvas
context as typed neighboring entities. A future resource list can attach those
entity IDs directly to a conversation without replacing the conversation card
or flattening rich content into one string.

Archiving a conversation records its prior position and moves it to the
off-canvas parking lot. The Developer Controls parking-lot button fits the
camera to parked sessions. Restoring a parked card returns it to its prior
position and makes it eligible for routing again when visible.

## Current constraints

- `//infinite_canvas:orchestration_dev` owns the dictation server, LiteLLM
  gateway, and canvas lifecycle. `//infinite_canvas:agent_dev` is an alias.
- One persistent Dictation scratchpad is reused and cleared after submission.
  Submitted turns are retained in conversation entities after orchestration.
- One Copilot request runs at a time.
- Simple `action` presentation stays in the primary orchestrator and does not
  create a conversation card, worker, or sub-session.
- Agent-created entities pass through a strict type allowlist and count bound.
  Browser and image entities accept only absolute HTTP(S) URLs.
- Conversation transcript storage is currently bounded by
  `CANVAS_ENTITY_TEXT_CAPACITY`.
- Linux uses the Copilot SDK bridge. Other platforms currently select an
  explicit unavailable stub.