diff infinite_canvas/README.md @ 280:49e9e591c9bb

Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
author MrJuneJune <me@mrjunejune.com>
date Tue, 18 Aug 2026 19:14:53 -0700
parents 8d560f50ed4c
children c57149ad216e
line wrap: on
line diff
--- a/infinite_canvas/README.md	Mon Aug 17 22:22:36 2026 -0700
+++ b/infinite_canvas/README.md	Tue Aug 18 19:14:53 2026 -0700
@@ -41,10 +41,14 @@
 - Left-mouse drag on an entity: select and move it
 - `P`: pin or unpin the selected entity in screen space so it remains visible
   while the camera pans and zooms; pinned entities show a Lucide pin glyph
+- `M`: show centered live dictation text and toggle microphone capture
+- `Enter`: send visible dictation text to the spatial agent
+- `Ctrl/Cmd+Enter`: submit the selected text area's thought to Copilot
 - `Delete`/`Backspace`: fade out and remove the selected entity
 - Development toolbar: spawn/clear entities, reset the camera, or restore the
   deterministic demo scene; spawned entities fade in and Clear fades all
-  entities out before removal
+  entities out before removal. Use the header arrow to collapse the toolbar
+  into a small restore button.
 - `View camera context`: open a live, scrollable snapshot of entities
   intersecting the camera viewport and their current semantic values
 - `Lucide icon gallery`: create or revisit a dedicated canvas region containing
@@ -87,6 +91,66 @@
 ISC-licensed vector data is generated from Lucide 1.31.0 and the visible grid
 rows are drawn directly with Raylib primitives.
 
+## Spatial agent sessions
+
+Authenticate the Copilot CLI once:
+
+```bash
+bazel run //mrjunejune:run_inference_stack -- --authenticate
+```
+
+Then start dictation, the Copilot LiteLLM gateway, and the canvas with one
+target:
+
+```bash
+bazel run //infinite_canvas:orchestration_dev
+```
+
+`//infinite_canvas:agent_dev` remains an alias. Closing the canvas stops the
+supervised local services.
+
+Press `M` to start or stop the local WebRTC microphone session. The browser
+transport is prewarmed and remains hidden: it acquires the native microphone
+permission, device, signaling, ICE connection, and transcript data channel
+ahead of the first hotkey while keeping the audio track disabled. `M` therefore
+only unmutes an already-ready channel, so speech from the beginning of the
+utterance is retained. Partial and final speech appear directly as minimal centered text
+inside a retained `Dictation` text-area entity spawned near the camera center.
+The entity word-wraps, scrolls to the latest line, and can be dragged by its
+header to any position on the board. The same scratchpad is reused for every
+thought and clears after submission; it does not create duplicate dictation
+windows. A small pulsing `Listening` indicator
+remains in the bottom-left while capture is active. In-progress speech uses
+muted text; `Backspace` clears the entire current thought without stopping
+capture. Pausing with `M` and resuming appends to the same thought rather than
+replacing it. Press `Enter` to commit and send the current utterance while the
+microphone stays live for the next thought; press `M` or `Escape` to stop it.
+For typed input, select a text area and press `Ctrl/Cmd+Enter`.
+Native CEF grants audio only to the fixed loopback dictation transport; it does
+not show an inaccessible browser permission prompt or grant microphone access
+to normal canvas web entities.
+
+The Copilot SDK receives the visible camera context, including existing
+conversation entity IDs, through the shared `canvas_orchestrator` prompt
+profile. It returns a routing decision to append to an existing session or
+create a new one. Requests run off the render thread through the shared
+`Inference_Bridge`. Conversation cards retain
+their own title, transcript, working state, scroll position, selection, and
+pinning state, forming the first rich-context container for future typed image
+and component resources. Runtime configuration and Copilot token paths come
+from the ignored `mrjunejune/.config`; no separate canvas environment file is
+required. The Dictation scratchpad is excluded from camera context because its
+text is submitted separately, and only visible conversation cards are eligible
+append targets. Conversation cards follow the latest appended message and can
+be collapsed from their header. The archive icon moves a session to the
+off-canvas parking lot, excluding it from context; `Parking lot (N)` in
+Developer Controls jumps there, and the restore icon returns the session to
+its previous board position. Collapse and expansion use the same eased retained
+animation model as accordion entities rather than snapping between heights.
+Canvas startup also pre-creates the persistent orchestration SDK session and
+three worker sessions, moving SDK session creation out of the first submitted
+thought.
+
 Tables support row selection, and notifications behave as bottom-right toast
 cards: multiple toasts overlap compactly, fan into a spaced stack on hover,
 and slide/fade away from either Undo or close. Position, size, stack expansion,