Mercurial
diff infinite_canvas/README.md @ 280:49e9e591c9bb
Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
| author | MrJuneJune <me@mrjunejune.com> |
|---|---|
| date | Tue, 18 Aug 2026 19:14:53 -0700 |
| parents | 8d560f50ed4c |
| children | c57149ad216e |
line wrap: on
line diff
--- a/infinite_canvas/README.md Mon Aug 17 22:22:36 2026 -0700 +++ b/infinite_canvas/README.md Tue Aug 18 19:14:53 2026 -0700 @@ -41,10 +41,14 @@ - Left-mouse drag on an entity: select and move it - `P`: pin or unpin the selected entity in screen space so it remains visible while the camera pans and zooms; pinned entities show a Lucide pin glyph +- `M`: show centered live dictation text and toggle microphone capture +- `Enter`: send visible dictation text to the spatial agent +- `Ctrl/Cmd+Enter`: submit the selected text area's thought to Copilot - `Delete`/`Backspace`: fade out and remove the selected entity - Development toolbar: spawn/clear entities, reset the camera, or restore the deterministic demo scene; spawned entities fade in and Clear fades all - entities out before removal + entities out before removal. Use the header arrow to collapse the toolbar + into a small restore button. - `View camera context`: open a live, scrollable snapshot of entities intersecting the camera viewport and their current semantic values - `Lucide icon gallery`: create or revisit a dedicated canvas region containing @@ -87,6 +91,66 @@ ISC-licensed vector data is generated from Lucide 1.31.0 and the visible grid rows are drawn directly with Raylib primitives. +## Spatial agent sessions + +Authenticate the Copilot CLI once: + +```bash +bazel run //mrjunejune:run_inference_stack -- --authenticate +``` + +Then start dictation, the Copilot LiteLLM gateway, and the canvas with one +target: + +```bash +bazel run //infinite_canvas:orchestration_dev +``` + +`//infinite_canvas:agent_dev` remains an alias. Closing the canvas stops the +supervised local services. + +Press `M` to start or stop the local WebRTC microphone session. The browser +transport is prewarmed and remains hidden: it acquires the native microphone +permission, device, signaling, ICE connection, and transcript data channel +ahead of the first hotkey while keeping the audio track disabled. `M` therefore +only unmutes an already-ready channel, so speech from the beginning of the +utterance is retained. Partial and final speech appear directly as minimal centered text +inside a retained `Dictation` text-area entity spawned near the camera center. +The entity word-wraps, scrolls to the latest line, and can be dragged by its +header to any position on the board. The same scratchpad is reused for every +thought and clears after submission; it does not create duplicate dictation +windows. A small pulsing `Listening` indicator +remains in the bottom-left while capture is active. In-progress speech uses +muted text; `Backspace` clears the entire current thought without stopping +capture. Pausing with `M` and resuming appends to the same thought rather than +replacing it. Press `Enter` to commit and send the current utterance while the +microphone stays live for the next thought; press `M` or `Escape` to stop it. +For typed input, select a text area and press `Ctrl/Cmd+Enter`. +Native CEF grants audio only to the fixed loopback dictation transport; it does +not show an inaccessible browser permission prompt or grant microphone access +to normal canvas web entities. + +The Copilot SDK receives the visible camera context, including existing +conversation entity IDs, through the shared `canvas_orchestrator` prompt +profile. It returns a routing decision to append to an existing session or +create a new one. Requests run off the render thread through the shared +`Inference_Bridge`. Conversation cards retain +their own title, transcript, working state, scroll position, selection, and +pinning state, forming the first rich-context container for future typed image +and component resources. Runtime configuration and Copilot token paths come +from the ignored `mrjunejune/.config`; no separate canvas environment file is +required. The Dictation scratchpad is excluded from camera context because its +text is submitted separately, and only visible conversation cards are eligible +append targets. Conversation cards follow the latest appended message and can +be collapsed from their header. The archive icon moves a session to the +off-canvas parking lot, excluding it from context; `Parking lot (N)` in +Developer Controls jumps there, and the restore icon returns the session to +its previous board position. Collapse and expansion use the same eased retained +animation model as accordion entities rather than snapping between heights. +Canvas startup also pre-creates the persistent orchestration SDK session and +three worker sessions, moving SDK session creation out of the first submitted +thought. + Tables support row selection, and notifications behave as bottom-right toast cards: multiple toasts overlap compactly, fan into a spaced stack on hover, and slide/fade away from either Undo or close. Position, size, stack expansion,