Mercurial

comparison infinite_canvas/README.md @ 280:49e9e591c9bb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
author MrJuneJune <me@mrjunejune.com>
date Tue, 18 Aug 2026 19:14:53 -0700
parents 8d560f50ed4c
children c57149ad216e
comparison
equal deleted inserted replaced
279:b3b547563ec7 280:49e9e591c9bb
39 - Ctrl/Cmd + mouse wheel: zoom around the cursor 39 - Ctrl/Cmd + mouse wheel: zoom around the cursor
40 - WASD or arrow keys: pan (disabled while a text area has keyboard focus) 40 - WASD or arrow keys: pan (disabled while a text area has keyboard focus)
41 - Left-mouse drag on an entity: select and move it 41 - Left-mouse drag on an entity: select and move it
42 - `P`: pin or unpin the selected entity in screen space so it remains visible 42 - `P`: pin or unpin the selected entity in screen space so it remains visible
43 while the camera pans and zooms; pinned entities show a Lucide pin glyph 43 while the camera pans and zooms; pinned entities show a Lucide pin glyph
44 - `M`: show centered live dictation text and toggle microphone capture
45 - `Enter`: send visible dictation text to the spatial agent
46 - `Ctrl/Cmd+Enter`: submit the selected text area's thought to Copilot
44 - `Delete`/`Backspace`: fade out and remove the selected entity 47 - `Delete`/`Backspace`: fade out and remove the selected entity
45 - Development toolbar: spawn/clear entities, reset the camera, or restore the 48 - Development toolbar: spawn/clear entities, reset the camera, or restore the
46 deterministic demo scene; spawned entities fade in and Clear fades all 49 deterministic demo scene; spawned entities fade in and Clear fades all
47 entities out before removal 50 entities out before removal. Use the header arrow to collapse the toolbar
51 into a small restore button.
48 - `View camera context`: open a live, scrollable snapshot of entities 52 - `View camera context`: open a live, scrollable snapshot of entities
49 intersecting the camera viewport and their current semantic values 53 intersecting the camera viewport and their current semantic values
50 - `Lucide icon gallery`: create or revisit a dedicated canvas region containing 54 - `Lucide icon gallery`: create or revisit a dedicated canvas region containing
51 the complete searchable, scrollable Lucide 1.31.0 icon set and fit the camera 55 the complete searchable, scrollable Lucide 1.31.0 icon set and fit the camera
52 to it 56 to it
84 The Lucide gallery is one scalable Raylib-rendered canvas entity rather than 88 The Lucide gallery is one scalable Raylib-rendered canvas entity rather than
85 thousands of individual scene entities or an embedded browser. Reopening it 89 thousands of individual scene entities or an embedded browser. Reopening it
86 reuses the existing entity and moves the camera back to its fitted bounds. Its 90 reuses the existing entity and moves the camera back to its fitted bounds. Its
87 ISC-licensed vector data is generated from Lucide 1.31.0 and the visible grid 91 ISC-licensed vector data is generated from Lucide 1.31.0 and the visible grid
88 rows are drawn directly with Raylib primitives. 92 rows are drawn directly with Raylib primitives.
93
94 ## Spatial agent sessions
95
96 Authenticate the Copilot CLI once:
97
98 ```bash
99 bazel run //mrjunejune:run_inference_stack -- --authenticate
100 ```
101
102 Then start dictation, the Copilot LiteLLM gateway, and the canvas with one
103 target:
104
105 ```bash
106 bazel run //infinite_canvas:orchestration_dev
107 ```
108
109 `//infinite_canvas:agent_dev` remains an alias. Closing the canvas stops the
110 supervised local services.
111
112 Press `M` to start or stop the local WebRTC microphone session. The browser
113 transport is prewarmed and remains hidden: it acquires the native microphone
114 permission, device, signaling, ICE connection, and transcript data channel
115 ahead of the first hotkey while keeping the audio track disabled. `M` therefore
116 only unmutes an already-ready channel, so speech from the beginning of the
117 utterance is retained. Partial and final speech appear directly as minimal centered text
118 inside a retained `Dictation` text-area entity spawned near the camera center.
119 The entity word-wraps, scrolls to the latest line, and can be dragged by its
120 header to any position on the board. The same scratchpad is reused for every
121 thought and clears after submission; it does not create duplicate dictation
122 windows. A small pulsing `Listening` indicator
123 remains in the bottom-left while capture is active. In-progress speech uses
124 muted text; `Backspace` clears the entire current thought without stopping
125 capture. Pausing with `M` and resuming appends to the same thought rather than
126 replacing it. Press `Enter` to commit and send the current utterance while the
127 microphone stays live for the next thought; press `M` or `Escape` to stop it.
128 For typed input, select a text area and press `Ctrl/Cmd+Enter`.
129 Native CEF grants audio only to the fixed loopback dictation transport; it does
130 not show an inaccessible browser permission prompt or grant microphone access
131 to normal canvas web entities.
132
133 The Copilot SDK receives the visible camera context, including existing
134 conversation entity IDs, through the shared `canvas_orchestrator` prompt
135 profile. It returns a routing decision to append to an existing session or
136 create a new one. Requests run off the render thread through the shared
137 `Inference_Bridge`. Conversation cards retain
138 their own title, transcript, working state, scroll position, selection, and
139 pinning state, forming the first rich-context container for future typed image
140 and component resources. Runtime configuration and Copilot token paths come
141 from the ignored `mrjunejune/.config`; no separate canvas environment file is
142 required. The Dictation scratchpad is excluded from camera context because its
143 text is submitted separately, and only visible conversation cards are eligible
144 append targets. Conversation cards follow the latest appended message and can
145 be collapsed from their header. The archive icon moves a session to the
146 off-canvas parking lot, excluding it from context; `Parking lot (N)` in
147 Developer Controls jumps there, and the restore icon returns the session to
148 its previous board position. Collapse and expansion use the same eased retained
149 animation model as accordion entities rather than snapping between heights.
150 Canvas startup also pre-creates the persistent orchestration SDK session and
151 three worker sessions, moving SDK session creation out of the first submitted
152 thought.
89 153
90 Tables support row selection, and notifications behave as bottom-right toast 154 Tables support row selection, and notifications behave as bottom-right toast
91 cards: multiple toasts overlap compactly, fan into a spaced stack on hover, 155 cards: multiple toasts overlap compactly, fan into a spaced stack on hover,
92 and slide/fade away from either Undo or close. Position, size, stack expansion, 156 and slide/fade away from either Undo or close. Position, size, stack expansion,
93 and dismissal all use eased frame-rate-independent transitions. Scrollable areas consume the 157 and dismissal all use eased frame-rate-independent transitions. Scrollable areas consume the