Mercurial

annotate infinite_canvas/README.md @ 280:49e9e591c9bb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
author MrJuneJune <me@mrjunejune.com>
date Tue, 18 Aug 2026 19:14:53 -0700
parents 8d560f50ed4c
children c57149ad216e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
1 # Infinite Canvas
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
2
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
3 Native and WebAssembly Raylib 6 prototype for a world-space, immediate-mode
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
4 infinite canvas.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
5
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
6 ## Build and run
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
7
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
8 ```bash
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
9 # Native development build with Raygui controls
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
10 bazel run //infinite_canvas:dev
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
11
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
12 # Start directly in the ten-browser stress scene
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
13 INFINITE_CANVAS_BROWSER_STRESS=1 bazel run //infinite_canvas:dev
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
14
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
15 # Native build without development UI
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
16 bazel run //infinite_canvas:app
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
17
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
18 # Browser build and local server
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
19 bazel build //infinite_canvas:infinite_canvas_wasm
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
20 bazel run //infinite_canvas:serve
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
21 ```
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
22
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
23 Open `http://127.0.0.1:4343/index.html` after starting the server. Pass a
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
24 different port as an argument only when needed.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
25
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
26 The native development target opens at 1920x1200 and starts with a gallery
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
27 layout containing one non-overlapping example of every entity type.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
28
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
29 Theme mode defaults to `auto`, which follows the OS light/dark preference at
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
30 startup. Rendering consumes a shared semantic `Canvas_Theme` palette for the
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
31 canvas, developer controls, and camera-pinned notifications. Until the planned
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
32 config file lands, `INFINITE_CANVAS_THEME=light|dark|auto` provides an explicit
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
33 override. The developer panel also includes a live light/dark mode toggle.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
34
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
35 ## Controls
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
36
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
37 - Middle-mouse drag, or Space + left-mouse drag: pan
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
38 - Mouse wheel: scroll vertically
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
39 - Ctrl/Cmd + mouse wheel: zoom around the cursor
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
40 - WASD or arrow keys: pan (disabled while a text area has keyboard focus)
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
41 - Left-mouse drag on an entity: select and move it
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
42 - `P`: pin or unpin the selected entity in screen space so it remains visible
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
43 while the camera pans and zooms; pinned entities show a Lucide pin glyph
280
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
44 - `M`: show centered live dictation text and toggle microphone capture
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
45 - `Enter`: send visible dictation text to the spatial agent
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
46 - `Ctrl/Cmd+Enter`: submit the selected text area's thought to Copilot
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
47 - `Delete`/`Backspace`: fade out and remove the selected entity
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
48 - Development toolbar: spawn/clear entities, reset the camera, or restore the
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
49 deterministic demo scene; spawned entities fade in and Clear fades all
280
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
50 entities out before removal. Use the header arrow to collapse the toolbar
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
51 into a small restore button.
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
52 - `View camera context`: open a live, scrollable snapshot of entities
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
53 intersecting the camera viewport and their current semantic values
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
54 - `Lucide icon gallery`: create or revisit a dedicated canvas region containing
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
55 the complete searchable, scrollable Lucide 1.31.0 icon set and fit the camera
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
56 to it
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
57 - `10 browser stress scene`: lay out ten compact independent browser views for
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
58 performance testing
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
59 - `White`/`Grid`: toggle the optional grid; the default canvas is pure white
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
60
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
61 Canvas primitives include shapes, lines, text, buttons, editable text areas,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
62 dropdowns, accordions, cards, calendars, switches, tables, notifications,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
63 scrollable areas, images, and web content. Accordions
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
64 expand with eased height, chevron, divider, and content transitions; cards
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
65 toggle emphasis, calendars select dates, and switches animate their track color
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
66 and thumb position when toggled. The default browser build embeds
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
67 `https://mrjunejune.com` in a camera-tracked iframe. Native builds open
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
68 `https://www.google.com` in a CEF off-screen browser whose pixel buffer is
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
69 uploaded into a Raylib texture. It remains inside the canvas and participates
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
70 in pan, zoom, clipping, and entity dragging on Windows, macOS, and Linux. Web
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
71 views show a Raylib-rendered Lucide back/forward toolbar and current URL only
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
72 while selected. The toolbar eases open by pushing the page viewport downward
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
73 and reverses the transition when deselected. Click the URL field to edit it;
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
74 Enter navigates, Escape cancels, and standard Ctrl/Cmd clipboard and selection
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
75 shortcuts work. Drag the remaining outer edge to move the view and drag the
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
76 bottom-right handle to resize it. Embedded pages inherit the selected canvas
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
77 light or dark color scheme where the browser backend supports it. Multiple web entities have independent
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
78 browser instances, URLs, focus, input, and rendering. Browser resources are
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
79 assigned only to visible entities, off-screen views are suspended, and the
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
80 native CEF pool supports up to twelve live browsers. Small background views
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
81 render at a reduced frame rate, larger views receive a moderate frame budget,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
82 and the focused view runs at 30 FPS so scenes with ten compact browsers remain
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
83 responsive without giving every page the maximum rendering budget. Native
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
84 browser viewport and texture resolution follow the displayed view size in
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
85 64-pixel buckets, capped at 1280x800, so resized views become sharper without
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
86 reallocating continuously during a drag.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
87
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
88 The Lucide gallery is one scalable Raylib-rendered canvas entity rather than
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
89 thousands of individual scene entities or an embedded browser. Reopening it
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
90 reuses the existing entity and moves the camera back to its fitted bounds. Its
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
91 ISC-licensed vector data is generated from Lucide 1.31.0 and the visible grid
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
92 rows are drawn directly with Raylib primitives.
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
93
280
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
94 ## Spatial agent sessions
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
95
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
96 Authenticate the Copilot CLI once:
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
97
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
98 ```bash
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
99 bazel run //mrjunejune:run_inference_stack -- --authenticate
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
100 ```
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
101
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
102 Then start dictation, the Copilot LiteLLM gateway, and the canvas with one
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
103 target:
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
104
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
105 ```bash
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
106 bazel run //infinite_canvas:orchestration_dev
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
107 ```
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
108
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
109 `//infinite_canvas:agent_dev` remains an alias. Closing the canvas stops the
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
110 supervised local services.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
111
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
112 Press `M` to start or stop the local WebRTC microphone session. The browser
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
113 transport is prewarmed and remains hidden: it acquires the native microphone
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
114 permission, device, signaling, ICE connection, and transcript data channel
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
115 ahead of the first hotkey while keeping the audio track disabled. `M` therefore
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
116 only unmutes an already-ready channel, so speech from the beginning of the
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
117 utterance is retained. Partial and final speech appear directly as minimal centered text
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
118 inside a retained `Dictation` text-area entity spawned near the camera center.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
119 The entity word-wraps, scrolls to the latest line, and can be dragged by its
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
120 header to any position on the board. The same scratchpad is reused for every
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
121 thought and clears after submission; it does not create duplicate dictation
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
122 windows. A small pulsing `Listening` indicator
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
123 remains in the bottom-left while capture is active. In-progress speech uses
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
124 muted text; `Backspace` clears the entire current thought without stopping
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
125 capture. Pausing with `M` and resuming appends to the same thought rather than
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
126 replacing it. Press `Enter` to commit and send the current utterance while the
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
127 microphone stays live for the next thought; press `M` or `Escape` to stop it.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
128 For typed input, select a text area and press `Ctrl/Cmd+Enter`.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
129 Native CEF grants audio only to the fixed loopback dictation transport; it does
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
130 not show an inaccessible browser permission prompt or grant microphone access
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
131 to normal canvas web entities.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
132
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
133 The Copilot SDK receives the visible camera context, including existing
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
134 conversation entity IDs, through the shared `canvas_orchestrator` prompt
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
135 profile. It returns a routing decision to append to an existing session or
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
136 create a new one. Requests run off the render thread through the shared
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
137 `Inference_Bridge`. Conversation cards retain
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
138 their own title, transcript, working state, scroll position, selection, and
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
139 pinning state, forming the first rich-context container for future typed image
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
140 and component resources. Runtime configuration and Copilot token paths come
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
141 from the ignored `mrjunejune/.config`; no separate canvas environment file is
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
142 required. The Dictation scratchpad is excluded from camera context because its
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
143 text is submitted separately, and only visible conversation cards are eligible
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
144 append targets. Conversation cards follow the latest appended message and can
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
145 be collapsed from their header. The archive icon moves a session to the
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
146 off-canvas parking lot, excluding it from context; `Parking lot (N)` in
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
147 Developer Controls jumps there, and the restore icon returns the session to
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
148 its previous board position. Collapse and expansion use the same eased retained
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
149 animation model as accordion entities rather than snapping between heights.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
150 Canvas startup also pre-creates the persistent orchestration SDK session and
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
151 three worker sessions, moving SDK session creation out of the first submitted
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
152 thought.
49e9e591c9bb Add persistent dictation, prewarmed WebRTC speech input, Copilot SDK routing, animated conversation lifecycle controls, parking, and architecture coverage.
MrJuneJune <me@mrjunejune.com>
parents: 278
diff changeset
153
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
154 Tables support row selection, and notifications behave as bottom-right toast
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
155 cards: multiple toasts overlap compactly, fan into a spaced stack on hover,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
156 and slide/fade away from either Undo or close. Position, size, stack expansion,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
157 and dismissal all use eased frame-rate-independent transitions. Scrollable areas consume the
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
158 mouse wheel without zooming the camera. Image
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
159 entities render HTTPS URLs or native filesystem paths through CEF; browser
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
160 builds render URLs or paths available in the Emscripten filesystem. Override
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
161 the demo image in native development with
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
162 `INFINITE_CANVAS_IMAGE_SOURCE=/path/to/image.png`.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
163
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
164 The first native build downloads the platform CEF 151 minimal distribution.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
165 Linux stages its runtime once under
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
166 `~/.cache/zenbu/infinite-canvas/cef-151`; application state and the RGBA paint
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
167 buffer remain owned by the Dowa arena, while CEF owns its browser internals.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
168
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
169 Buttons animate on hover/press and toggle when clicked. Dropdowns open a real
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
170 option menu with hover and selection states. Selected text areas accept typing,
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
171 wrap text, support click/drag selection and scrolling, and provide
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
172 Ctrl/Cmd+A/C/X/V clipboard shortcuts plus Shift selection, word navigation,
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
173 Home/End, Backspace, and Delete. Drag a text area's narrow outer padding to
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
174 move it without changing the selection.
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
175
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
176 Camera context is derived from the visible world rather than a hidden
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
177 conversation store. Panning entities outside the viewport removes them;
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
178 zooming out includes and concatenates newly visible entities. Each entry
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
179 contains identity, type, placement, pin state, and type-specific state such as
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
180 text content, dropdown selection, switch value, selected calendar date or table
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
181 row, scroll position, image source, and browser URL. The current text format is
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
182 intentionally provisional so it can later be replaced by the final API
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
183 contract.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
184
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
185 The prototype uses the Inter variable font under the SIL Open Font License.
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
186
278
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
187 Agent-oriented architecture and extension notes live in
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
188 [`docs/README.md`](docs/README.md).
8d560f50ed4c Improve infinite canvas interactions and browser chrome
MrJuneJune <me@mrjunejune.com>
parents: 276
diff changeset
189
276
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
190 ## Memory ownership
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
191
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
192 The application creates one Dowa arena. Application state and the fixed-capacity
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
193 entity array are arena-owned, there are no per-frame first-party allocations,
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
194 and the arena is released once during native shutdown. Browser lifetime memory
b55c22cff335 Add interactive infinite canvas prototype
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
195 is reclaimed with the WebAssembly runtime when the page closes.