Zenbu

Dictation and Spatial Agent Sessions
Login

Dictation and Spatial Agent Sessions

Agent event inspector

Both production and development open a modal agent event inspector with Primary+Shift+I; production does not expose the rest of the developer toolbar. It retains the latest 512 chronological events, including outbound warm/turn calls, every inbound bridge event, request and conversation IDs, token counts, failures, and the canvas routing decisions made after a structured response is parsed. Copy all copies the event buffer directly, Download JSONL writes the same ordered log to the local Yuu state directory, and Clear log resets only this diagnostic ring buffer. Inspector and camera-context text use the same read-only selection behavior: click-drag highlights text, Ctrl/Cmd+A selects all, and Ctrl/Cmd+C copies it. The default layout keeps the active system prompt and current visible canvas context above the event stream. Persistent SDK memory is currently disabled and is labeled as such. User-authored persistent memory is separate: the pinned brain entity expands into an editable panel, and Save writes agent-memory.txt under the local Infinite Canvas state directory. Only the saved version is injected into subsequent turns, inside an explicitly untrusted user-memory section. Spatial Clear and scene snapshots do not delete or replace it. The active system prompt remains read-only.

Prototype flow

The first orchestration slice connects three retained systems:

  1. Primary+Shift+D reuses the single retained Dictation text-area scratchpad and toggles a hidden browser transport for http://127.0.0.1:8090/?canvas=1.
  2. Canvas_Web_Surface_Set_Dictation_Active() deterministically starts or stops capture on an already-negotiated WebRTC client. The hidden transport acquires its stream, completes signaling/ICE, and opens the transcript data channel during startup with its audio track disabled. CEF grants microphone access only to the loopback dictation origin.
  3. Status, partial, and final transcript events cross the platform boundary:
    • native CEF encodes each event in the document title and OnTitleChange() queues it for the caption;
    • WebAssembly uses window.postMessage() from a hidden iframe.
  4. main.c accumulates speech in the scratchpad and submits it to Canvas_Agent_Service only when the user presses Enter. A selected text area can enter the same path with Ctrl/Cmd+Enter. Enter sends a commit command through the live transcript data channel, so the current utterance finalizes without closing the microphone session.
  5. agent_service_copilot.c sends the new thought plus the saved user memory and Canvas_Scene_Build_Visible_Context() through the shared asynchronous Inference_Bridge.
  6. The Bazel-managed Copilot SDK sidecar uses the canvas_orchestrator profile and returns JSON choosing action or conversation presentation, create or append routing, a conversation entity ID, and optional typed showcase entities. The canvas materializes only the requested surface.

Up to four submissions may be active. The current pulsing Raylib/Lucide indicator shows the active request count; hovering reveals its working label, and dragging it changes the world position for that request's eventual result. A one-off action replaces it with the resulting entity or compact notification and does not create another agent/session. Complex work creates a retained conversation card after orchestration completes; continuations append to an already-visible conversation.

The scratchpad is excluded from serialized camera context because the submitted thought is already sent separately. Only conversation entities currently in view are valid append targets; moving a conversation out of view removes it from the orchestrator's routing choices.

Every visible entity carries a bounded context_weight from 0 through 3. Primary+W cycles the selected entity and draws a small Lucide weight badge. Visibility still determines what is sent. Explicit user references always win; otherwise a uniquely highest-weight relevant conversation becomes the preferred continuation and supplies its persistent SDK session ID. Equal top weights intentionally remain ambiguous. Explicit parallel/separate-task language bypasses continuation and allocates an independent session.

Service boundary

agent_service_copilot.c submits work to Inference_Bridge, whose reader thread owns events from the JSON-line Python sidecar so SDK latency never blocks Raylib. Four fixed native request slots accept independent turns. Each retained conversation stores its own Copilot SDK session ID, so unrelated conversations can run concurrently while the sidecar's per-conversation lock keeps same-session transcript order deterministic.

The adapter warms four orchestration-capable sessions during initialization. New standalone tasks receive a unique request session; once a conversation entity is created, that session ID is retained with the entity and reused for weighted continuations. SDK permission requests use approve-all handling so sessions do not stop for interactive approval. Canvas still exposes no SDK tools by default; autonomy applies to the bounded orchestration surface rather than unrestricted host access. The LiteLLM-backed route currently uses GPT-5.4 because LiteLLM's open GitHub Copilot issue #33893 sends GPT-5.6 to the incompatible Chat Completions endpoint; redundant provider retries are disabled until a direct Responses route is available.

Production restart behavior

//yuu:agent restores workspace.zmap from the machine-local Yuu state directory. Changed user-owned scene state is atomically saved every five seconds and flushed once more during graceful shutdown. Conversation transcripts, geometry, pinning, context weights, and SDK session IDs are part of that snapshot. Camera target, zoom, viewport dimensions, and pinned screen-space bounds are restored together so reopening does not reframe the world or resize pinned entities. Spatial catalog entities remain owned by canvas.db, while Agent Memory and keyboard bindings keep their separate files.

Development mode uses a separate canvas-dev.db plus dev-* workspace, Memory, shortcut, event-log, browser-profile, and Copilot-marker namespaces. Only the Copilot authentication token is shared with production.

An in-flight native request is not serialized. Its transient working flag and response buffer disappear on restart, while the last completed transcript and session ID remain. The next routed turn lazily asks the sidecar to resume that SDK session; if the SDK no longer has it, the sidecar creates a replacement using the same deterministic ID.

Conversation entities

CANVAS_ENTITY_CONVERSATION is intentionally more than an editable text area. It retains:

Images, browser views, and other components already appear in visible canvas context as typed neighboring entities. A future resource list can attach those entity IDs directly to a conversation without replacing the conversation card or flattening rich content into one string.

Archiving a conversation records its prior position and moves it to the off-canvas parking lot. The Developer Controls parking-lot button fits the camera to parked sessions. Restoring a parked card returns it to its prior position and makes it eligible for routing again when visible.

Current constraints