Zenbu

Documentation
Login

Documentation

title: Spatial persistence and hydration status: partially implemented audience:

Spatial persistence and hydration

Purpose

Yuu should become a local-first spatial context system rather than a process that loads one complete scene into memory. The camera is both a viewport and an AI-context lens:

SQLite is the first source of truth because it is portable, transactional, fast on local machines, and easy to move as one artifact. The storage API must not expose SQLite details to canvas interaction or rendering code, so a hosted PostgreSQL/PostGIS implementation can implement the same contract later.

Implementation status

The first isolated storage slice is implemented in //yuu:canvas_store_sqlite:

The native //yuu:dev target now provides the first frame-loop integration:

This is intentionally a development proving path. Runtime entity edits are not yet synchronized back to stable catalog rows, and each refresh rebuilds the small resident scene synchronously. Asynchronous query generations, typed payload hydration, .zmap migration, and database backup/export remain planned phases below.

Goals

Non-goals for the first implementation

Core model

The persistent catalog and the resident Raylib scene are different layers:

SQLite workspace database
  ├── workspace and camera checkpoints
  ├── entity catalog and stable IDs
  ├── R-tree world bounds
  ├── summaries and hydration descriptors
  ├── type-specific payloads or external references
  └── semantic regions and relationships
             │ viewport query
             ▼
Resident set
  ├── nearby catalog entries
  ├── visible summary/full entities
  └── active external resources
             │ materialize
             ▼
Canvas_Scene
  └── hot Raylib/CEF entities used by the current frame

Canvas_Scene should remain the hot retained render state. It should not become the durable database model or contain every entity in an unbounded workspace.

Stable identity

Separate three identities:

Identity Scope Purpose
entity_pk One SQLite database Integer key required by SQLite R-tree
entity_id Stable across exports/imports Public opaque 128-bit ID
domain key Entity-type-specific Conversation, asset, document, or component ID

The current numeric Canvas_Entity.id is a process-local rendering handle. During migration it may temporarily mirror entity_pk, but persisted relationships and agent contracts must use stable entity_id or a typed domain key.

Proposed SQLite schema

The exact migration SQL belongs beside the future store implementation. This shape defines ownership and query behavior:

CREATE TABLE schema_meta (
  schema_version INTEGER NOT NULL,
  created_at_ms INTEGER NOT NULL,
  application_id TEXT NOT NULL
);

CREATE TABLE workspace (
  workspace_id BLOB PRIMARY KEY,
  title TEXT NOT NULL,
  revision INTEGER NOT NULL,
  created_at_ms INTEGER NOT NULL,
  updated_at_ms INTEGER NOT NULL
);

CREATE TABLE camera_checkpoint (
  workspace_id BLOB NOT NULL,
  checkpoint_name TEXT NOT NULL,
  target_x REAL NOT NULL,
  target_y REAL NOT NULL,
  zoom REAL NOT NULL,
  viewport_width INTEGER NOT NULL,
  viewport_height INTEGER NOT NULL,
  updated_at_ms INTEGER NOT NULL,
  PRIMARY KEY (workspace_id, checkpoint_name)
);

CREATE TABLE entity (
  entity_pk INTEGER PRIMARY KEY,
  entity_id BLOB NOT NULL UNIQUE,
  workspace_id BLOB NOT NULL,
  entity_type INTEGER NOT NULL,
  domain_kind INTEGER NOT NULL,
  domain_id BLOB,
  title TEXT NOT NULL DEFAULT '',
  summary TEXT NOT NULL DEFAULT '',
  z_order INTEGER NOT NULL,
  flags INTEGER NOT NULL,
  payload_kind INTEGER NOT NULL,
  payload_version INTEGER NOT NULL,
  payload_bytes INTEGER NOT NULL DEFAULT 0,
  external_uri TEXT,
  revision INTEGER NOT NULL,
  created_at_ms INTEGER NOT NULL,
  updated_at_ms INTEGER NOT NULL
);

CREATE VIRTUAL TABLE entity_bounds USING rtree(
  entity_pk,
  min_x, max_x,
  min_y, max_y
);

CREATE TABLE entity_runtime_state (
  entity_pk INTEGER PRIMARY KEY,
  active INTEGER NOT NULL,
  value INTEGER NOT NULL,
  updated_at_ms INTEGER NOT NULL,
  FOREIGN KEY (entity_pk) REFERENCES entity(entity_pk) ON DELETE CASCADE
);

CREATE TABLE entity_payload (
  entity_pk INTEGER PRIMARY KEY,
  codec INTEGER NOT NULL,
  payload BLOB NOT NULL,
  checksum BLOB NOT NULL,
  FOREIGN KEY (entity_pk) REFERENCES entity(entity_pk) ON DELETE CASCADE
);

CREATE TABLE region (
  region_id BLOB PRIMARY KEY,
  workspace_id BLOB NOT NULL,
  title TEXT NOT NULL,
  summary TEXT NOT NULL,
  instructions TEXT NOT NULL DEFAULT '',
  revision INTEGER NOT NULL
);

CREATE TABLE region_entity (
  region_id BLOB NOT NULL,
  entity_id BLOB NOT NULL,
  role INTEGER NOT NULL,
  PRIMARY KEY (region_id, entity_id)
);

The vendored SQLite target enables SQLITE_ENABLE_RTREE, and focused tests create/query the virtual table through Deita. Database access should use //deita:deita; missing backup or transaction features should be added to Deita rather than bypassing the first-party abstraction throughout Yuu.

Viewport query

The basic intersection query returns catalog rows, not full payloads:

SELECT
  e.entity_pk,
  e.entity_id,
  e.entity_type,
  e.domain_kind,
  e.domain_id,
  e.title,
  e.summary,
  e.z_order,
  e.flags,
  e.payload_kind,
  e.payload_version,
  e.payload_bytes,
  e.external_uri,
  b.min_x,
  b.max_x,
  b.min_y,
  b.max_y
FROM entity_bounds AS b
JOIN entity AS e ON e.entity_pk = b.entity_pk
WHERE e.workspace_id = ?
  AND b.max_x >= ?
  AND b.min_x <= ?
  AND b.max_y >= ?
  AND b.min_y <= ?
ORDER BY e.z_order;

Query a preload rectangle larger than the visible viewport. The visible viewport is the hot region; the surrounding margin is warm metadata. A query should run when the camera exits an inner guard rectangle, changes zoom enough to alter desired detail, or the workspace revision changes. It should not run for every subpixel camera movement.

Each asynchronous query carries a generation number. Results from an older camera generation are discarded rather than materialized after the user has moved elsewhere.

Residency states

An entity progresses independently through these states:

ABSENT
  -> CATALOG
  -> SUMMARY
  -> FULL
  -> ACTIVE_RESOURCE

Any loading state may become FAILED with an explicit retryable error.
State Resident data
ABSENT Stable ID may be known through a relationship only
CATALOG Type, bounds, title, flags, payload descriptor
SUMMARY AI-readable summary and cheap visual placeholder
FULL Complete persistent payload required by the entity
ACTIVE_RESOURCE Runtime-only CEF browser, GPU texture, SDK handle, or stream

Downgrading releases runtime resources first, then full payloads. Catalog rows remain cheap and may stay cached across nearby camera movement.

Hydration contract

Hydration must be selected by entity type and payload kind, not implemented as one generic JSON decoder:

typedef struct {
  boolean (*load_summary)(...);
  boolean (*load_full)(...);
  boolean (*activate)(...);
  void (*deactivate)(...);
  void (*release_full)(...);
  boolean (*append_context)(...);
} Canvas_Entity_Hydrator;

The eventual API should be exposed through a small store/residency boundary, for example:

Hydration work must not block the Raylib frame. Database reads, file decoding, network fetches, and browser creation run asynchronously. The main thread applies bounded completed work between input and rendering.

Entity-specific hydration

Entity family Catalog/summary Full hydration Active resource
Shapes, lines, buttons, switches All visual state inline Usually unnecessary None
Text and text areas Title and excerpt Full text payload Editing buffers already in scene
Conversation Title, latest-turn summary, unread/working state Requested transcript window Copilot session only while needed
Image Dimensions, media type, thumbnail reference Original asset metadata Decoded texture or CEF image view
Browser URL, title, favicon/preview Navigation/session metadata Pooled CEF browser
Calendar/table/card Compact semantic summary Type-specific rows/items None unless externally backed
Generated component Schema name and safe summary Versioned component data Allowlisted renderer only

Rendering hydration and context hydration are related but not identical. An off-screen conversation may need transcript hydration because the user asked about it explicitly. A visible image may need a texture for rendering but only caption metadata for AI context.

Movement and writes

Entity movement, resize, and z-order updates use one transaction:

BEGIN IMMEDIATE
  UPDATE entity revision/z_order/updated_at
  UPDATE entity_bounds min/max coordinates
  INSERT append-only mutation record when history is enabled
COMMIT

Do not update the R-tree only after a drag finishes if other viewport or context queries can run during dragging. The UI may coalesce writes to one transaction every short interval and always flush at drag completion.

Persistent semantic membership is separate:

Camera checkpoints

Persist one reserved last-view checkpoint per workspace and allow named viewpoints later.

The checkpoint is user/workspace state, not part of an entity payload.

Context planning

The camera query supplies candidates; it must not blindly concatenate every full payload.

Context assembly should proceed in budgeted layers:

  1. Workspace and active region summaries.
  2. Stable ID, type, bounds, title, and summary for visible entities.
  3. Type-specific semantic state for the most relevant visible entities.
  4. Full payload excerpts requested by the orchestrator or selected by policy.
  5. Explicitly referenced off-screen entities and relationships.

Every context entry includes its stable ID and hydration level. If more detail exists but was not loaded, expose a retrieval capability instead of pretending the summary is complete. Parked or archived conversations remain excluded by policy even if their geometry intersects a large viewport.

The context planner owns token and byte budgets. The renderer does not decide which transcript window or document section an agent receives.

Cache and eviction policy

Begin with deterministic limits rather than a complex adaptive cache:

Eviction must never discard dirty state. Dirty entities flush successfully or surface an error before their full payload is released.

Database export and bucket upload

Do not upload the live database file while SQLite is using WAL mode. A valid export flow is:

  1. Flush dirty entity and camera writes.
  2. Use SQLite's backup API to write a temporary standalone database.
  3. Run PRAGMA quick_check on the copy.
  4. Record schema version, workspace revision, byte length, and checksum in a small manifest.
  5. Optionally encrypt the database copy before it leaves the machine.
  6. Upload to an immutable object key containing workspace ID, revision, and checksum.
  7. Update a small latest pointer using a conditional write.

Credentials, Copilot tokens, CEF caches, decoded textures, and other machine-local runtime state must never enter the workspace database.

The first synchronization model is single-writer backup/restore. If a database has changed locally and remotely, preserve both copies and require an explicit choice. Multi-writer merge requires stable operation IDs, an append-only change log, and conflict semantics; it is a later architecture milestone.

Relationship to .zmap

The current .zmap format remains useful for:

It should not become the long-term query engine. Migration should decode a validated .zmap, assign stable entity IDs, and insert catalog, bounds, and payload rows in one transaction. Database exports supersede .zmap for full workspace backup after parity is proven.

Module ownership

Proposed first-party targets:

Target Responsibility
//yuu:canvas_store Backend-neutral workspace/catalog API
//yuu:canvas_store_sqlite Deita/SQLite schema, queries, migrations, export
//yuu:canvas_residency Camera query generations, hot/warm sets, eviction
//yuu:canvas_hydration Hydrator registry and completion queue
//yuu:canvas_context Budgeted AI context planning
//yuu:scene_store Existing .zmap snapshot compatibility

Keep SQL and database handles out of canvas.c, main.c, and entity drawing functions. Those files consume store results and report mutations through the interface.

Delivery phases

Phase 0: contracts and measurements

Phase 1: SQLite catalog and camera

Phase 2: viewport residency

Phase 3: entity-specific hydration

Phase 4: context planner

Phase 5: export and restore

Phase 6: hosted backend

Acceptance criteria