diff wiki/README.md @ 279:b3b547563ec7

Add Google connector service and agent wiki Implement the C/Seobeo Google Drive and Gmail connector with encrypted OAuth storage, Zenbu authentication, browser testing, AI tool discovery, chunked HTTP decoding, and Bazel coverage. Consolidate repository guidance into progressive wiki documentation and enforce arena-first allocation for new first-party C code. Co-authored-by: Copilot <[email protected]> Copilot-Session: 84c338fd-0939-4bb3-b7f3-1062eb213e5d
author MrJuneJune <me@mrjunejune.com>
date Mon, 17 Aug 2026 22:22:36 -0700
parents
children
line wrap: on
line diff
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/wiki/README.md	Mon Aug 17 22:22:36 2026 -0700
@@ -0,0 +1,302 @@
+---
+title: Zenbu repository wiki
+status: canonical
+audience:
+  - humans
+  - AI agents
+last_reviewed: 2026-08-17
+---
+
+# Zenbu repository wiki
+
+This page is the monorepo knowledge router. It explains what Zenbu contains and
+directs humans and agents to the smallest canonical documentation set needed
+for a task.
+
+## Reading protocol
+
+```text
+repository index -> relevant project wiki -> BUILD graph -> source and tests
+```
+
+Do not scan every project document by default. Choose documentation from the
+routing table below. For cross-project work, inspect Bazel dependencies first,
+then read only the wikis owned by actual producers and consumers.
+
+## Documentation routing
+
+| Task paths or topic | Canonical documentation |
+| --- | --- |
+| `connectors/`, Google OAuth, Drive, Gmail, AI context tools | [`connectors/wiki/README.md`](../connectors/wiki/README.md) |
+| `design_system/`, shared components, tokens, HTML authoring | [`design_system/wiki/README.md`](../design_system/wiki/README.md) |
+| `seobeo/`, HTTP, TLS, servers, clients, SSE, WebSockets | [`seobeo/README.md`](../seobeo/README.md), then only the relevant page in [`seobeo/docs/`](../seobeo/docs/) |
+| `mrjunejune/`, personal site, auth UI, conversations, media | [`mrjunejune/README.md`](../mrjunejune/README.md) |
+| `hg-web/`, Mercurial repository browser | [`hg-web/README.md`](../hg-web/README.md) |
+| `dowa/`, arenas, strings, arrays, hash maps, math | [`dowa/README.md`](../dowa/README.md) |
+| `deita/`, SQLite wrapper | [`deita/README.md`](../deita/README.md) |
+| `s3/`, presigned uploads | Source header and `BUILD`; no dedicated wiki yet |
+| `markdown_converter/` | Package `BUILD`, public headers, and tests |
+| `gui_ze/`, asset bundling macros | [`gui_ze/wiki/README.md`](../gui_ze/wiki/README.md) |
+| Experiments and standalone prototypes | The package README when present, then its `BUILD` |
+
+## Monorepo map
+
+### Shared foundations
+
+- `dowa`: core C types, arena allocation, strings, dynamic arrays/hash maps,
+  and math.
+- `deita`: SQLite connection, prepared query, and result-set wrapper.
+- `seobeo`: networking, HTTP client/server, TLS, SSE, and WebSockets.
+- `auth`: Zenbu users, sessions, cryptography, and reusable HTTP auth.
+- `config`: Bazel platform configuration.
+
+### Services and applications
+
+- `connectors`: Google data connector service and AI tool boundary.
+- `mrjunejune`: personal website and production server.
+- `hg-web`: Mercurial repository browser.
+- `npc`: MCP-oriented C server experiment.
+- `dictation`, `schwab_trader`, `infinite_canvas`, and other package roots:
+  focused applications or prototypes.
+
+### Web and asset systems
+
+- `design_system`: shared light-DOM Web Components and design tokens.
+- `gui_ze`: Bazel macros for asset copying, bundling, and transformations.
+- `assets`: shared icons and other reusable assets.
+- `markdown_converter`: C/WASM markdown conversion.
+- `rich_editor` and `react_games`: browser bundles consumed by applications.
+
+## Build and workflow
+
+Zenbu is a Mercurial repository. Use:
+
+```sh
+hg status
+hg diff
+```
+
+### Bazel-only rule
+
+Zenbu is Bazel-only. Bazel is not just the C build tool; it is the repository's
+execution and dependency boundary.
+
+Use Bazel for:
+
+- C/C++ libraries, binaries, and tests;
+- shell and Python entry points;
+- Node/Bun-driven frontend generation;
+- WASM compilation;
+- asset copying, image conversion, and bundles;
+- downloaded tools, browser runtimes, and model runtimes;
+- integration and end-to-end tests.
+
+Do not add or rely on:
+
+- Makefiles or direct `make` workflows;
+- CMake project files;
+- manual `cc`, `clang`, or `gcc` build commands;
+- `npm run`, `bun run`, or `pip install` as the normal repository interface;
+- undocumented system packages or `/usr/local` libraries;
+- scripts that bypass Bazel runfiles and declared `data`.
+
+An upstream ecosystem command may run inside a Bazel rule or a narrowly scoped
+maintenance workflow, but the committed user/agent interface must be a Bazel
+target.
+
+Build and test from the repository root:
+
+```sh
+bazel build //seobeo:seobeo
+bazel test //seobeo/tests:seobeo_http_framing_test
+
+bazel build //mrjunejune:mrjunejune_server
+bazel test //mrjunejune/test:integration_test
+
+bazel build //connectors:connector_server
+bazel test //connectors:connector_tests
+
+bazel build //hg-web:hg_web_server
+```
+
+Inspect a package's `BUILD` file before broad searches. Keep dependencies on the
+smallest target that needs them.
+
+## Package selection guide
+
+Choose an existing first-party package before writing a new helper or directly
+depending on third-party code.
+
+| Need | Use | Typical Bazel label | Do not substitute by default |
+| --- | --- | --- | --- |
+| Integer aliases, booleans, arenas, strings, arrays, hash maps, JSON helpers, math | `dowa` | `//dowa:dowa` | libc allocation scattered through request code, a second containers library |
+| SQLite connections and prepared queries | `deita` | `//deita:deita` | direct `sqlite3_*` calls in application packages |
+| TCP/TLS primitives without HTTP | `seobeo_min` | `//seobeo:seobeo_min` | raw sockets/OpenSSL setup duplicated in applications |
+| HTTP server without WebSockets | `seobeo_tcp_server` | `//seobeo:seobeo_tcp_server` | another embedded HTTP server |
+| HTTP server with WebSockets | `seobeo_tcp_server_ws` | `//seobeo:seobeo_tcp_server_ws` | direct WebSocket framing |
+| Outbound HTTP/TLS only | `seobeo_tcp_client` or compatibility alias `seobeo_client` | `//seobeo:seobeo_tcp_client` | libcurl subprocesses or hand-built HTTP clients |
+| Full HTTP/TLS/WebSocket stack | `seobeo` | `//seobeo:seobeo` | depending on every network source manually |
+| Verbose Seobeo diagnostics | `seobeo_debug` | `//seobeo:seobeo_debug` | permanent `printf` debugging |
+| User, password, session, cookie, CSRF, and auth crypto | `auth` | `//auth:auth`, `//auth:auth_http` | creating another login/session store |
+| Google OAuth, Drive, Gmail, encrypted provider credentials, AI connector tools | `connectors` | `//connectors:connector_core`, `//connectors:connector_service_lib` | provider token logic inside unrelated applications |
+| S3 signing and presigned uploads | `s3` | `//s3:s3` | ad-hoc AWS signature code |
+| Native and WASM markdown conversion | `markdown_converter` | `//markdown_converter:markdown_to_html_c`, `//markdown_converter:markdown_to_html_wasm` | a second markdown pipeline |
+| Shared Web Components, tokens, and UI primitives | `design_system` | package targets in `//design_system` | application-local copies of shared controls |
+| Shared icons and generated image assets | `assets` | `//assets:icons` | copying asset files between applications |
+| Web asset moves, bundles, WebP conversion, Bun actions, macOS bundles | `gui_ze` macros | `//gui_ze:gui_ze.bzl` | custom copy scripts and undeclared output folders |
+| Shared rich text editor browser bundle | `rich_editor` | package targets in `//rich_editor` | forking editor code into an app |
+| Personal site server and its production bundle | `mrjunejune` | `//mrjunejune:mrjunejune_server` | treating site-specific code as a generic library |
+| Mercurial browser service | `hg-web` | `//hg-web:hg_web_server` | shelling out from unrelated UI code without its API boundary |
+| MCP-style C HTTP server experiment | `npc` | `//npc:npc` | assuming it is a general connector framework |
+| Raylib application rules | `third_party/raylib` macro plus first-party app package | `//third_party/raylib:raylib.bzl` | manual platform linker flags |
+| Qwen/llama.cpp runtime orchestration | `qwen3_vl` | `//qwen3_vl:model`, `//qwen3_vl:serve` | untracked local model commands |
+
+### Package boundary rules
+
+- Put reusable behavior in the narrowest shared package that owns the concept.
+- Applications may compose libraries; libraries must not depend on applications.
+- Avoid circular ownership such as `dowa` knowing about Seobeo or a site.
+- Expose public headers through the owning library target.
+- Add deps to the smallest target that compiles the source using them.
+- Use platform-aware aliases already provided by packages such as Seobeo and S3.
+- Search for an existing helper or macro before creating a sibling implementation.
+
+## Third-party dependency policy
+
+Third-party code enters Zenbu through one of two Bazel-controlled paths:
+
+1. A pinned Bzlmod dependency or repository rule in `MODULE.bazel`.
+2. A vendored wrapper under `third_party/<name>/BUILD`.
+
+Applications should depend on public Bazel labels, not include vendored source
+paths or reproduce platform flags.
+
+### Prefer first-party wrappers
+
+- Use Deita instead of direct SQLite APIs unless changing Deita itself.
+- Use Seobeo instead of raw sockets, OpenSSL transport code, or curl processes.
+- Use the S3 package instead of implementing AWS signing in an application.
+- Use `gui_ze` macros instead of manual generated-asset copying.
+- Use `markdown_converter` rather than adding another markdown dependency.
+- Use `auth` for Zenbu identity; external OAuth credentials belong in
+  `connectors`, not a parallel user system.
+
+### Adding an external dependency
+
+Before adding one:
+
+1. Search first-party packages and existing `third_party/` wrappers.
+2. Confirm the behavior cannot be implemented safely with an existing library.
+3. Add the dependency through `MODULE.bazel` when Bzlmod support is suitable;
+   otherwise add a focused `third_party/<name>/BUILD` wrapper.
+4. Pin versions and SHA-256 hashes for downloaded archives/files.
+5. Preserve license files and upstream notices.
+6. Expose only the smallest useful Bazel target and visibility.
+7. Keep platform selection inside the wrapper, not every consumer.
+8. Add or update lockfiles when the ecosystem requires them.
+9. Document why the dependency exists and which first-party package owns its
+   abstraction boundary.
+
+Do not fetch executable code at ordinary runtime when it can be a declared
+Bazel dependency or data artifact.
+
+### Existing external systems
+
+| External system | Bazel integration and intended use |
+| --- | --- |
+| OpenSSL | Bzlmod `@openssl`; TLS and crypto, normally behind Seobeo/auth/connectors |
+| SQLite | `//third_party/sqlite3`; normally consumed through Deita |
+| Emscripten | `@emsdk` local override; WASM targets such as markdown conversion |
+| Node/npm | rules_nodejs and aspect_rules_js; locked frontend/test dependencies |
+| Python/pip | rules_python toolchain and package-specific locked hubs |
+| Bun | pinned platform archives and `gui_ze` actions |
+| Chromium/Playwright | pinned browser runtime for browser and integration tests |
+| Tectonic | pinned runtime for LaTeX rendering |
+| FFmpeg | `//third_party/ffmpeg` wrapper for media processing |
+| Raylib | `//third_party/raylib` plus `raylib_binary` for native/web applications |
+| libuv | `//third_party/libuv` for event-loop experiments such as Postdog |
+| LuaJIT | `//third_party/luajit` for packages that explicitly declare it |
+| Mercurial | wheel/runtime wrappers for hg-web tooling |
+| CEF | repository rule under `third_party/cef` for the infinite-canvas native surface |
+| llama.cpp/CUDA runtime | pinned archives exposed through `qwen3_vl` Bazel targets |
+
+## First-party conventions
+
+- Prefer Dowa integer and boolean aliases plus `TRUE`/`FALSE`.
+- Allocation is arena-first. New first-party C code must not call raw
+  `malloc`, `calloc`, `realloc`, or `free`.
+- Create a `Dowa_Arena` at the owner boundary, pass it into helpers, and use
+  `Dowa_Arena_Allocate`, arena string helpers, and the `_Arena` array/hash-map
+  macros.
+- Free each locally owned arena exactly once with `Dowa_Arena_Free` on every
+  return path. Never individually free an arena-owned pointer.
+- Borrowed request arenas are never freed by callees.
+- Long-lived objects should own an arena whose lifetime matches the object.
+  A non-arena allocation is permitted only at a true allocator/runtime or
+  external-API ownership boundary, must be narrowly documented, and must use
+  the matching first-party/external destructor. It is not a convenience escape
+  hatch.
+- Use Seobeo logging and response-map conventions in Seobeo services.
+- Keep runtime secrets in one ignored, documented service config file.
+- Commit placeholder templates only; do not commit real credentials.
+- Preserve platform-aware Bazel aliases and precise dependencies.
+
+### Arena enforcement
+
+Run:
+
+```sh
+bazel test //:arena_policy_test
+```
+
+The policy target fails when:
+
+- a new first-party C/H file introduces raw allocation calls; or
+- a legacy file increases its reviewed raw-allocation baseline.
+
+The baseline is technical debt, not permission to add more allocation in those
+files. Changes should reduce it. Updating the baseline upward requires an
+explicitly documented allocator/runtime boundary and review.
+
+Typical ownership:
+
+```c
+Dowa_Arena *p_arena = Dowa_Arena_Create(64 * 1024);
+if (!p_arena)
+  return FALSE;
+
+char *copy = Dowa_Arena_Allocate(p_arena, length + 1);
+if (!copy)
+{
+  Dowa_Arena_Free(p_arena);
+  return FALSE;
+}
+
+/* All arena-owned values die together. */
+Dowa_Arena_Free(p_arena);
+```
+
+## Wiki model
+
+The repository index is the schema and router. Project wikis are synthesized,
+canonical knowledge for a bounded domain. Source code, `BUILD` files, tests,
+and committed config templates are evidence.
+
+When a task changes knowledge:
+
+1. Update the owning project wiki.
+2. Update this routing page only if ownership, package boundaries, or canonical
+   documentation locations changed.
+3. Do not create one-off top-level notes or duplicate normative instructions.
+4. Split a project wiki only when its index becomes hard to navigate; every new
+   page must be linked from the project wiki.
+
+## Current wiki coverage
+
+| Domain | State |
+| --- | --- |
+| Repository map and agent routing | Canonical |
+| Connectors and AI retrieval | Canonical |
+| Design system | Canonical multi-page wiki |
+| Seobeo | README plus focused protocol docs |
+| Other packages | Existing README or source-first; promote to a wiki when the domain accumulates durable operational knowledge |