Mercurial

comparison wiki/README.md @ 279:b3b547563ec7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add Google connector service and agent wiki Implement the C/Seobeo Google Drive and Gmail connector with encrypted OAuth storage, Zenbu authentication, browser testing, AI tool discovery, chunked HTTP decoding, and Bazel coverage. Consolidate repository guidance into progressive wiki documentation and enforce arena-first allocation for new first-party C code. Co-authored-by: Copilot <[email protected]> Copilot-Session: 84c338fd-0939-4bb3-b7f3-1062eb213e5d
author MrJuneJune <me@mrjunejune.com>
date Mon, 17 Aug 2026 22:22:36 -0700
parents
children
comparison
equal deleted inserted replaced
278:8d560f50ed4c 279:b3b547563ec7
1 ---
2 title: Zenbu repository wiki
3 status: canonical
4 audience:
5 - humans
6 - AI agents
7 last_reviewed: 2026-08-17
8 ---
9
10 # Zenbu repository wiki
11
12 This page is the monorepo knowledge router. It explains what Zenbu contains and
13 directs humans and agents to the smallest canonical documentation set needed
14 for a task.
15
16 ## Reading protocol
17
18 ```text
19 repository index -> relevant project wiki -> BUILD graph -> source and tests
20 ```
21
22 Do not scan every project document by default. Choose documentation from the
23 routing table below. For cross-project work, inspect Bazel dependencies first,
24 then read only the wikis owned by actual producers and consumers.
25
26 ## Documentation routing
27
28 | Task paths or topic | Canonical documentation |
29 | --- | --- |
30 | `connectors/`, Google OAuth, Drive, Gmail, AI context tools | [`connectors/wiki/README.md`](../connectors/wiki/README.md) |
31 | `design_system/`, shared components, tokens, HTML authoring | [`design_system/wiki/README.md`](../design_system/wiki/README.md) |
32 | `seobeo/`, HTTP, TLS, servers, clients, SSE, WebSockets | [`seobeo/README.md`](../seobeo/README.md), then only the relevant page in [`seobeo/docs/`](../seobeo/docs/) |
33 | `mrjunejune/`, personal site, auth UI, conversations, media | [`mrjunejune/README.md`](../mrjunejune/README.md) |
34 | `hg-web/`, Mercurial repository browser | [`hg-web/README.md`](../hg-web/README.md) |
35 | `dowa/`, arenas, strings, arrays, hash maps, math | [`dowa/README.md`](../dowa/README.md) |
36 | `deita/`, SQLite wrapper | [`deita/README.md`](../deita/README.md) |
37 | `s3/`, presigned uploads | Source header and `BUILD`; no dedicated wiki yet |
38 | `markdown_converter/` | Package `BUILD`, public headers, and tests |
39 | `gui_ze/`, asset bundling macros | [`gui_ze/wiki/README.md`](../gui_ze/wiki/README.md) |
40 | Experiments and standalone prototypes | The package README when present, then its `BUILD` |
41
42 ## Monorepo map
43
44 ### Shared foundations
45
46 - `dowa`: core C types, arena allocation, strings, dynamic arrays/hash maps,
47 and math.
48 - `deita`: SQLite connection, prepared query, and result-set wrapper.
49 - `seobeo`: networking, HTTP client/server, TLS, SSE, and WebSockets.
50 - `auth`: Zenbu users, sessions, cryptography, and reusable HTTP auth.
51 - `config`: Bazel platform configuration.
52
53 ### Services and applications
54
55 - `connectors`: Google data connector service and AI tool boundary.
56 - `mrjunejune`: personal website and production server.
57 - `hg-web`: Mercurial repository browser.
58 - `npc`: MCP-oriented C server experiment.
59 - `dictation`, `schwab_trader`, `infinite_canvas`, and other package roots:
60 focused applications or prototypes.
61
62 ### Web and asset systems
63
64 - `design_system`: shared light-DOM Web Components and design tokens.
65 - `gui_ze`: Bazel macros for asset copying, bundling, and transformations.
66 - `assets`: shared icons and other reusable assets.
67 - `markdown_converter`: C/WASM markdown conversion.
68 - `rich_editor` and `react_games`: browser bundles consumed by applications.
69
70 ## Build and workflow
71
72 Zenbu is a Mercurial repository. Use:
73
74 ```sh
75 hg status
76 hg diff
77 ```
78
79 ### Bazel-only rule
80
81 Zenbu is Bazel-only. Bazel is not just the C build tool; it is the repository's
82 execution and dependency boundary.
83
84 Use Bazel for:
85
86 - C/C++ libraries, binaries, and tests;
87 - shell and Python entry points;
88 - Node/Bun-driven frontend generation;
89 - WASM compilation;
90 - asset copying, image conversion, and bundles;
91 - downloaded tools, browser runtimes, and model runtimes;
92 - integration and end-to-end tests.
93
94 Do not add or rely on:
95
96 - Makefiles or direct `make` workflows;
97 - CMake project files;
98 - manual `cc`, `clang`, or `gcc` build commands;
99 - `npm run`, `bun run`, or `pip install` as the normal repository interface;
100 - undocumented system packages or `/usr/local` libraries;
101 - scripts that bypass Bazel runfiles and declared `data`.
102
103 An upstream ecosystem command may run inside a Bazel rule or a narrowly scoped
104 maintenance workflow, but the committed user/agent interface must be a Bazel
105 target.
106
107 Build and test from the repository root:
108
109 ```sh
110 bazel build //seobeo:seobeo
111 bazel test //seobeo/tests:seobeo_http_framing_test
112
113 bazel build //mrjunejune:mrjunejune_server
114 bazel test //mrjunejune/test:integration_test
115
116 bazel build //connectors:connector_server
117 bazel test //connectors:connector_tests
118
119 bazel build //hg-web:hg_web_server
120 ```
121
122 Inspect a package's `BUILD` file before broad searches. Keep dependencies on the
123 smallest target that needs them.
124
125 ## Package selection guide
126
127 Choose an existing first-party package before writing a new helper or directly
128 depending on third-party code.
129
130 | Need | Use | Typical Bazel label | Do not substitute by default |
131 | --- | --- | --- | --- |
132 | Integer aliases, booleans, arenas, strings, arrays, hash maps, JSON helpers, math | `dowa` | `//dowa:dowa` | libc allocation scattered through request code, a second containers library |
133 | SQLite connections and prepared queries | `deita` | `//deita:deita` | direct `sqlite3_*` calls in application packages |
134 | TCP/TLS primitives without HTTP | `seobeo_min` | `//seobeo:seobeo_min` | raw sockets/OpenSSL setup duplicated in applications |
135 | HTTP server without WebSockets | `seobeo_tcp_server` | `//seobeo:seobeo_tcp_server` | another embedded HTTP server |
136 | HTTP server with WebSockets | `seobeo_tcp_server_ws` | `//seobeo:seobeo_tcp_server_ws` | direct WebSocket framing |
137 | Outbound HTTP/TLS only | `seobeo_tcp_client` or compatibility alias `seobeo_client` | `//seobeo:seobeo_tcp_client` | libcurl subprocesses or hand-built HTTP clients |
138 | Full HTTP/TLS/WebSocket stack | `seobeo` | `//seobeo:seobeo` | depending on every network source manually |
139 | Verbose Seobeo diagnostics | `seobeo_debug` | `//seobeo:seobeo_debug` | permanent `printf` debugging |
140 | User, password, session, cookie, CSRF, and auth crypto | `auth` | `//auth:auth`, `//auth:auth_http` | creating another login/session store |
141 | Google OAuth, Drive, Gmail, encrypted provider credentials, AI connector tools | `connectors` | `//connectors:connector_core`, `//connectors:connector_service_lib` | provider token logic inside unrelated applications |
142 | S3 signing and presigned uploads | `s3` | `//s3:s3` | ad-hoc AWS signature code |
143 | Native and WASM markdown conversion | `markdown_converter` | `//markdown_converter:markdown_to_html_c`, `//markdown_converter:markdown_to_html_wasm` | a second markdown pipeline |
144 | Shared Web Components, tokens, and UI primitives | `design_system` | package targets in `//design_system` | application-local copies of shared controls |
145 | Shared icons and generated image assets | `assets` | `//assets:icons` | copying asset files between applications |
146 | Web asset moves, bundles, WebP conversion, Bun actions, macOS bundles | `gui_ze` macros | `//gui_ze:gui_ze.bzl` | custom copy scripts and undeclared output folders |
147 | Shared rich text editor browser bundle | `rich_editor` | package targets in `//rich_editor` | forking editor code into an app |
148 | Personal site server and its production bundle | `mrjunejune` | `//mrjunejune:mrjunejune_server` | treating site-specific code as a generic library |
149 | Mercurial browser service | `hg-web` | `//hg-web:hg_web_server` | shelling out from unrelated UI code without its API boundary |
150 | MCP-style C HTTP server experiment | `npc` | `//npc:npc` | assuming it is a general connector framework |
151 | Raylib application rules | `third_party/raylib` macro plus first-party app package | `//third_party/raylib:raylib.bzl` | manual platform linker flags |
152 | Qwen/llama.cpp runtime orchestration | `qwen3_vl` | `//qwen3_vl:model`, `//qwen3_vl:serve` | untracked local model commands |
153
154 ### Package boundary rules
155
156 - Put reusable behavior in the narrowest shared package that owns the concept.
157 - Applications may compose libraries; libraries must not depend on applications.
158 - Avoid circular ownership such as `dowa` knowing about Seobeo or a site.
159 - Expose public headers through the owning library target.
160 - Add deps to the smallest target that compiles the source using them.
161 - Use platform-aware aliases already provided by packages such as Seobeo and S3.
162 - Search for an existing helper or macro before creating a sibling implementation.
163
164 ## Third-party dependency policy
165
166 Third-party code enters Zenbu through one of two Bazel-controlled paths:
167
168 1. A pinned Bzlmod dependency or repository rule in `MODULE.bazel`.
169 2. A vendored wrapper under `third_party/<name>/BUILD`.
170
171 Applications should depend on public Bazel labels, not include vendored source
172 paths or reproduce platform flags.
173
174 ### Prefer first-party wrappers
175
176 - Use Deita instead of direct SQLite APIs unless changing Deita itself.
177 - Use Seobeo instead of raw sockets, OpenSSL transport code, or curl processes.
178 - Use the S3 package instead of implementing AWS signing in an application.
179 - Use `gui_ze` macros instead of manual generated-asset copying.
180 - Use `markdown_converter` rather than adding another markdown dependency.
181 - Use `auth` for Zenbu identity; external OAuth credentials belong in
182 `connectors`, not a parallel user system.
183
184 ### Adding an external dependency
185
186 Before adding one:
187
188 1. Search first-party packages and existing `third_party/` wrappers.
189 2. Confirm the behavior cannot be implemented safely with an existing library.
190 3. Add the dependency through `MODULE.bazel` when Bzlmod support is suitable;
191 otherwise add a focused `third_party/<name>/BUILD` wrapper.
192 4. Pin versions and SHA-256 hashes for downloaded archives/files.
193 5. Preserve license files and upstream notices.
194 6. Expose only the smallest useful Bazel target and visibility.
195 7. Keep platform selection inside the wrapper, not every consumer.
196 8. Add or update lockfiles when the ecosystem requires them.
197 9. Document why the dependency exists and which first-party package owns its
198 abstraction boundary.
199
200 Do not fetch executable code at ordinary runtime when it can be a declared
201 Bazel dependency or data artifact.
202
203 ### Existing external systems
204
205 | External system | Bazel integration and intended use |
206 | --- | --- |
207 | OpenSSL | Bzlmod `@openssl`; TLS and crypto, normally behind Seobeo/auth/connectors |
208 | SQLite | `//third_party/sqlite3`; normally consumed through Deita |
209 | Emscripten | `@emsdk` local override; WASM targets such as markdown conversion |
210 | Node/npm | rules_nodejs and aspect_rules_js; locked frontend/test dependencies |
211 | Python/pip | rules_python toolchain and package-specific locked hubs |
212 | Bun | pinned platform archives and `gui_ze` actions |
213 | Chromium/Playwright | pinned browser runtime for browser and integration tests |
214 | Tectonic | pinned runtime for LaTeX rendering |
215 | FFmpeg | `//third_party/ffmpeg` wrapper for media processing |
216 | Raylib | `//third_party/raylib` plus `raylib_binary` for native/web applications |
217 | libuv | `//third_party/libuv` for event-loop experiments such as Postdog |
218 | LuaJIT | `//third_party/luajit` for packages that explicitly declare it |
219 | Mercurial | wheel/runtime wrappers for hg-web tooling |
220 | CEF | repository rule under `third_party/cef` for the infinite-canvas native surface |
221 | llama.cpp/CUDA runtime | pinned archives exposed through `qwen3_vl` Bazel targets |
222
223 ## First-party conventions
224
225 - Prefer Dowa integer and boolean aliases plus `TRUE`/`FALSE`.
226 - Allocation is arena-first. New first-party C code must not call raw
227 `malloc`, `calloc`, `realloc`, or `free`.
228 - Create a `Dowa_Arena` at the owner boundary, pass it into helpers, and use
229 `Dowa_Arena_Allocate`, arena string helpers, and the `_Arena` array/hash-map
230 macros.
231 - Free each locally owned arena exactly once with `Dowa_Arena_Free` on every
232 return path. Never individually free an arena-owned pointer.
233 - Borrowed request arenas are never freed by callees.
234 - Long-lived objects should own an arena whose lifetime matches the object.
235 A non-arena allocation is permitted only at a true allocator/runtime or
236 external-API ownership boundary, must be narrowly documented, and must use
237 the matching first-party/external destructor. It is not a convenience escape
238 hatch.
239 - Use Seobeo logging and response-map conventions in Seobeo services.
240 - Keep runtime secrets in one ignored, documented service config file.
241 - Commit placeholder templates only; do not commit real credentials.
242 - Preserve platform-aware Bazel aliases and precise dependencies.
243
244 ### Arena enforcement
245
246 Run:
247
248 ```sh
249 bazel test //:arena_policy_test
250 ```
251
252 The policy target fails when:
253
254 - a new first-party C/H file introduces raw allocation calls; or
255 - a legacy file increases its reviewed raw-allocation baseline.
256
257 The baseline is technical debt, not permission to add more allocation in those
258 files. Changes should reduce it. Updating the baseline upward requires an
259 explicitly documented allocator/runtime boundary and review.
260
261 Typical ownership:
262
263 ```c
264 Dowa_Arena *p_arena = Dowa_Arena_Create(64 * 1024);
265 if (!p_arena)
266 return FALSE;
267
268 char *copy = Dowa_Arena_Allocate(p_arena, length + 1);
269 if (!copy)
270 {
271 Dowa_Arena_Free(p_arena);
272 return FALSE;
273 }
274
275 /* All arena-owned values die together. */
276 Dowa_Arena_Free(p_arena);
277 ```
278
279 ## Wiki model
280
281 The repository index is the schema and router. Project wikis are synthesized,
282 canonical knowledge for a bounded domain. Source code, `BUILD` files, tests,
283 and committed config templates are evidence.
284
285 When a task changes knowledge:
286
287 1. Update the owning project wiki.
288 2. Update this routing page only if ownership, package boundaries, or canonical
289 documentation locations changed.
290 3. Do not create one-off top-level notes or duplicate normative instructions.
291 4. Split a project wiki only when its index becomes hard to navigate; every new
292 page must be linked from the project wiki.
293
294 ## Current wiki coverage
295
296 | Domain | State |
297 | --- | --- |
298 | Repository map and agent routing | Canonical |
299 | Connectors and AI retrieval | Canonical |
300 | Design system | Canonical multi-page wiki |
301 | Seobeo | README plus focused protocol docs |
302 | Other packages | Existing README or source-first; promote to a wiki when the domain accumulates durable operational knowledge |