Mercurial

annotate asyncio_threads/inference/README.md @ 279:b3b547563ec7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add Google connector service and agent wiki Implement the C/Seobeo Google Drive and Gmail connector with encrypted OAuth storage, Zenbu authentication, browser testing, AI tool discovery, chunked HTTP decoding, and Bazel coverage. Consolidate repository guidance into progressive wiki documentation and enforce arena-first allocation for new first-party C code. Co-authored-by: Copilot <[email protected]> Copilot-Session: 84c338fd-0939-4bb3-b7f3-1062eb213e5d
author MrJuneJune <me@mrjunejune.com>
date Mon, 17 Aug 2026 22:22:36 -0700
parents 46daba6e3cf4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
48
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
1 Inference Questions
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
2
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
3 Context
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
4
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
5 You are tasked with building a simplified inference engine component responsible for handling incoming user requests for a large language model (LLM). To optimize throughput and GPU utilization, the engine must batch multiple requests together, run the inference call once per batch, and then deconstruct the results to return token-level output to the individual users.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
6
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
7 Objective
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
8
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
9 Complete the provided Python class, BatchInferenceEngine by implementing the methods necessary to:
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
10 Queue incoming user requests.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
11 Process a batch when the queue reaches a defined batch size.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
12
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
13 Simulate the token-level output from an LLM and correctly associate each generated token with its original request.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
14
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
15
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
16 Task Requirements
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
17
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
18
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
19
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
20
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
21
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
22 Implement the logic for $enqueue\_request$.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
23
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
24
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
25
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
26 Implement the logic for $\_process\_batch$.
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
27
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
28
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
29
46daba6e3cf4 Few python scrtips to show how to use asychio.
MrJuneJune <me@mrjunejune.com>
parents:
diff changeset
30 Demonstrate the usage by creating 7 unique requests and enqueueing them one by one. Show the state of the queue and the processed tokens after each batch run.