Mercurial

comparison qwen3_vl/README.md @ 275:78699f810817

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add Qwen3-VL and WebRTC dictation services Add Bazel targets for the CUDA-backed Qwen3-VL server and a local WebRTC faster-whisper dictation service. Co-authored-by: Copilot <[email protected]> Copilot-Session: e3d8cb06-6c95-4ae0-9757-651d3796ab00
author MrJuneJune <me@mrjunejune.com>
date Mon, 17 Aug 2026 10:58:47 -0700
parents
children 49e9e591c9bb
comparison
equal deleted inserted replaced
274:c9be578316a6 275:78699f810817
1 # Qwen3-VL 30B A3B
2
3 This package runs `Qwen3-VL-30B-A3B-Instruct` through a Bazel-pinned
4 llama.cpp CUDA runtime. It is tailored to the current WSL machine: an RTX 4070
5 Ti with 12 GB VRAM and 32 GB host RAM.
6
7 The default model is Unsloth's 11.26 GB `Q2_K` GGUF plus the 1.08 GB FP16
8 vision projector. This lower quantization is intentional: the official
9 18.56 GB `Q4_K_M` model cannot fit on this GPU and is too large for the
10 current WSL memory allocation. Model weights are downloaded to
11 `~/.cache/zenbu/qwen3-vl-30b-a3b`, not the repository or Bazel cache.
12
13 ## Setup
14
15 Verify that Bazel's pinned runtime can see the GPU:
16
17 ```bash
18 bazel run //qwen3_vl:preflight
19 ```
20
21 Download and verify the model files:
22
23 ```bash
24 bazel run //qwen3_vl:download_model
25 ```
26
27 Start an interactive image prompt:
28
29 ```bash
30 bazel run //qwen3_vl:chat -- \
31 --image /path/to/image.jpg \
32 --prompt "Describe this image." \
33 --n-predict 256
34 ```
35
36 Start the OpenAI-compatible server:
37
38 ```bash
39 bazel run //qwen3_vl:serve
40 curl.exe http://127.0.0.1:8080/health
41 ```
42
43 The pinned CUDA executable runs on the Windows side of WSL interop. Windows
44 browsers and clients can use `http://127.0.0.1:8080`; use `curl.exe`, rather
45 than Linux `curl`, when checking it from a WSL shell.
46
47 Runtime tuning is available through environment variables:
48
49 ```bash
50 QWEN3_VL_GPU_LAYERS=16 \
51 QWEN3_VL_CONTEXT_SIZE=1024 \
52 bazel run //qwen3_vl:chat -- --prompt "Hello" --n-predict 64
53 ```
54
55 Increase `QWEN3_VL_GPU_LAYERS` until GPU memory is nearly full. Reduce it if
56 llama.cpp reports an allocation failure. Keep the context small on this
57 machine because the KV cache also consumes GPU and system memory.
58
59 The WSL instance currently exposes about 15 GB of RAM even though the host has
60 32 GB. If model loading fails under memory pressure, create `%UserProfile%\.wslconfig`
61 on Windows with:
62
63 ```ini
64 [wsl2]
65 memory=28GB
66 swap=16GB
67 ```
68
69 Then run `wsl --shutdown` from Windows before retrying.