# Qwen3-VL 30B A3B

This package runs `Qwen3-VL-30B-A3B-Instruct` through a Bazel-pinned
llama.cpp CUDA runtime. It is tailored to the current WSL machine: an RTX 4070
Ti with 12 GB VRAM and 32 GB host RAM.

The default model is Unsloth's 11.26 GB `Q2_K` GGUF plus the 1.08 GB FP16
vision projector. This lower quantization is intentional: the official
18.56 GB `Q4_K_M` model cannot fit on this GPU and is too large for the
current WSL memory allocation. Model weights are downloaded to
`~/.cache/zenbu/qwen3-vl-30b-a3b`, not the repository or Bazel cache.

## Setup

Verify that Bazel's pinned runtime can see the GPU:

```bash
bazel run //qwen3_vl:preflight
```

Download and verify the model files:

```bash
bazel run //qwen3_vl:download_model
```

Start an interactive image prompt:

```bash
bazel run //qwen3_vl:chat -- \
  --image /path/to/image.jpg \
  --prompt "Describe this image." \
  --n-predict 256
```

Start the OpenAI-compatible server:

```bash
bazel run //qwen3_vl:serve
curl.exe http://127.0.0.1:8080/health
```

The pinned CUDA executable runs on the Windows side of WSL interop. Windows
browsers and clients can use `http://127.0.0.1:8080`; use `curl.exe`, rather
than Linux `curl`, when checking it from a WSL shell.

Runtime tuning is available through environment variables:

```bash
QWEN3_VL_GPU_LAYERS=16 \
QWEN3_VL_CONTEXT_SIZE=1024 \
  bazel run //qwen3_vl:chat -- --prompt "Hello" --n-predict 64
```

Increase `QWEN3_VL_GPU_LAYERS` until GPU memory is nearly full. Reduce it if
llama.cpp reports an allocation failure. Keep the context small on this
machine because the KV cache also consumes GPU and system memory.

The WSL instance currently exposes about 15 GB of RAM even though the host has
32 GB. If model loading fails under memory pressure, create `%UserProfile%\.wslconfig`
on Windows with:

```ini
[wsl2]
memory=28GB
swap=16GB
```

Then run `wsl --shutdown` from Windows before retrying.