Doubling Qwopus 3.6 on a single RTX 4090

Local large language models are bottlenecked by their sequential token output: each new word has to wait for the previous one to finish, the way cars queue up on a single-lane road. On a single RTX 4090, that ceiling is roughly 44 tokens per second for a dense 27B model in 4-bit quantization. It is fine on paper; in practice it is the lag you feel between sentences whenever a local model is writing a long answer.

Multi-Token Prediction (MTP) — popularised by DeepSeek V3 and adopted by Qwen 3.5 / 3.6 / Qwopus — collapses the road into multiple lanes. Instead of asking the model "what is the next token?", MTP asks it to guess two or three tokens ahead in the same forward pass and verify those guesses in the same pass. When the guesses are correct (and they usually are), you get two or three tokens for the cost of one. The verifier guarantees the output is identical to greedy decoding — there is no quality trade-off.

Jackrong/Qwopus3.6-27B-v2-MTP-GGUF is a dense qwen35-architecture model fine-tuned on top of Qwen 3.6-27B with the MTP heads baked in. 27.32 billion parameters, distributed as a single GGUF file, available in twelve quantizations from Q2_K (10.9 GB) to BF16 (54.7 GB). For a 24 GB GPU the sweet spot is Q4_K_M at 16.8 GB — it leaves about 7 GB of headroom for context and activations.

License is Apache 2.0, so commercial use is fine. The author's recommended sampler is temperature=0.6 top_p=0.95 top_k=20; the context window is 49,152 tokens (I used 8,192 for the bench).

Before installation specifics, here is what the thing actually does. I picked three prompts that exercise very different token distributions: idiomatic Python, a one-off DevOps shell scripting question, and a step-by-step math proof. The whole point of MTP is that its benefit varies with token predictability — and that effect shows up clearly here.

Three prompts × three modes (baseline / MTP n=2 / MTP n=3) = 9 runs. The chart shows generation speed in tokens per second; the second panel is the speedup factor over baseline. Same model, same hardware, same answer.

Honest question: Ollama was already installed, has a clean Modelfile API, and would have saved me from compiling C++ from source. So I tried it first.

The import succeeds — the blob is hard-linked, no re-download, Ollama recognises the architecture as qwen35. The very first generate call, however, throws an error instead of a response:

{ "error": "failed to initialize model:
            qwen3next: layer 64 missing attn_qkv/attn_gate projections" }

Zero tokens generated. To prove the runtime itself is fine, a plain non-MTP qwen3.5:0.8b from the existing library runs cleanly at 332 t/s on the same API. It is specifically the MTP-augmented layer 64 that breaks Ollama.

Here is the root cause. Ollama's MTP support (PR #15980) was designed for Gemma 4, which keeps its draft model in a separate safetensors file pointed at via a new DRAFT Modelfile keyword. The Qwen 3.5 / 3.6 family — including Qwopus — takes the opposite approach: the draft heads are baked directly into layer 64 of the same GGUF. When Ollama's qwen3next loader walks that layer expecting standard attention projections, it finds MTP draft-head weights instead and bails out.

The fix has to come from Ollama upstream. A future minor release should add qwen3next MTP support, at which point a five-line Modelfile is the entire install. Until then, llama.cpp is the only engine that can actually use this GGUF.

Six steps. Open-source toolchain. No Docker, no Kubernetes, no hosted dashboards.

1. Clone the upstream ggml-org/llama.cpp main branch — MTP merged in PR #22673, May 2026.
2. Configure cmake with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89. Pinning sm_89 (the RTX 4090's native arch) turns a 25-minute build into ~7 minutes on a 28-thread box.
3. Compile: cmake --build build -j$(nproc). You end up with llama-cli, llama-server, and llama-bench in build/bin/.
4. Download the model. Point HF_HOME at a data disk (the Q4_K_M is 16.8 GB; do not put it on your system drive). hf download Jackrong/Qwopus3.6-27B-v2-MTP-GGUF Qwopus3.6-27B-v2-MTP-Q4_K_M.gguf finishes in about 3 minutes.
5. Run llama-server with the MTP flags (see below).
6. Done — OpenAI-compatible API on port 8080, the built-in web UI at the same URL.

MODEL=/d/hugging_face_cache/hub/models--Jackrong--Qwopus3.6-27B-v2-MTP-GGUF/\
snapshots/<hash>/Qwopus3.6-27B-v2-MTP-Q4_K_M.gguf

./build/bin/llama-server \
    -m "$MODEL"                                      \
    -ngl 99 -c 8192 -fa on                           \
    --spec-type draft-mtp --spec-draft-n-max 3       \
    -np 1 --jinja                                    \
    --cache-type-k q8_0 --cache-type-v q8_0          \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0   \
    --host 0.0.0.0 --port 8080

Bind on 0.0.0.0 so other machines on your LAN can reach the API. --cache-type-k q8_0 --cache-type-v q8_0 halves the KV-cache memory with no measurable quality drop at sensible context lengths. -np 1 is required for MTP today.

A 15-minute narrated tutorial covers the same material: use cases first, then the benchmark, then the Ollama failure, then the working llama.cpp path, then an "Advanced" deep-dive into the install. The deck was rendered with Remotion and narrated with a local supertonic-tts M1 voice on the same RTX 4090.

Heads up — from here on, it gets technical. If you just wanted the recipe, you already have it. If you want to reproduce on your own machine, here's every step plus every wall I hit on the first run.

• NVIDIA RTX 4090, 24 GB VRAM (the magic number for 27B Q4_K_M)
• PopOS 22.04 (Ubuntu base), kernel 6.17
• CUDA toolkit 12.9 on disk with the matching 13.0 driver
• A separate data volume for the GGUF (/d/hugging_face_cache/) — keep model weights off the system drive

Pin a specific Python version and keep dependencies project-local so nothing leaks into the system. Watch out for any stray VIRTUAL_ENV that might be active in your shell — it picks up the wrong interpreter at venv creation time.

$ asdf local python 3.12.3
$ unset VIRTUAL_ENV
$ python -m venv .venv
$ .venv/bin/pip install huggingface_hub pillow matplotlib

$ git clone --depth 1 https://github.com/ggml-org/llama.cpp
$ cd llama.cpp
$ /usr/bin/cmake -B build \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=89 \
    -DLLAMA_CURL=ON
$ /usr/bin/cmake --build build -j$(nproc)

$ export HF_HOME=/d/hugging_face_cache HF_HUB_ENABLE_HF_TRANSFER=1
$ .venv/bin/hf download Jackrong/Qwopus3.6-27B-v2-MTP-GGUF \
      Qwopus3.6-27B-v2-MTP-Q4_K_M.gguf \
      --cache-dir /d/hugging_face_cache/hub

HF Xet does the heavy lifting under the hood — parallel chunks, resume on disconnect. About 3 minutes on a reasonable connection.

The first time the --spec-type draft-mtp --spec-draft-n-max 2 combination worked end-to-end. 76.7 t/s out of the gate. Once stable, bumping n-max to 3 picks up another 5–15 % on most prompts (and the headline 2.21× on math).

Nothing was smooth on the first pass. If you are following along at home, this is the section that will save you the most time.