Back to Blogs
#gemma#tpu#jax#colab

Gemma 4 on a Hosted Colab TPU: The QAT Repack, Served in Two Minutes

One notebook loads the Gemma 4 E2B QAT repack onto a Colab v5e-1, serves it behind an OpenAI-compatible endpoint with a chat page, and steps through thinking, sampling, stop sequences, long prompts and a CPU comparison.

12 min read
Cover for Gemma 4 on a Hosted Colab TPU: The QAT Repack, Served in Two Minutes

This article provides a step by step guide to serving Gemma 4 E2B on a hosted Colab v5e-1 runtime: one TPU v5e chip. A single notebook downloads the QAT repack, loads it with a pure-JAX engine, serves it behind an OpenAI-compatible endpoint with a browser chat page, and then exercises the serving options an application uses.

https://github.com/xbill9/colab

The model is ready to chat 117 seconds after Run all, and decodes at 123.7 to 124.6 tokens per second. One Colab v5e-1 runtime, 2026-11-17:

Seconds after Run all Stage
8 chip and versions read
63 6.56 GB checkpoint downloaded
93 weights on the chip
116 first request compiled
117 server and chat page up
296 six more chat shapes compiled in the background

What Is Being Served?

The model is xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text. Google trained Gemma 4 E2B with quantization-aware training, which holds every weight on a 4-bit grid during training. The repack stores those trained 4-bit levels and their trained scales in the W4A16 format, with the vision and audio towers removed.

Build Download
Google gemma-4-E2B-it-qat-w4a16-ct 8.32 GB
Repack gemma-4-E2B-it-qat-q4_0-w4a16-ct 7.51 GB
Repack, text only (used here) 6.56 GB
Repack, int4 embeddings too (-text-emb4) 2.83 GB

Google's W4A16 export re-rounds every group of 32 weights onto a new grid. Served by vLLM on one v5e chip, the two builds decode at the same speed, 136.6 and 136.5 tokens per second, and the repack scores 0.678 against 0.655 on a 3,880-record test suite. The -text-emb4 build is the smallest download, and the engine used here reads bf16 embedding tables, so the notebook uses -text.


At This Point You Should Have…

  • A Google account with Colab access. A TPU v5e-1 runtime is listed as eligible on the free tier for this account.
  • Nothing else. Every download is public and no Hugging Face token is needed.

Step 1 — Open the Notebook

Two notebooks share the same first two sections. gemma4_e2b_basic_v5e1 stops once the model is served and the chat page is up. gemma4_e2b_advanced_v5e1 continues through every step below. Open one in Colab, pick Runtime → Change runtime type → v5e-1 TPU, then Runtime → Run all.

Both are built from jupytext sources in src/parts/, so the shared cells exist once. Executed copies from real v5e-1 runs are in runs/2026-11-17/.


Step 2 — Read the Chip While the Model Downloads

The first cell starts the download in its own process and returns at once:

download = subprocess.Popen(
    [sys.executable, "-c", DOWNLOAD, MODEL_ID, MODEL_REV],
    stdout=subprocess.PIPE,
    stderr=download_log,
    text=True,
)
Enter fullscreen mode Exit fullscreen mode
downloading xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text @ 315aec06 in the background (pid 2547)
Enter fullscreen mode Exit fullscreen mode

While it runs, the next cell reports the software, the chip and the host:

software
  Python        3.13.18
  jax           0.11.2
  jaxlib        0.11.2
  libtpu        0.0.50
  transformers  5.22.1
  huggingface_hub 1.33.0

TPU
  devices       1 x TPU v5 lite
  HBM           16.91 GB, 0.00 GB in use
  $TPU_ACCELERATOR_TYPE = v5e-1

host
  CPU           24 x AMD EPYC 7B13
  RAM           50.5 GB
  free disk     241 GB
Enter fullscreen mode Exit fullscreen mode

Then two kernels show what the chip does:

bf16 matmul 8192x8192: 6.17 ms  ->  178 TFLOP/s
1 GiB elementwise:     3.62 ms  ->  593 GB/s of HBM traffic
Enter fullscreen mode Exit fullscreen mode

Decoding one token reads every weight from HBM, so the bandwidth figure is the one that sets decode speed.


🔎 Tip: Start the Long Download First

A 6.56 GB checkpoint takes about a minute on Colab. Started in the first cell, it finishes while the version and chip cells run, and the load cell waited 0 seconds for it:

checkpoint: 6.56 GB, done 63 s after the first cell (this cell waited 0 s for it)
Enter fullscreen mode Exit fullscreen mode

The download runs in a separate Python process. When that process exits, the VM gets back the host RAM the downloader held.


Step 3 — Pick the Engine That Answers First

Start-up time decides the engine for a demo. Three were measured on this chip type:

Engine Time to first answer Where it runs
tpu-jax (used here) 117 s after Run all, download included TPU
vLLM on TPU, tpu-vllm-v5e1-2b rig 17 min from TPU grant, 16 min on a container restart TPU
llama.cpp llama-server 2.6 s from launch to healthy host CPU

tpu-jax is a Gemma 4 E2B decoder in pure JAX. It compiles only the shapes a request uses. llama.cpp starts fastest because it has nothing to compile, and it has no TPU backend, so on this runtime it uses the host CPU. Step 12 measures what that costs.


Step 4 — Load the Weights

The engine is cloned at a pinned commit. dequant_at_load=True unpacks the 4-bit weights to bf16 once, at load:

ENGINE = JaxGemmaEngine(
    MODEL_ID, quant_mode="w4a16", max_model_len=MAX_MODEL_LEN, dequant_at_load=DEQUANT_AT_LOAD
)
ENGINE.load(local_dir=str(MODEL_DIR))
Enter fullscreen mode Exit fullscreen mode
engine: tpu-jax 4b9f8e9, dequant_at_load=True, 9.26 GB of weights on the chip, loaded in 17 s
HBM in use: 9.27 of 16.91 GB
Enter fullscreen mode Exit fullscreen mode

On a Colab v5e-1 on 2026-10-21, the same engine and repack decoded at 130.8 tokens per second unpacked and 103.1 packed, with identical tokens, using 9.30 GB of HBM against 6.62 GB. The chip has room for the unpacked weights, so the notebook defaults to True.


Step 5 — Serve It Behind an OpenAI Endpoint

A standard-library HTTP server runs in a background thread, with routes shaped like vLLM's: /health, /version, /v1/models, /metrics and /v1/chat/completions. Any OpenAI client works against it.

Thinking is a chat-template switch, passed the way vLLM takes it, in chat_template_kwargs. The model writes its reasoning between the special tokens <|channel> and <channel|>, and the server returns that text as reasoning_content and the rest as content:

class ReasoningSplitter:
    """Turn generated token ids into (kind, text) deltas, kind being 'reasoning' or 'content'."""

    LABEL = "thought\n"

    def feed(self, tid):
        if tid in (CH_OPEN, CH_CLOSE):
            self.kind = "reasoning" if tid == CH_OPEN else "content"
            self.ids, self.sent = [], 0
            return None
        self.ids.append(tid)
        text = TOK.decode(self.ids, skip_special_tokens=True)
        # the "thought" label and half-decoded characters are held back here
        delta, self.sent = text[self.sent :], len(text)
        return (self.kind, delta) if delta else None
Enter fullscreen mode Exit fullscreen mode

One short request compiles the shape most chat turns use:

warm-up: 'Hello! How can I help you today?' in 24 s, including the compile
OpenAI-compatible server on http://127.0.0.1:8000/v1
Enter fullscreen mode Exit fullscreen mode

🔎 Tip: Round max_tokens Up So One Program Serves Many Requests

The engine compiles one program per pair of padded prompt length and answer budget. Prompts pad to 64, 128, 256 and so on. The server rounds the answer budget up to 512, 2048 or 4096 tokens and cuts the stream at the requested max_tokens:

budget = next((b for b in ANSWER_BUDGETS if b >= max_tokens), max_tokens)
Enter fullscreen mode Exit fullscreen mode

A request for 16 tokens and one for 500 share one compiled program. Every response carries x_new_shape: true when serving it compiled a new one, so a slow first token can be traced to a compile.


Step 6 — The Chat Page

The chat page is the repository's ui/chat_ui.py, adapted from the tpu-vllm-v5e1-2b rig's page. It is a small standard-library proxy that serves the page and forwards /v1/* to the server, and it reads the server's tpu_jax: counters, or vLLM's when pointed at a vLLM server. The notebook starts it and shows it in the cell output through Colab's port proxy:

output.serve_kernel_port_as_iframe(UI_PORT, height=720)
Enter fullscreen mode Exit fullscreen mode

Its "what is running" panel reads the server:

  "model": {
    "id": "xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text",
    "owned_by": "tpu-jax",
    "max_model_len": 4096
  },
  "version": "tpu-jax 4b9f8e9 (JAX 0.11.2)",

READY: model served and chat page up 117 s after the first cell started
Enter fullscreen mode Exit fullscreen mode

The side panel has max tokens, temperature, top-k and a system prompt, and shows each reply's first-token time and decode rate. The page sends no thinking flag, so the server's THINKING_DEFAULT decides. Run THINKING_DEFAULT = True in a new cell and the next reply shows its reasoning in a collapsible section.


Step 7 — Compile the Chat Shapes in the Background

A conversation's first turns move through the 128, 256 and 512-token prompt buckets, and a thinking answer needs the 2048-token budget. A background thread compiles those shapes one at a time, holding the engine lock for each, so a chat message sent meanwhile waits for at most one compile:

background compiles (waited 178 s for them here):
  prompt bucket  128, answer budget  512:  25.5 s
  prompt bucket  256, answer budget  512:  29.0 s
  prompt bucket   64, answer budget 2048:  23.1 s
  prompt bucket  128, answer budget 2048:  25.7 s
  prompt bucket  512, answer budget  512:  46.6 s
  prompt bucket  256, answer budget 2048:  29.2 s
Enter fullscreen mode Exit fullscreen mode

Every sample after this point runs on a warm shape.


Step 8 — Thinking Off and On

Every sample goes through the official openai client. The same question, greedy, both ways:

r_think = ask(
    "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?",
    thinking=True,
    temperature=0,
    max_tokens=2048,
)
Enter fullscreen mode Exit fullscreen mode
[stop] prompt 43 tok, completion 391 tok, first token 0.19 s, decode 123.7 tok/s, total 3.3 s
[stop] prompt 50 tok, completion 857 tok, first token 0.17 s, decode 123.9 tok/s, total 7.1 s

thinking used 857 tokens against 391 without; wall time 7.1 s against 3.3 s
Enter fullscreen mode Exit fullscreen mode

Both answers say the ball costs $0.05. Thinking more than doubles the tokens and the wall time, at the same decode rate.


Step 9 — Sampling, Seeds and Top-k

A fixed seed repeats a sampled answer exactly, and other seeds vary it:

seed 7: Sweet treats after dark.
seed 7: Sweet treats after dark.
seed 8: Stay hungry after dark with our magical late-night bakery.
seed 9: Wake up your night with delicious treats from [Bakery Name]!

seed 7 repeated exactly: True
distinct answers across seeds 7, 8, 9: 3 of 3
Enter fullscreen mode Exit fullscreen mode

top_k travels in the request body, because it is outside the OpenAI schema:

greedy                      124.6 tok/s  (157 tokens)
temperature 1, top-k off    123.1 tok/s  (145 tokens)
temperature 1, top-k 64     119.4 tok/s  (145 tokens)
Enter fullscreen mode Exit fullscreen mode

On the vLLM rig on the same chip type, top-k 64 took decode from 125 to 45 tokens per second. In this engine it costs about 4 percent.


Step 10 — max_tokens and Stop Sequences

max_tokens=20:
Mercury: The smallest planet in our solar system and the closest to the Sun.
Venus: The

[length] prompt 26 tok, completion 20 tok, first token 0.16 s, decode 125.9 tok/s, total 0.3 s

stop=['Saturn']:
Mercury
Venus
Earth
Mars
Jupiter

[stop] prompt 26 tok, completion 12 tok, first token 0.01 s, decode 126.5 tok/s, total 0.1 s
Enter fullscreen mode Exit fullscreen mode

"Saturn" arrives as more than one token. The server holds back any text that could be the start of a stop sequence until the next tokens settle it, so the answer ends cleanly at "Jupiter".


Step 11 — Multi-Turn Chat and JSON

The client keeps the history and sends it back each turn, with a system prompt setting the voice:

You are in the North Atlantic Ocean. This is near the coast of North America.

The nearest land is the coast of Newfoundland, Canada. It is a relatively close point.

[stop] prompt 79 tok, completion 18 tok, first token 0.13 s, decode 118.8 tok/s, total 0.3 s
Enter fullscreen mode Exit fullscreen mode

JSON by instruction, then parsed:

{"name": "Ada", "city": "Lisbon", "year": 2019}

parsed: {'name': 'Ada', 'city': 'Lisbon', 'year': 2019}
Enter fullscreen mode Exit fullscreen mode

The server does not constrain decoding to a schema, so the cell checks that the reply parses.


Step 12 — Long Prompts and the Context Limit

Each prompt length lands in a new bucket, so the first call compiles. The second call shows the time a user sees once the shape is warm:

prompt tokens first call s warm first token s decode tok/s
          108         0.15              0.015          0.0
          668        50.78              0.042         58.5
         2588        41.84              0.230        123.6
         3868        43.15              0.232         80.8
Enter fullscreen mode Exit fullscreen mode

The context limit is 4,096 tokens. With the weights unpacked, an 8,192-token prefill asks XLA for 16.77 GB against 15.75 GB free, so the server refuses longer prompts with a clear error and keeps serving:

HTTP 400: prompt is 6414 tokens; this server's limit is 4095

server still answering: OK.
Enter fullscreen mode Exit fullscreen mode

Step 13 — Call a Deployed Endpoint (Optional)

The same ask() helper takes any OpenAI client. Add a Colab secret named REMOTE_OPENAI_BASE_URL, for example http://<tpu-ip>:8000/v1 for the vLLM rig, and section 4 repeats thinking off, thinking on and a 256-token decode against it. Without the secret it prints a skip line:

no REMOTE_OPENAI_BASE_URL set: skipping the deployed-endpoint comparison
Enter fullscreen mode Exit fullscreen mode

vLLM returns reasoning in its own field only when it is started with a reasoning parser (--reasoning-parser gemma4).


Step 14 — llama.cpp on the Host CPU

xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf holds the same trained weights as a 2.62 GB GGUF. The notebook downloads llama.cpp release b12061 and starts its server with the reasoning parser:

llama = subprocess.Popen(
    [
        str(LLAMA_DIR / "llama-server"), "-m", str(gguf), "--host", "127.0.0.1", "--port", str(LLAMA_PORT),
        "-c", str(MAX_MODEL_LEN), "-t", str(threads), "--jinja", "--reasoning-format", "deepseek", "--no-webui",
    ],
Enter fullscreen mode Exit fullscreen mode
llama.cpp b12061 and 2.62 GB GGUF fetched in 27 s
llama-server (24 threads) healthy 2.6 s after launch; first answer 3.8 s later
Enter fullscreen mode Exit fullscreen mode

The same three requests, through the same client:

llama.cpp CPU: thinking off                  43   392        1.79          3.8    stop
llama.cpp CPU: thinking on                   50   870        2.81          3.7    stop
llama.cpp CPU: greedy 256                    21   163        1.89          4.0    stop

greedy decode, 256 tokens: TPU 124.6 tok/s, CPU 4.0 tok/s (31.2x)
Enter fullscreen mode Exit fullscreen mode

Compare and Contrast

Gemma 4 E2B on a Colab v5e-1 runtime Ready after Decode tok/s Thinking-on answer
tpu-jax on the TPU 🥇 117 s after Run all 124.6 7.1 s
llama.cpp on the host CPU 🥈 2.6 s after launch 4.0 235 s
vLLM on the TPU (rig, 2026-11-03) 17 min from TPU grant 125 not run here

So, Which One?

For a live demo on a Colab TPU, serve with tpu-jax: chatting in under two minutes, 123.7 to 124.6 tokens per second, an OpenAI-compatible API that existing clients use unchanged, and a chat page embedded in the notebook.

For many users at once, deploy vLLM, which batches requests, and point section 4 at it. This engine serves one request at a time.

Keep llama.cpp for machines with no accelerator. On this runtime it starts in seconds and decodes at 4 tokens per second.


Summary

The goal of this article was to serve Gemma 4 E2B on a hosted Colab TPU quickly enough for a live demo and to step through the common serving options. The key to the solution was overlapping the download with the chip cells, an engine that compiles only the shapes a request uses, and a background thread that compiles the chat shapes after the server is up. The Colab v5e-1 results were:

  • 🟢 Server and chat page up 117 seconds after Run all
  • 🟢 123.7 to 124.6 tokens per second, with thinking on or off
  • 🟢 Thinking on uses 857 tokens against 391, returned as reasoning_content
  • 🟢 Seeds repeat exactly; top-k 64 costs about 4 percent
  • 🟢 Stop sequences end cleanly; over-limit prompts get an HTTP 400 and the server keeps answering
  • 🟢 A chat page embedded in the notebook shows what is running and each reply's speed
  • ⚠️ One request at a time; the context limit is 4,096 tokens
  • ⚠️ A new prompt size compiles on first use, 23 to 51 seconds
  • ❌ llama.cpp has no TPU backend: 4.0 tokens per second on the host CPU, 31.2x slower

Scope: Gemma 4 E2B-it QAT repack, text only, one Colab v5e-1 runtime on 2026-11-17, one run of the notebook through the Colab CLI, one request at a time, greedy decoding unless stated. The packed-against-unpacked comparison comes from a Colab v5e-1 run on 2026-10-21, the vLLM start-up and top-k figures from the tpu-vllm-v5e1-2b rig on 2026-11-03, and the repack-against-Google figures from vLLM on one v5e chip on 2026-09-29. Section 4, the deployed endpoint, was skipped in this run.

The strategy for serving Gemma 4 on a hosted Colab TPU was validated with an incremental step by step approach.


References