DEV Community curation

Insights, Ideas & Innovation

Explore the latest insights, development trends, practical guides, and technology stories from the global developer community.

Cover for Porting Gemma 4 E2B to SGLang-JAX on One Cloud TPU v5e
#googlecloud#tpu

Porting Gemma 4 E2B to SGLang-JAX on One Cloud TPU v5e

SGLang-JAX serves the Gemma 4 31B and 26B A4B on TPU and fails to load E2B. A two-file patch adds per-layer embeddings, shared K/V and the double-wide MLP; on one flex-start v5e-1 it then serves 159.9 tokens per second for one request and 4,249.4 at 64, 2.52x vLLM, with GSM8K level with the bf16 JAX engine.

Nov 910 min read
70Read Article
Cover for Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B
#googlecloud#tpu

Six Serving Engines on One Cloud TPU v5e Chip: Gemma 4 E2B

vLLM, SGLang-JAX, a pure-JAX engine, a Rust model over PJRT, and TorchTPU in Python and Rust serve Gemma 4 E2B on one flex-start v5e-1, with the same prompts, the same 1-to-64-request sweep and the same GSM8K questions. vLLM is the engine that serves out of the box; SGLang-JAX with an E2B patch is the fastest at every load from 8 requests up.

Nov 912 min read
132Read Article
Cover for Meme Monday
#discuss#watercooler

Meme Monday

Meme Monday! Today's cover image comes from the last thread. DEV is an inclusive space! Humor in...

Nov 91 min read
3912Read Article
Cover for Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
#gemma#vllm

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions. GSM8K is level on every engine; vLLM is the fastest up to 16 requests and within 3% of SGLang beyond.

Nov 812 min read
161Read Article