Writing · 19 articles
Notes on building with language models
Building an inference engine from scratch, then the systems stack behind serving, then transformers from first principles.
Latest · November 2, 2026 · 12 min read
Mixture of Experts for LLM Inference
Why 671B can run like ~37B per token: FFN as memory, routers and top-k, fine-grained experts, shared experts, routing collapse, and the all-to-all serving tax.
Read it →Series I · 6 parts
Inference Engineering
From the memory wall to GPUs, kernels, quantization, parallelism, and MoE: how LLM serving actually works.
- iFundamentals of LLM InferenceClassical ML inference is cheap on a CPU. LLMs are not. This is the long walk from SIMD and O(n²) attention to compute vs memory bandwidth, HBM, prefill vs decode, the roofline, KV cache, and the two clocks you actually feel.Oct 8 · 16 min
- iiInside the GPU for LLM InferenceLast time we named the memory wall. This time we open the chip that creates it: SMs, warps, SIMT, HBM, tensor cores, low precision, and why NVLink beats the network long before you care about flops.Oct 12 · 18 min
- iiiKernels and Memory in LLM InferenceFlashAttention through four generations, paged KV, GQA and MLA, flash-decoding, CUDA graphs. Same attention math. Different routes for the bytes.Oct 18 · 20 min
- ivQuantization for LLM InferenceShrink the bytes themselves: float ladders, outlier features, W4A16 vs W8A8, GPTQ, SmoothQuant, AWQ, rotations, KV cache quant, and how to choose without believing vendor slides.Oct 26 · 12 min
- vParallelism for LLM InferenceWhen the model no longer fits: collectives, the NVLink gap, tensor and pipeline parallelism, expert and sequence parallelism, data-parallel copies, and why traffic decides placement.Oct 30 · 11 min
- viMixture of Experts for LLM InferenceWhy 671B can run like ~37B per token: FFN as memory, routers and top-k, fine-grained experts, shared experts, routing collapse, and the all-to-all serving tax.Nov 2 · 12 min
Series II · 7 parts
Transformers from first principles
From tokens and dot products to a trained GPT-2, one idea at a time.
- iBefore an LLM Can Read Your Prompt, It Has to Break It ApartBefore attention, transformers, or embeddings, every LLM turns your text into token IDs. Here's how tokenization works—and why BPE became the default compromise.Aug 31 · 10 min
- iiDot Product: The Magic Behind AttentionBefore Q, K, and V feel magical, you need one idea from linear algebra: the dot product measures how aligned two vectors are and that alignment is the heart of attention.Sep 3 · 7 min
- iiiSelf Attention: Going One Level DeeperThe dot product told us how to compare two vectors. Now we build self-attention on top of it why a static embedding isn't enough, what Q, K and V actually do, and what the whole thing looks like in code.Sep 7 · 6 min
- ivAttention: Causal Masking, MHA, MQA and GQAOne head looking at the whole sentence is not enough, and it is not what production models do. Causal masking, multi-head attention, and the two tricks that make long context affordable at inference time.Sep 8 · 8 min
- vMulti-Head Latent AttentionGQA shrinks the KV cache by sharing heads. MLA takes a different route: compress keys and values into a learned latent, cache that instead, and keep RoPE on a separate track so the math still works.Sep 12 · 5 min
- viPositional Embeddings: Why Attention Needs OrderAttention sees a bag of tokens, not a sequence. We try crude position hacks, watch them fail, then arrive at sinusoidal encodings and why the heatmap gets calm as depth increases.Sep 15 · 6 min
- viiI Trained a GPT-2 From Scratch on Tiny ShakespeareAfter writing about attention on paper, I wanted to feel the full stack: BPE tokens, causal MHA, Pre-LN blocks, AdamW, and a model that actually spits out Romeo dialogue on a Colab T4.Sep 18 · 7 min
Series III · 1 part
Group theory and position
Why every relative positional encoding is a turn, a fade or a slide.
Series IV · 4 parts
Building with LLMs
Essays on the systems around the model: routing, harnesses, token budgets and decision models.
- iReducing Token Consumption in LLM ApplicationsArchitectural patterns for lowering token usage in production AI systems—beyond max_tokens limits—to cut costs, reduce latency, and scale better.Jul 3 · 5 min
- iiHow I Think About Graph, Loop, and Harness EngineeringGraph, loop, and harness engineering aren't new buzzwords—they're layered patterns for designing agentic systems around a model.Aug 9 · 3 min
- iiiUnderstanding the Router Pattern in LLM ApplicationsRouting in LLM systems isn't one idea—it can mean choosing a model, an agent, a workflow, or a tool, and those problems are not the same.Aug 17 · 6 min
- ivSystem 1 Decision Models: Laya, Token by TokenLaya is a System 1 decision engine: one bidirectional pass, MASK-slot options, choice/score/noul readouts, and RLCD calibration. A token-by-token walkthrough of the architecture.Sep 24 · 12 min