Building an LLM Inference Engine from Scratch

Mon, 01 Jan 0001 00:00:00 +0000

Building an LLM Inference Engine from Scratch

I recently started exploring machine learning and AI from a systems engineering perspective. While most people begin by focusing on model training, I found myself much more interested in the inference side of the stack.

I wanted to understand how deep learning models and neural networks work under the hood, but what fascinated me even more were the engineering breakthroughs that made modern LLM serving possible. Everywhere I looked, people were discussing innovations such as PagedAttention in vLLM, RadixAttention in SGLang, continuous batching, speculative decoding, fused attention kernels, and distributed inference.

GPU, CUDA, and the Ecosystem A Progressive Learning Path

Mon, 01 Jan 0001 00:00:00 +0000

Tier 1: GPU Architecture (deep understanding)

Stanford CS149 Lecture 7 — Kayvon Fatahalian

The best lecture on SIMT execution model, warp scheduling, memory hierarchy, and occupancy. Start here. Videos on YouTube.

URL: https://cs149.stanford.edu

“Programming Massively Parallel Processors” — Kirk, Hwu, El Hajj (4th ed.)

The canonical textbook. Chapters 1-4 = architecture fundamentals. 4th edition covers tensor cores and Hopper.

URL: https://www.sciencedirect.com/book/9780124159921/programming-massively-parallel-processors

Brendan Lynskey — NVIDIA GPU Architectures Series

Deep dives on warp scheduling internals, SIMT vs SIMD nuance, memory latency per architecture, TMA on Hopper. Diagrams + assembly snippets.

Hari's Blog

Building an LLM Inference Engine from Scratch