E5Inference & serving

Inference and serving is the engineering of getting tokens out of a trained model quickly and cheaply, which is where most of an LLM's real-world cost actually lives. A model that is accurate but too slow or too expensive to run never reaches users, so this discipline treats latency, throughput, and memory as first-class concerns.

You will implement the decoding strategies that shape every output (greedy, beam, top-k, nucleus, temperature), then the KV cache that cuts per-token cost from quadratic to linear. From there you cover speculative decoding with a draft model (and the proof that it preserves the target distribution), post-training quantization with GPTQ, AWQ, and GGUF, and the memory management behind PagedAttention and continuous batching. The discipline closes with how production engines like vLLM, TGI, and SGLang combine these techniques.

This matters the moment you move from a notebook to a service: the same techniques underlie LLM serving in pillar I and the cost-and-latency engineering of pillar J.