emilio andere

Kimi K3.1 Pro is live on Inference Router from day 0. K3.1 is @Kimi_Moonshot's most capable open-weight model yet, with a 1M context window and a reported 65.7% on Terminal-Bench 4.0. Run it in production with at least 99.9% uptime SLA, US hosting and ZDR available. No upfront commitment.

Introducing Inference Router: enterprise-grade inference on demand. Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, Qwen3.8 Flash, Muse Spark 1.3, Mercury 3 and more, available with an uptime SLA of at least 99.9%. US hosting and web search built in. you.com/router

we launched the most comprehensive ai performance engineering repo in the world follow and save to keep up with the series. links in thread 🧵 part 9: GPU Mode lectures 🐐 this one entry is a full resource repo of its own. GPU MODE collects lectures, slides, notebooks, and code covering GPU programming and the systems behind modern AI training and inference. the lectures help AI performance engineers diagnose slow training and inference and decide what to optimize. the examples show how kernel memory access, numerical precision, GPU communication, and runtime scheduling affect model performance. GPU MODE's speakers connect those problems to implementation details: - profiling with PyTorch and Nsight, tracing operations to GPU kernels and memory transfers, with timing examples that account for warmup and synchronization. - GPU memory and execution, using coalescing, shared-memory tiling, and occupancy to explain how thread mapping and data reuse affect performance. - kernel development with Triton, CuTe, and Gluon, including tensor layouts that map data to threads and memory, and control over asynchronous GPU operations. - attention kernels, working through FlashAttention's tiling and FlashAttention-3's overlap of data movement, matrix multiplication, and softmax on Hopper. - kernel fusion, inspecting code from torch.compile and writing custom kernels that reduce intermediate memory traffic, with examples that compare memory use and execution time. - low-precision computation, covering FP8 and FP4 formats, quantized optimizer states, and the rounding and scaling choices that affect numerical error. - distributed training, following PyTorch gradient synchronization into NCCL AllReduce and examining the communication required to combine results across GPUs. - inference serving, studying SGLang's CPU/GPU scheduling overlap and vLLM speculative decoding examples that weigh accepted tokens against drafting and verification costs. use the collection to build a study plan or investigate a bottleneck in a real workload. work through the relevant implementation, check its assumptions and numerical results, and measure the tradeoff on the hardware and input shapes that matter for the application.

this is a must-read if you want to be a top 1% ai performance engineers follow and save to keep up with the series. links in thread 🧵 part 8: Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems Amey Agrawal and coauthors explain how serving decisions shape the timing of streamed LLM responses. for AI performance engineers evaluating batching, chunked prefill, or speculative decoding, the paper provides a framework for assessing token delivery against application-specific timing targets. the authors connect scheduling behavior, token arrival traces, and service targets: - prefill and decode interference: a long prefill can delay ongoing decode work. chunked prefill divides that prompt processing across batches, so the amount of prefill work per batch can affect both first-token latency and interruptions to existing streams. - first-token latency: time to first token (TTFT) includes scheduling delay and prompt processing. the authors propose profiling isolated requests across prompt lengths and adding a scheduling allowance to set first-token deadlines. - averages and latency distributions: time per output token (TPOT) can hide generation pauses, while percentiles of time between tokens (TBT) omit where those pauses occurred within a response. per-request token arrival traces preserve the order and gaps observed by the client. - speculative decoding and buffering: several tokens can arrive in one burst. a client can buffer them for steady display, letting work completed ahead of the consumption rate cover some later gaps. - deadline-based evaluation: fluidity-index measures the fraction of deadlines met within a request. tokens arriving before their deadlines build slack. if a token misses its deadline, Etalon counts missed deadline slots and resets subsequent deadlines from that token's arrival. - fluid token generation rate: the highest playback rate inferred from the evaluated requests that meets a chosen fluidity target for a specified share of requests. this supports API comparisons under a defined fluidity requirement. - capacity planning: Etalon estimates the maximum request rate a replica can sustain under specified targets on the tested workload. this informs estimates of how many replicas a deployment needs. for a controlled deployment comparison, keep the model, hardware, workload, and service targets fixed, then use capacity search to compare serving configurations. engineers choose the settings to evaluate.

we gathered all the resources you'll ever need to become the most cracked ai performance engineer follow and save to keep up with the series. links in thread 🧵 part 7: Efficiently Scaling Transformer Inference Reiner Pope and coauthors studied PaLM inference on TPU v4. the paper gives performance engineers a way to reason about where weights, activations, and KV caches should live, what must move each step, and which transfers limit latency. the paper's mechanisms suggest these checks for an inference deployment: - prefill and decode can have different arithmetic intensity, meaning computation per byte transferred. prefill can process many prompt tokens in one pass; standard autoregressive decode produces one new token per sequence per step. larger decode batches amortize weight reads, but add sequence-specific KV cache traffic. sweep batch size with context length and profile the linear layers and attention to locate the bandwidth or compute limit. - tensor parallelism can reduce local compute without removing the communication bottleneck. in the paper's 1D layout, activation-aggregation time stays roughly constant as chip count grows for a fixed workload. 2D partitioning splits both weight dimensions so each chip communicates smaller activation shards. evaluate this against the actual interconnect and matrix dimensions before increasing the parallelism degree. - the best tensor to move changes with the token batch. weight-stationary layouts exchange activations; weight-gathered layouts transfer more weights to reduce activation exchange. large prefills can justify that trade because their activation tensors are large. count the tokens processed per forward pass, including prompt tokens during prefill, when comparing those layouts. - multiquery attention shares one KV head across query heads, but a head-sharded layout can replicate that cache across devices. the paper shards attention across the batch during decode and exchanges current-step activations to reduce cache reads per chip. in vLLM's grouped-query attention deployments, check whether tensor parallelism exceeds the KV-head count and introduces cache replication before estimating per-GPU capacity. - KV cache capacity and prefill attention intermediates are separate memory problems. at a fixed batch size and query-head count, sharing KV heads does not shrink the full attention-score tensor. for dense prefill, that tensor's element count grows with the square of sequence length. Appendix G discusses microbatching to reduce temporary allocations and FlashAttention to avoid materializing the full score matrix. use that distinction to identify whether a memory failure comes from retained KV state or temporary attention allocations. - communication overlap depends on the execution schedule and tensor layout. the authors use Looped CollectiveEinsum to overlap collectives with matrix multiplication, including choosing output sharding that exposes overlap opportunities. inspect a GPU timeline for exposed communication and dependent compute stalls; an asynchronous collective does not establish that its cost is hidden. - quantization changes different bottlenecks depending on what is quantized. the paper stores weights in int8 while retaining bfloat16 matrix arithmetic. this reduces weight-loading traffic, but does not provide faster arithmetic for compute-bound batches. evaluate weight, activation, and KV cache precision against the measured bottleneck and benchmark the effect on both phases. - benchmark the operating point that the application needs. the paper compares latency with accelerator-time per token and model FLOPS utilization. in a serving system, track time to first token, inter-token latency, and throughput under the target load. keep queueing time separate from prefill time. the practical use is to narrow an optimization experiment: estimate compute, HBM traffic, and collective costs for the actual workload, inspect the bottleneck, change the relevant layout or precision, and remeasure at the same latency target.