Colin Lowenberg

Our talk at @aiDotEngineer World's Fair SF is now live! 🎥 @dylan and I explored "Optimizing Open Models for Production" - the techniques that make LLM inference faster, more efficient, and cheaper at @nebiustf. We covered a few key optimizations: - KV caching - Reuse previously computed key-value states instead of recomputing them. One of the biggest performance wins, with 5-10x speedups in some scenarios. - Cache-aware routing - Route requests to workers that already have the relevant KV cache, improving cache hit rates and reducing redundant computation. - Speculative decoding - A smaller model drafts tokens while a larger model verifies them, accelerating generation without sacrificing output quality. - Disaggregated prefill and decoding - Prefill is compute-heavy, while decoding is memory-bandwidth-heavy. Running them on separate GPU pools lets us optimize each independently. - Quantization - Smaller models, less memory, faster inference. But aggressive quantization can hurt quality, so finding the right balance is key. Lots of practical techniques for getting more performance out of open models! 🎥 Watch the talk: youtube.com/watch?v=TRe1u7dH… 📊 Slides: sujee.dev/events/2026-06-29_…