Daily Dose of Data Science

72 techniques to optimize LLMs in production: (bookmark this) LLMs break every assumption about regular ML inference. A traditional ML model (CNN, transformer classifier, XGBoost) produces its output in a single forward pass. Nothing carries over between requests. The GPU does the same kind of work each time. LLMs don't work like that. The output is generated one token at a time, autoregressively, which turns a single request into hundreds of sequential forward passes. The prefill stage is compute-bound, while decode is memory-bandwidth-bound, and running them together on the same GPU hurts both. The KV cache grows with conversation length and is shared across requests, so routing is no longer about least-busy servers but about which replica already has the relevant prefix cached. MoE models add expert parallelism on top of that. None of this exists in traditional ML serving. That is why an entirely new stack of optimizations has emerged specifically for LLM inference, spanning compression, attention, KV cache management, batching, decoding, parallelism, and routing. I recently mapped out 72 techniques that optimize LLMs in production, grouped into nine pillars. I recently wrote an article (shared below) that covers how LLM inference differs from regular ML inference and why each of these pillars exists. Over to you: What other LLM optimization techniques would you add here?

RAG vs. Graph RAG vs. Agentic RAG, clearly explained! Standard RAG embeds documents into vectors and retrieves the most similar chunks via similarity search. For direct factual lookups, this works well. But it breaks down when a query needs to connect facts spread across multiple documents. Similarity search retrieves individual chunks, not the relationships between them. Graph RAG adds a knowledge graph layer on top. → During indexing, an LLM extracts entities and relationships from the documents. → During retrieval, the system traverses these connections instead of relying on embedding similarity alone. This is what enables multi-hop queries. Say a vector DB stores three facts about internal services: ↳ "The checkout service uses payments API." ↳ "The payments API runs on cluster-3." ↳ "Cluster-3 is scheduled for maintenance on Friday." Someone asks: "Will the checkout service be affected by Friday's maintenance?" Vector search can likely retrieve facts 1 and 3 because the query mentions "checkout service" and "Friday maintenance." But it will miss fact 2, which connects the payments API to cluster-3. That middle fact sits too far from the query in embedding space. It mentions neither "checkout" nor "maintenance," so it never makes it into the retrieved context. A knowledge graph connects these as linked entities, and graph traversal finds the full path in one query. Agentic RAG takes a different approach entirely. Instead of a fixed retrieval pipeline, an LLM agent decides at query time which tools to invoke, which sources to query, and in what order. Check the visual below to understand the three architectures thoroughly. One thing to note here is that these three aren't levels of sophistication that you need to graduate through. Instead, they solve different query types. ↳ Single-hop factual lookups → standard RAG ↳ Multi-hop relationship queries → Graph RAG ↳ Dynamic multi-source tasks with tool use → Agentic RAG Once the right architecture is in place, the next leverage point is efficiency. Most RAG architectures rely heavily on vector search, and that layer can be made 32x more memory-efficient using binary quantization. I covered the full implementation in the article below. 👉 Over to you: Which RAG architecture are you running in production?

NVIDIA released a new RL framework: Your entire setup is one Python file: - write a class with a step() function - return a reward from it - the framework handles the rest, up to 1T-class MoE training In a typical RL framework, the environment and reward logic are added inside the trainer's codebase. Adding a new task requires writing framework code, registering it, and wiring up config for it. The reward usually comes from a separately trained reward model, which needs its own GPUs and placement next to the actor and critic. NVIDIA's Molt framework drops both these requirements. Every run takes an agent_path flag pointing to one Python module, and the trainer code never changes. The module gives you two ways to plug in an agent. > The first is an Env class where you write a step() function that receives the model's output and returns a reward, and the framework handles everything else, i.e., calling the model, tokenizing, and looping turn after turn until the episode ends. > The second is a ChatAgent class, for agents that already exist as their own loop. Here, you write the loop yourself, calling the model through a standard OpenAI or Anthropic client, and the framework spins up a local server so those calls hit its vLLM engines instead of an external API. Either way, the trainer sees the same thing, i.e., a trajectory of tokens and a reward at the end. That reward can come from a string-match grader, code executed in a sandboxed subprocess, or an LLM-as-judge call. So there is no reward model to train or place, and no environment code living inside the framework. Under the hood, Molt runs a single trainable actor (plus an optional critic for PPO) on FSDP2, vLLM engines for rollout, and Ray queues between them, so generation, training, and weight sync overlap instead of running in lockstep. Scaling just requires setting some flags on the same command. The framework also splits the model across GPUs for you, so the script that trains an 8B model on one node can also train DeepSeek-V3-scale MoE models on a cluster by raising the parallelism flags. Here's the repo: github.com/NVIDIA-NeMo/labs-… Since the framework is new, I am still experimenting with it and will share more about my learnings soon. To dive deeper into how top AI labs are building RL Agents in 2026, I wrote a full article about it. It covers the evolution from RLHF to GRPO with verifiable rewards and using an LLM-as-judge reward function. This idea extends RL training to non-verifiable agentic tasks without writing custom reward functions. Read it below.

Chat with video using EmbeddingGemma 2 (100% local): EmbeddingGemma 2 by @GoogleDeepMind maps text, images, video, and audio into a shared vector space. This makes it possible to search video frames with natural language. Devs have already built native Rust implementations for local inference. Unsloth also provides a 176 MB, 4-bit GGUF checkpoint for text embeddings, although that version doesn't include the full multimodal pipeline. I tested the video-search use case by building a chat interface that finds matching frames and jumps to their timestamps. On my Apple M3 Pro, using the CPU: - Indexing took 29s for a 60s video - Search took 25 ms on average across 9 queries - The saved frame vectors occupied 180 KiB. The video is indexed once. Each new search embeds the query and compares it with the saved frame vectors. Processing stays entirely on the laptop. The video below depicts the entire flow in action. That said, model quantization reduces the memory needed for the weights. The retrieval index has its own storage cost, which grows as you add videos. To dive deeper into reducing that cost, I wrote a full article on binary quantization. It explains how storing vectors with one bit per dimension reduces their raw storage by 32x compared with float32, and includes a working retrieval implementation. Read it below.

8 AI model architectures, visually explained: LLMs get so much attention that it's easy to overlook the rest of the field. Different inputs, tasks, and deployment constraints have led to very different model designs. Here are eight specialized AI model architectures: 1. LLM (Large language models) Text is split into tokens, mapped to embeddings, and processed through transformer layers. The model generates its response by predicting successive tokens. ↳ GPT, Claude, Gemini, Llama. 2. LCM (Large concept models) Moves prediction to the sentence level. Sentences become SONAR embeddings, and the model predicts the next sentence representation. Meta explored both autoregressive and diffusion-based versions. ↳ Meta's LCM research introduced this approach. 3. LAM (Large action models) Connects a user's request to actions in an environment. These systems interpret intent, plan steps, and use interfaces or tools, often checking results before continuing. ↳ Rabbit R1, Microsoft UFO, Claude Computer Use. 4. MoE (Mixture of experts) A router assigns token representations to selected expert networks inside the model. Activating a subset of experts adds model capacity without running every expert for every token. ↳ Mixtral, DeepSeek, Switch Transformer. 5. VLM (Vision-language models) Combines visual representations with language processing. A vision encoder extracts image features, which are connected to a language model so it can answer questions about visual content. ↳ GPT-4V, Gemini, LLaVA. 6. SLM (Small language models) Uses fewer parameters to reduce memory and compute requirements. These models can run locally, with quantization often reducing their deployment footprint further. ↳ Phi-3, Gemma, Mistral 7B, Llama 3.2 1B. 7. MLM (Masked language models) Learns by predicting hidden tokens using context on both sides. This bidirectional training produces representations useful for retrieval, classification, and other language-understanding tasks. ↳ BERT, RoBERTa, DeBERTa. 8. SAM (Segment Anything Model) An image encoder processes the image, while a prompt encoder handles points, boxes, or masks. A mask decoder combines them to identify the requested regions. ↳ Meta's SAM supports interactive segmentation and image annotation. These aren't eight mutually exclusive categories. A small model can process images, and a language model can use MoE layers. Some labels describe architecture, while others describe training or intended use. There's another approach I kept outside the image because it alters how a model handles context rather than introducing a separate model family. Recursive language models, from MIT, put the input into a Python environment that a root model can inspect through code. The model can search or slice that input, then make recursive model calls on selected portions instead of loading everything into its own context window. The reported experiments handled inputs beyond 10 million tokens without retraining, with gains over the base model even on some tasks whose inputs already fit its context window. I recently wrote a full breakdown of how this works, including why accessing a large input and reasoning reliably over it are different problems. Read it below.

7 MoE routing strategies in production, clearly explained: (bookmark this) Sending every token to its highest-scoring experts can make an MoE model slower. The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused. With fixed capacity, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings. MoE routing has to select useful experts while distributing work efficiently. The seven strategies in this visual handle that tradeoff differently. > Token-choice routing Each token selects its top-k experts using router scores. The expert count per token stays fixed, but expert loads can vary. Load-balancing mechanisms help prevent traffic from concentrating on a few experts. > Expert-choice routing Each expert selects its highest-scoring tokens within a fixed capacity. Expert workloads stay balanced, while individual tokens can receive different amounts of expert computation. > Dynamic routing The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive, although router confidence doesn't directly measure token difficulty. > Balanced assignment A global matching procedure assigns tokens under expert load constraints. Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment adds overhead. > Hash-based routing A deterministic hash maps tokens to experts without a learned gate. This removes router learning, but expert selection has less flexibility to adapt to context. > Soft MoE routing Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens, changing what an expert processes. > Group-limited routing The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes. These choices affect how serving engines batch expert computation and distribute work across GPUs. Evenly distributing tokens doesn't guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time. To dive deeper into MoE engineering, I covered everything in an article recently, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations. It follows what happens after the router selects experts and explains the tradeoffs that determine latency and throughput. Read it below.

Another insane Grok @bot feature! Several apps exist because the workflows they support cannot be represented in a standard document. They need tables, charts, and controls that respond as someone interacts with the data. Pages in Grok Bot lets the bot build these directly into a document. You describe what you need in chat. The bot creates the Page and saves it to its Library, where you can reopen it, share it, and ask for changes. The same document can contain the explanation, the data, and the interface for exploring it. For simpler trackers and dashboards, this starts to cover what would otherwise require a separate app. I tested this myself by building a Notion-like Content OS with a pipeline board, KPI cards, and charts that update when I drag a card to Posted. The recording shows the complete setup in action. If you want to get started with Grok Bot, I wrote a full masterclass covering the shared computer, memory, skills, routines, and macOS setup. Read it below.

Harness Engineering 2.0 is here! We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too. And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side. You provide one task and then run it across several model x harness configs that you want to test. Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns: - total cost - execution time - whether the task passed - files and artifacts produced If you want to try it yourself, it is implemented within the open-source HarnessRouter project. Here's the repo: github.com/HarnessRouter/har… (don't forget to star it ⭐ ) HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like: - Codex - Hermes - Claude code - DeepSeek Harness - System One, powered by Jev - And 25 more agent harnesses In Harness Arena, the cheapest run does not automatically win. A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass. Here's the result from one Harness Arena test: - Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits - Claude Code + Opus 4.8: 3/5 successful runs using 223 credits That is around 474x difference in credits, while the more expensive configuration also failed two runs. You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task. This helps answer practical questions such as: - Should this task run on Codex or Claude? - Can a cheaper config still pass every check? - Which setup is best to handle this workload? Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures. So you are no longer choosing an agent stack based only on a model leaderboard. Instead, you can benchmark the complete runtime on the task your product actually needs to solve. To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves. It also covers the local setup, a working request, and how sessions and files move through the system. Read it below.

A new technique just made attention 23x faster! At 10M tokens, sparse attention slows down not because of attention itself but rather because of deciding which keys to attend to. Let's understand why and how a new technique solves it without scanning every key. 1) Full attention at long context: In full attention, every query is compared against every earlier key. This leads to O(T^2) complexity for prefill, and at 10M tokens it dominates the forward pass. 2) Sparse attention: Sparse attention lets each query read only a small set of <k> keys instead of all of them, so the attention becomes cheap. But something still has to pick those <k> keys, and at long context, this step becomes the bottleneck. 3) Why picking keys is still quadratic: DeepSeek's DSA uses a lightning indexer that scores every earlier key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of nearby keys. Each of them scans the whole context for every query, so the indexer stays O(T^2), and at 10M tokens, it takes 98% of the layer's time. Block methods have another issue. Blocks group keys by position, so one relevant key within filler can get averaged away in the block summary. 4) How LSA picks keys: Latent Sparse Attention (LSA) is a new approach that groups keys by meaning instead of position. - Each key is projected into a 64-dim index space, and every 4 consecutive keys are pooled into one. - When a 512-key block fills up, it's clustered into buckets of ~8 similar keys, with one centroid per bucket. Finished blocks are never rebuilt. - A query scores only the centroids, keeps the top 1,024 buckets, and expands them into 4,096 keys, plus 4 sink tokens and the 512 most recent keys. - Exact FlashAttention then runs over that list with an unchanged kernel. If you've built RAG, this is like cluster-based vector search running inside attention. The kernel design even cites FAISS. At 10M tokens, LSA's prefill is 72.4x faster than full attention on Qwen-3.8-27B and 22.7x faster than GLM-5.3-Flash's own sparse attention. The time to first token improves: - 9.2x on Qwen - 2.2x on GLM Below ~512k tokens, it's roughly break-even. Here's the paper: arxiv.org/abs/2610.42147 That said, LSA attempts to find the keys more cheaply, not storing them. The KV cache still grows linearly with context, and at 10M tokens, managing that memory becomes the next engineering problem. I wrote an article on how 12 ways serving systems handle it, covering paging, offloading, quantizing, and more. Read it below.