Akshay 🚀

My lecture at MIT!✨ From Physics to Linear Algebra & Machine learning, I have learned a lot from MIT! Yesterday, I had the honour of delivering a guest lecture on The state of AI Engineering, exploring: - Prompt Engineering - Retrieval Augmented Generation. - Fine-Tuning Large Language Models. - And, how LLMs are bringing a paradigm shift Thank you @RoyShilkrot for the invitation! 🙏 Grateful for the opportunity to engage with such brilliant minds! If you are interested AI Engineering, I write a free weekly newsletter → @ML_Spring

Your agent remembers too much. Decision models like Jev can fix that. Let me explain: A common agent-memory pipeline sends conversations, tool results, and task outcomes to an extractor. The extractor proposes possible memories, including user preferences, facts, and procedures. Take a travel assistant, and look at what the user says: - I usually prefer an aisle seat. - For this flight, book a window seat. - The flight is delayed by 25 minutes. These statements should not be stored in the same way. The aisle-seat preference can help with later bookings. The window-seat request belongs to a single flight. The delay has no use once the flight is over. A strong memory system may already make this kind of distinction. It can record where a fact came from and resolve it against facts already stored. It can also track when a fact becomes valid or invalid. If the source has enough context, the window-seat request can stay tied to one trip. So you should not add a decision model just to repeat what the memory system already does. A decision model becomes useful when the app has its own rules for what may enter long-term memory. A travel app might require that: - Only preferences the user stated or confirmed become account-level memories. - A preference inferred from one booking stays attached to that trip. - Passport numbers and payment details never go into general memory. - Ambiguous candidates stay in the session until the user confirms them. These rules need semantic judgment. Once a candidate is classified, ordinary code can enforce the policy. The hard part is telling a lasting preference from a one-time instruction, or from weak evidence for an inferred preference. A decision model can make that decision before the app writes anything to long-term memory. For each candidate, the app sends the decision model the proposed fact and its source evidence. It also sends the current user, trip, and session state. It can ask typed questions like these: - Did the user state the preference in explicit terms? - Should it apply past the current trip? - What retention scope should it have? - Would reuse on a later booking create a real risk? The decision model returns choices, scores, and probabilities. Several questions can share the same state in a single request. Application code then applies the thresholds: ↳ Store the aisle-seat preference for the user. ↳ Keep the window-seat request attached to this trip. ↳ Leave the delay in the current session. ↳ Reject sensitive fields before they enter general memory. The app can then send approved candidates as episodes to a memory system such as Graphiti. Graphiti runs extraction, entity resolution, provenance, temporal updates, and retrieval. Graphiti is open source if you want to inspect that memory layer: github.com/getzep/graphiti If Graphiti's current ingestion already meets the needs of your app, a decision model is unnecessary. Add a decision model when you need to check product-specific rules before anything is sent to Graphiti. That includes whether a preference was confirmed, and whether a fact is safe to store. If decision models are new to you, I explained how Jev works in the article quoted below.

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: github.com/shepherd-agents/s… (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. I wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. all of this is done using @Cometml's open-source project Opik. The article is quoted below.

NVIDIA just fixed text-to-SQL’s biggest problem! the release is called Auto Ontology, and it targets the reason an agent can write valid SQL against a company database and still return the wrong number. the database stores columns, and the agent has to guess what they mean. a column called extendedprice might hold revenue, and two systems in the same company can define revenue differently. Auto Ontology gives the agent an ontology, a schema that maps business concepts to the columns that hold them and defines how they relate, before any query is written. it drafts that schema from your database metadata instead of making you write it, and a human certifies it. the ontology holds three things: → terms: business concepts like customer, order, or revenue, with descriptions and synonyms, mapped to the exact columns that store them. → SQL attributes: reviewed expressions like a margin or a rolling total, reused so every measure is computed the same way each time. → relationships: how tables join, including bridge tables, inferred from keys and the data itself. to draft these, it profiles columns, samples values, extracts terms table by table, and mines past query history for SQL attributes. everything lands in Postgres, with embeddings in pgvector. at query time, the agent maps the question to certified terms, follows relationships to find the joins, and writes SQL. it checks filter values against the real data, repairs the query if a check fails, and returns the answer with its SQL and steps. when the data can't support a question, the agent says so. in NVIDIA's demo, a question about return on ad spend on a sales database with no ad data gets a clarifying reply offering revenue instead of an invented number. so the ontology tells the agent how to compute an answer and also which questions have one at all. as the architecture in the image shows, an ingestion service compiles the ontology from your source databases, and any agent harness reaches it through MCP, CLI, or REST API. it supports ten engines including Postgres, Snowflake, Databricks, and DuckDB, runs on Nemotron by default or any OpenAI-compatible endpoint, and is Apache 2.0. check it out on GitHub: github.com/NVIDIA/auto-ontol… the same principle of defining a schema before the model extracts anything also decides whether agent memory works. I wrote about that in the article quoted below.

Pages unlock the real power of Grok Bot Your bots can now publish their work as live pages instead of leaving it buried in chat. Here's how they work & 3 things I built: - A live HQ office for my 14-bot organisation - A custom analytics dashboard - Interactive learning guides

As an AI engineer, please learn: - Learn the roofline model and why decode is memory-bound - Deploy vLLM and SGLang, then read their schedulers - Understand paged attention from the code, not the blog post - Build observability before you optimize anything - Track TTFT, inter-token latency, throughput, queue depth - Use Grafana + Prometheus for inference dashboards - Turn on prefix caching and find which workloads it helps - Learn continuous batching and chunked prefill - Run load tests with 1000+ concurrent requests - Report p50, p95, p99, never just the mean - Master quantization tradeoffs (FP8, INT4, AWQ, GPTQ) - Learn speculative decoding and where it stops helping - Set up KV cache eviction for long contexts - Try disaggregated prefill and decode serving - Learn Kubernetes for AI workloads and autoscale on queue depth - Learn how inference costs break unit economics - Build your own model router by cost, latency, quality - Create a token budgeting system per request - Build one inference service and benchmark it publicly - Read inference research instead of model release news - Start sharing your optimization benchmarks I put together a 10-week plan that covers every one of these at 30 minutes a day. It is 50 sessions, split between reading the theory and building on your own service, and all of them feed one artifact: an inference service you deploy, instrument, load test past 1000 concurrent requests, tune, and publish as a reproducible benchmark. It is open on GitHub and contributions are welcome, especially newer sources worth adding. I am working through it myself and will share more content on this going forward, so stay tuned. GitHub repo: github.com/patchy631/time-to… (don't forget to star 🌟)

Temperature, top-k and top-p, clearly explained! (how an LLM picks its next token) An LLM never picks a word directly. At every generation step, it gives a raw score to every token in its vocabulary. It then turns those raw scores into probabilities and samples one token. Temperature, top-k and top-p shape that sampling step. None of them touches the model's weights. Here's how a typical sampler applies them: ↳ Temperature: it rescales the raw scores. Low values sharpen the distribution, so a few tokens dominate. High values flatten it, so unlikely tokens get a real chance. ↳ Top-k: it keeps only the k most likely tokens. With top_k at 50, it drops every token ranked below 50, whatever the distribution looks like. ↳ Top-p: it keeps the smallest set of tokens whose probabilities add up to at least p. With top_p at 0.9, an obvious next token leaves only a few tokens, and an open-ended one leaves many. ↳ Sample: the sampler picks one token from the tokens that remain. The key difference: top-k always keeps the same number of tokens, while top-p adapts to how confident the model is. Low temperature doesn't make a model more correct, though. It only reduces randomness, so a model without the right context can still be confidently wrong. Good starting points for temperature: - Classification and extraction: 0.0 to 0.2 - Code generation: 0.0 to 0.3 - General chat: 0.5 to 0.8 - Brainstorming: 0.7 to 1.2 Tune temperature first, then add top-p to trim the long tail of unlikely tokens. While you evaluate, change one setting at a time instead of all three at once. The diagram below shows the complete flow. - Temperature reshapes the probabilities. - Top-k keeps a fixed number of tokens. - Top-p keeps just enough tokens to reach a set share of probability. I wrote a full breakdown of LLM inference in the article below.

Grok Bot masterclass. In this video, I cover everything you need to understand and use Grok @Bot. Computer use, connectors, skills you teach with a screen recording, routines, and going from 1 to 14 bots that run like a company while I only talk to 4 of them. Enjoy! Chapters: 00:00 - Intro 01:22 - What a Grok Bot really is (a new hire with a laptop) 02:23 - Every bot gets its own computer 03:35 - Installation and pricing 05:01 - Creating your first bot (no system prompt to write) 06:29 - Routines: tasks your bot runs on a schedule 07:30 - Computer use and connected apps 11:50 - Skills: a recipe card for your AI 13:07 - Teach a Task: turning a screen recording into a skill 18:38 - One fat bot vs. 14 bots (and why both fail) 19:39 - Run it like a company: you, executives, workers 21:00 - The Notion task board that tracks every bot 21:31 - Designing the org with Claude 23:11 - Why my bots are named after TV characters 23:50 - Leadership in one group chat 26:31 - Chief of staff: inbox sweep across Gmail, Slack and X 27:10 - Head of content: post analysis and competitor watch 29:44 - Design team: on-brand illustrations and Typefully drafts 32:46 - Engineering team: GitHub issue to PR with one label 35:17 - Claude Code on your bot's computer 37:05 - Recap 38:12 - Outro Cheers! :)

Harness Engineering 2.0 is here! After protocols for tools, agent communication, and agent interfaces, harnesses now have one too. Developers just built UHP (Unified Harness Protocol), a standardized interface for running agent tasks across any agent harness. HarnessRouter is its open-source server implementation. Here’s how it works: - The product backend sends a task with a `harness_id`. - HarnessRouter starts the selected harness in its own session. - The harness runs its native agent loop and streams back progress. - Files, artifacts, failures, and the final output return through the same interface. Without UHP, integrating every harness needs separate handling for task execution, sessions, streaming, files, recovery, and output formats. With UHP, Codex, Claude Code, Gemini CLI, Hermes, Pi, and other harnesses can plug into the same product backend while keeping their native tools, skills, models, and agent loops. This means developers can: - Switch harnesses without rebuilding the product integration. - Run the same task across different harnesses. - Add new harnesses without maintaining another execution stack. HarnessRouter currently supports 30 harness bases. Here’s the repo: github.com/HarnessRouter/har… (don’t forget to star it ⭐) As per official benchmarks, changing only the harness while keeping the model fixed reduced costs by 25% to 64% across several healthcare compliance tasks. And changing both the harness and model widened the difference to 99.8% between the highest and lowest-cost configurations. Developers can test their own workloads across different configurations using Harness Arena. Alibaba Cloud's Qoder has also adopted UHP, making it available as an agent backend through the HarnessRouter API. If you want to dive deeper, I already wrote an article that explains why model routing is different from harness routing and what supporting multiple agent runtimes actually involves. It also covers UHP, the local setup, a working API call, and how sessions and files work. Read it below.

The next step after Jev: A decision model like Jev bills and computes the full context on every call, so a 40-message chat checked after each message processes the early messages 40 times. Devs just built Sage2, a streaming decision model that computes each token of a conversation once. The setup is closer to a database cursor than an API call. A session holds the context on the server, and new messages are appended to it, not re-sent. The cached computation over the earlier context is reused. Each new message adds only its own tokens to the cost, and every registered question is re-evaluated against the full history. Each question returns a typed answer with a calibrated probability, so thresholds and routing stay in plain application code. On progressive context, this makes decisions 3x faster, with responses in tens of ms. Sage2 is built by @levantolabs, the team behind the first decision model, launched before Jev. The video below depicts the model in action on a car dealership chat, and I partnered with the team to put this together. More details below.

I gave my Grok @bot team a shared Notion task board. Now every bot logs each task it picks up, so I can see everything happening across my org and steer it from one board. Grok Bot masterclass video dropping this wed. Stay tuned!

Agent framework vs. agent harness, visually explained:

Another banger from DeepSeek. dsh-superagent is an open-source proactive coding agent that coordinates multiple agents to tackle GitHub tasks in parallel, handle review feedback, and report back in your team chat without needing you to babysit every step.

RAG vs. retriever-free RAG, clearly explained! (powered by decision models like Jev) In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector index. When a query arrives, the system embeds it and pulls the top-k chunks with the closest vectors. A reranker then reorders those candidates against the query, and the best few go to the LLM as context. The problem is that the similarity search decides what the LLM can ever see. If the right passage never made the top-k, nothing later in the pipeline can bring it back, not even a strong reranker. Retriever-free RAG, from a new Cornell and ASAPP paper, keeps the documents and the LLM but replaces similarity search with navigation. The documents are organized as a document tree, like a table of contents, and a decision model (Jev 1.13) walks down it. A decision model doesn't generate text. It reads one shared context and returns a probability for every yes/no question about it, in one forward pass per stage. Here's how a query moves through the tree: ↳ Top documents: the model reads the query beside every candidate document, asks "Is this relevant?" for each, and keeps the best few. ↳ Top passages: it opens only those documents and asks the same question for every passage inside them. ↳ Most relevant? (optional): a single multiple-choice question reorders the top 20 passages (top 5 on Natural Questions). Dropped documents are never opened, so their passages are never scored. On Natural Questions (2.68M Wikipedia paragraphs), its top pick contained the answer 65.4% of the time. Embedding search plus an 8B reranker got 48.0%, and GPT-5 reranking got 60.8%. The gain comes from ranking, not coverage. Embedding search's top 100 holds the answer more often than the navigated shortlist (98.4% vs 94.0%), but navigation is better at putting it first. Median retrieval takes under 2 seconds per query, but each method was timed in a different setup, so it isn't a controlled speed comparison. The main limit is that it needs a sensible hierarchy, and a document pruned early is gone for good. It also skips embedding search only at query time, since the Wikipedia tree was built offline using embeddings. The diagram below depicts the complete flow. - RAG retrieves by similarity, then reranks. - Retriever-free RAG navigates a document tree with a decision model. - The LLM answers from the top passages. If you're building a RAG system, this is worth benchmarking against your current setup. Here's the paper: arxiv.org/abs/2610.35868 If you're new to decision models, I've quoted my full breakdown of how Jev works below.

Big moment for Voice AI. A typical voice-agent pipeline has three model stages. Speech-to-text converts the caller’s audio into text. The LLM processes that text and generates a response. Text-to-speech converts the response back into audio. For speech accuracy, information can be lost at two boundaries. - STT must preserve what the caller said on the way in. - TTS must preserve what the LLM generated on the way out. If a caller says “RX-B17,” the transcription model could hear “RX-D17” before the LLM receives it. Even if the LLM generates “RX-B17” correctly, the speech model could still make the final audio sound like “RX-D17.” The LLM’s reasoning accuracy is a separate problem. Both speech layers must preserve the information entering and leaving it. This becomes important with medication names, dosages, appointment times, tracking numbers, and alphanumeric codes. Deepgram built Flux TTS for the speech layer. In an evaluation containing drug names, tracking numbers, dollar amounts, technical strings, and alphanumerics, Flux recorded a 3.4% word error rate. I tested the complete voice loop through a pet-hospital conversation about my dog Biscuit’s prescription refill. The agent had to understand and repeat the medication, dosage, prescription number, and a collection time that changed during the call. The test worked well. The agent retained the changing details, the voice stayed natural throughout, and the exchange felt like a call I would be comfortable having with an actual pet hospital. Build with Flux TTS: fandf.co/4r78t5R For every $1 spent on Flux TTS, Deepgram will match $1 in credits, up to $500. My recording is below, and I worked with the Deepgram team to put this post together.

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: github.com/magnitudedev/magn… (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

The easiest way to find which OCR solution works best for you: I ran DeepSeek-OCR, PaddleOCR-VL and Datalab Surya on the same 150 pages, from handwritten notes and rotated scans to dense tables and a filled tax form. Why? Because one overall score only tells you who's ahead on average, not where a parser breaks on documents like yours. So every page was checked with simple yes/no tests: is this line present, is this cell under the right headings, does A come before B. Here's what I found: - No single parser won on every type of page - DeepSeek-OCR was the best on tables at 87.2 - On handwritten notes, one parser passed 5/6 tests while the other two passed 0 - On a filled 1040 tax form, PaddleOCR-VL returned only the form's title The video below shows every page, what each parser returned, and why each test passed or failed. So the right pick depends on your documents, not on who has the best average. The tool I used is OmniParse Bench, and it's 100% open-source. It has 16,288 tests across 2,937 pages in 92 languages, each tagged by what makes a page hard, like handwriting, rotation, tiny text, forms and tables. Scoring is done by open code with no LLM judge. So you can plug in any OCR solution and see exactly where it holds up and where it breaks. GitHub repo: github.com/datalab-to/omni_p… (don't forget to star 🌟)

I'm blown away by Grok @bot. This is the 14-bot team that handles my inbox, research, code, and design, and I only talk to 4 of them. (bookmark so you don't lose this) My executive staff of Pam, Tyrion, Gilfoyle, and Rachel takes every request, hands it to their teams, and brings the finished work back to me. I'm publishing the full video soon. It will cover: - How to reproduce this exact setup - How Grok Bot works under the hood - How to add skills and integrations - How to run it with Claude Code, Codex, or any harness This is still a work in progress, getting better every day. Also, name your bots after characters you actually know, because talking to Dwight feels far more human than talking to agent-3. Stay tuned!