Thomas Wolf

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: huggingface.co/spaces/huggin…

The future of software In the limit, of course, software won't mean anything anymore. Everything digital will be instantly created at will. But in the interim period we'll wander through an increasingly odd in-between age for software. This transition periodwill be defined by three new precepts. Everflux software - instead of stepping from release to release, months apart, software now evolves continuously. Software creation has broken the sound barrier of our own keystrokes, and every model release pushes it faster. There is no reason to wait weeks or months for the next release when a single company merges over 30,000 code changes a day. Cyber capabilities are also progressing at an increasing pace. Bug and vulnerability fixes need to ship quickly and continuously. And every piece of software sits in a lattice of dependencies, each one now shipping at a blistering pace. Software will have to move accordingly fast just to keep from breaking as everything around it shifts. Translucent sources – A new territory has opened between open source and closed source. Today Claude will happily pick apart any Electron app, and any binary shipped with debug symbols has become a reverse-engineering playground for LLMs. In other words, much so-called closed-source software is now open source in all but name, and keeping code closed makes less and less sense. Even the notion of IP and copyright is breaking in software: Apps don't even need to be accessed to be reproduced. I'm writing this text on a personal text editor I prompted Claude to vibe-code from a few videos of people demoing editor features they liked. A mere description of the interface is increasingly enough for an LLM to rebuild the internals, whether or not they resemble the original. Software IP seems to be dissolving like cotton candy in a raccoon's paws. Bespoke features - Now that all software is open source, nothing prevents us from modifying everything. And personalization is just one prompt and a couple million tokens away, so there is absolutely no reason anymore to wait for someone else to add this feature you want. Everyone becomes their own product manager. Want a search box, an API bridge, a custom button? It's a couple of minutes away, mostly executed in the background of your daily life. Soon we'll want to personalize everything. A major challenge appears when bespoke features crash into our first tenet: everflux software. How do you customize software without cutting yourself off from upstream? I could fork it for good to add my search box, but I'd lose all future benefit from the software creator, be it new security updates or other updates to the rest of the software. How do you personalize software that was never meant to be personalized, without shouldering all of its future maintenance? Merge conflicts won't be the exception anymore; they'll be a constant of everflux software. Agents will rebase our personal features onto each new release overnight, and software will look less like a product and more like a living species, evolving through constant mutation. Image: M.C. Escher, Metamorphosis II, November 1939 - March 1940

same open model, different harness => very different results Joël set out to find which harness actually gets the most out of open models in November 2026

By the way you should follow Augustin on X - I’m trying to get him to post more - it’s one of the most cracked roboticist I’ve ever met

this METR report on AI R&D acceleration at Anthropic is very inward looking imo. it counts the lab's compute, hires and internal data, then credits ~80% of the speed-up to AI what I see from the outside: labs are becoming schelling points for money, redistributing it to an ecosystem they created to accelerate their own development it's a symbiosis. the best data and RL environment companies now know where models will fail before the labs do, and the labs just buy what they propose. that feeds back into every lab at once a lot of the "AI accelerating AI R&D" happens outside the lab

Kimi-K3.1-Pro by Moonshot AI: interesting facts It's huge, and sparse • It has 2.8T total parameters, but only 104B are active per token. • Along with Kimi-K3, it's the largest real model release on the Hub. The next biggest are Qwen3.8-2.4T and DeepSeek-V4-Pro (1.6T). • It has 896 experts and sends each token to 16 of them. (DeepSeek-V4-Pro has 2 times less and Mistral-Large-4 has 7 times less) Clever attention for a 1M-token context • Most of its 93 layers use linear attention (KDA). Every 4th layer uses DeepSeek-style sparse attention, which only looks at the top 2048 tokens. • This is how it handles a 1M-token context (the same as the other frontier open models) • It uses a custom activation called SiTU-GLU and has a 401M-parameter vision encoder (MoonViT-V2), so it takes both text and images. Small on disk for its size • It was trained to run in 4-bit (MXFP4), so the whole repo is only ~1.56 TB. • Stored in regular BF16 it would be about 5.6 TB(!) Benchmarks (from the model card) • It goes back and forth with Claude Opus 5.5 and GPT-6 Astra. It's top of the card's table on BrowseComp (92.1), HLE with tools (70.0) and SWE Marathon (51.7). • It's weaker on pure reasoning without tools and on cyber security. • Moonshot ran most of their own coding evals using the Claude Code harness. License (a modified MIT) • It's free to use, modify and redistribute. • API businesses making over $20M a year need a deal with Moonshot. • Products with over 100M monthly users or over $20M monthly revenue must display "Kimi K3.1-Pro". On the Hub • It went public about 1.5 hours ago and already has 214 likes, about 9× Kimi-K3's first hour. It's #19 on the trending list. • It isn't gated, but loading it in transformers requires "trust_remote_code"