ML/AI research engineer. Ex stats professor. Author of "Build a Large Language Model From Scratch" (https://xitter.twitterapi.xyz/t.co/O8LAAMRzzW) & reasoning (https://xitter.twitterapi.xyz/t.co/5TueQKx2Fk)
United States
Joined October 2012
- Tweets20.4K
- Following1.2K
- Followers538K
- Likes26.2K
Pinned Tweet
I put together a mega write-up on GPT-6 Astra & looped transformers.
How looped transformers / recurrent depth works, cost-tradeoffs, whether it hides reasoning traces, with lots of figures and a tour of recent looped transformer research.
Hillclimbing: Make the model better (best intentions).
Benchmarking: Measure how much better (best effort).
Benchmaxxing: Make the numbers look better (best optics).
Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.
In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.
In the first half of 2026, the trend towards "some full attention" hybrids continued.
And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.
The plot legend here means,
- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.
The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.
In particular, there are two distinct efficiency gains here:
1) Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)
2) Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).
What could possibly go wrong?
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Catching up on the 10 (!) open LLM architecture releases from the past month. Quite a lot happened in October.
I guess we can call it "Open October"!
1. Kolibri-1 (78B): A notable model from Germany. Features classic Grouped Query Attention (GQA) and Sliding Window Attention (SWA) attention module.
2. Ling 3.1 Flash (560B): Uses Kimi Delta Attention and the eighth uses gated Multi-head Latent Attention (MLA) inspired by DeepSeek.
3. Muse Spark Open (816B): Besides Musse Glimmer, the first major open weight release from Meta (since Llama). Uses classic GQA + SWA but adds a LatentMoE.
4. Nemotron 3.5 Super VL (120B): Combines Mamba-2 layers with just eight attention layers. Also LatentMoE
5. Dory1.0 (4B): A looped transformer design.
6. Reflection Beam (501B): GQA+SWA architecture.
7. Recursive Flint (40B): Note that "Recursive" is the company name; the architecture is a fairly standard MoE with GQA and SWA.
8. MiniMax M3.1-Flash (428B): Uses their own MiniMax Sparse Attention module.
9. LFM3 (1.7B): Hybrid architecture that mixes short convolutions and attention.
10. Mistral Large 4 (1T) combines MLA with sparse attention.
Will probably cover/discuss things in more depth in the coming weeks!
You can find more details and higher res figures in the full gallery: sebastianraschka.com/llm-arc…
The complete reasoning-from-scratch video series is now up (and linked) here: magazine.sebastianraschka.co…
(Taking a short break but will be back with LLM model and research updates in November!)
Reasoning From Scratch round 8, the finale!
This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs.
(Planning to take a brief break, but I will be back!)
00:00 Introduction and series recap
01:07 Why distill a reasoning model?
02:51 Soft vs. hard distillation
11:06 On-policy vs. off-policy distillation
18:09 Preparing data for reasoning distillation
22:35 Loading the DeepSeek R1 MATH dataset
28:04 Inspecting reasoning traces and final answers
31:27 Tokenization and chat templates
35:58 Pre-tokenizing the training dataset
39:01 Sequence lengths and the 2,048-token limit
42:38 Creating training and validation splits
45:32 Loading the Qwen3-0.6B base model
46:24 Computing answer-token log probabilities
54:10 Cross-entropy loss for distillation
57:25 Training and validation loss helpers
1:02:03 Implementing the distillation training loop
1:07:26 Running the standalone training script
1:09:41 Plotting training and validation losses
1:11:31 Evaluating checkpoints on MATH-500
1:14:09 Comparing teachers and distillation results
1:16:34 Future directions for reasoning models
1:19:41 Bonus exercises and next steps
And one more time, the YouTube link: youtube.com/watch?v=msp69FR5…
Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2.
Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks.
00:00 Introduction and recap
01:52 Interpreting basic GRPO training metrics
06:34 Planned improvements to GRPO
08:58 Running longer training jobs with Python scripts
13:39 Running the baseline GRPO training script
17:29 Loading and plotting training logs
19:29 Diagnosing unstable training
23:55 Evaluating checkpoints on MATH-500
26:26 Downloading existing checkpoints
30:09 Tracking advantage statistics
34:53 Understanding entropy
40:32 Computing entropy in PyTorch
44:17 Interpreting entropy values
48:58 Adding entropy tracking to GRPO
53:36 Analyzing advantage and entropy metrics
56:18 Stabilizing GRPO with clipped policy ratios
1:03:27 Implementing the clipped policy loss
1:09:39 Analyzing clipped policy training results
1:11:25 KL divergence and reward hacking
1:15:12 Adding a KL loss term
1:20:34 Limitations of the simplified KL loss
1:23:04 Format rewards and think tags
1:25:47 Adding special tokens to the tokenizer
1:30:29 Implementing the format reward
1:35:56 Analyzing format reward training
1:38:25 Rewarding format only for correct answers
1:40:48 Further GRPO improvements from research
1:45:43 Next steps and distillation
Spontaneous morning chat with Juergen Schmidhuber and Jake Newfield philosophizing about AI.
Sry my mic somehow disconnected but Juergen had the more interesting points anyways!
Our CEO @Jake_Newfield brought together @SchmidhuberAI ("father of AI" and pioneer of deep learning) and @rasbt (author of Build a Large Language Model), to discuss humanity’s future with AI: youtube.com/watch?v=fcOTx06q…
Reasoning from scratch, round number 6!
An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO).
00:00 Introduction
01:54 What makes a reasoning model different?
04:25 Reasoning traces and model capability
08:29 Accuracy and format rewards
11:34 Aha moments and DeepSeek-R1 training
14:41 Reasoning effort and answer length
18:38 RLHF and RLVR
23:04 GRPO vs. PPO
26:40 GRPO explained with a cooking analogy
31:43 The KL term and simplified GRPO
35:04 Loading the pretrained model
36:07 Loading the MATH training data
39:26 Sampling model responses
46:30 Computing verifiable rewards
49:55 Computing advantages
51:54 Token and sequence log probabilities
55:29 Implementing sequence log probabilities
57:37 Fixing the inference-mode error
1:02:24 Computing the GRPO loss
1:04:37 Putting the GRPO step together
1:09:19 The GRPO training loop
1:12:57 Training settings, logging, and checkpoints
1:17:24 Running training and inspecting outputs
1:19:28 Loading and evaluating checkpoints
1:22:33 MATH-500 results and training stability
1:24:05 Memory requirements and next steps
Took a while, but here it is, a mega write-up on language models for text classification and, of course, Jev.
It's basically a visual guide to RNNs, CNNs, transformers, and calibration, with hands-on experiments on accuracy and efficiency.
And here's the link to the full article: magazine.sebastianraschka.co…
A little Ember-1 tl;dr.
Seems like a great model!
(Was recently asked on a podcast, given $ xx million, what's the best way to develop a frontier LLM today? My recommendation was: start with an existing one and spend that budget on post-training. Great example here.)
Also kudos to fireworks AI for admitting that they built on top of Kimi (I remember the awkward Mistral 3 release)
xitter.twitterapi.xyz/rasbt/status/199955783…
Hold on a sec, Mistral 3 Large uses the DeepSeek V3 architecture, including MLA?
Just went through the config files; the only difference I could see is that Mistral 3 Large used 2x fewer experts but made each expert 2x large.
8
30
12,900
Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and self-refinement.
00:00 Introduction and inference-time scaling recap
05:02 Loading the pretrained LLM
08:00 Comparing and scoring model answers
10:18 Building a rule-based scorer
17:53 Token probabilities and sequence likelihood
26:47 Computing token probabilities in PyTorch
30:12 Token indexing and shifted targets
37:27 Log probabilities and numerical stability
45:57 Scoring answers with average log probabilities
56:24 How self-refinement works
59:07 Generating critiques and revised answers
1:01:00 Implementing the self-refinement loop
1:05:57 MATH-500 evaluation results
1:07:35 Takeaways and next steps
The main appeal of open-source agent harnesses isn't that they are free, but that we can inspect what they are doing on our computers.
MiMo-V2.6 is "simply" the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average).
With "simple," I mean a classic Grouped Query Attention (GQA) with Sliding Window Attention (SWA) at a tiny 128-token window size.
So, that underlines one of the points I've been trying to make in recent months: most of the progress still comes from the data and post-training recipe improvements. Fancy attention variants are just mostly efficiency tweaks.
What are some of the training data improvements and recipe improvements? The MiMo team shared a pretty detailed technical report. Lots to carefully digest there, but in short, there are a few things that stood out:
1. An increase in agent tasks; also training across different harnesses (the average DeepSWE pass@1 accuracy on held-out harnesses improved from approximately 50% -> 66%).
2. Better reward signals: they replaced a simple correctness verifier with an agentic grader that looks at the execution traces as well.
3. Large RL batches (1,568 prompts × 16 rollouts = 25,088 trajectories) and 2.7–3.7 billion training tokens per update (unclear, though, what the predecessor used).
MiMo-V2.6-Pro debuts as the top open weights model on the Artificial Analysis Intelligence Index (46). At $0.13 per Intelligence Index task, it lands on the Intelligence vs. Cost per Task Pareto frontier
@Xiaomi has just released MiMo-V2.6-Pro, an open weights model with major advances in intelligence over its predecessor, MiMo-V2.5-Pro (Intelligence Index: 26). Despite the improvement, it retains the same attractive pricing at $0.435 per 1M input tokens (with a 99% cache-hit discount) and $0.87 per 1M output tokens. This makes MiMo-V2.6-Pro one of the most cost-efficient models to deploy.
MiMo-V2.6-Pro is an MoE model with 1.02T total parameters and 42B active parameters. Stay tuned for additional analysis of the model.
Check out MiMo-V2.6-Pro full benchmarking breakdown here: artificialanalysis.ai
It’s easy to dismiss Jev it as “just a classifier”.
But people (me included) who have been training encoder-style models for classification for many years know they were usually special-purpose and limited in some way.
The breakthrough of Jev is that it generalizes well.
And I’d say the secret sauce is probably more in the data than in the training algorithm. (Plus a nice API design on top of it.)
poor guy claim to have built Jev a year ago but no one cared, and now Jev stole all the thunder
many people are saying “you gotta tell your story” or “marketing is important”, and they just completely missed what actually made the difference here
i just looked into this laya model laya.convaiinnovations.com/ and:
- it only supports 512-1k context… a lot of use cases won’t fit at all
- evaluating the model directly shows its accuracy is as good as a coin flip. in order to get good results, you need to first fine tune it
i’m sorry, but that’s not Jev
there’s a massive gap between an interesting research and a useful product
you can “tell your story” all you like, but you can’t blame Jev for stealing your thunder when Jev did all the work to make a well-packaged solution anyone can just grab and go
Jev is not completely new from an academic sense, just like how ChatGPT was not the first LLM
don’t underestimate the effort and value in putting together something that’s actually good enough for adoption - it makes all the difference
Inference scaling part 1.
Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (improving answer accuracy by>2x)
00:00 Introduction and recap
00:31 Training-time and inference-time scaling
07:52 What we'll implement
11:47 Notebook setup and model loading
17:43 Building a flexible text generation function
24:40 Chain-of-thought prompting
28:26 Sampling and output diversity
33:43 Next-token logits and greedy decoding
38:20 Temperature scaling step by step
42:46 Softmax and token probabilities
47:42 Multinomial sampling
54:51 Adding temperature sampling to text generation
59:31 Top-p filtering step by step
1:10:23 Adding top-p filtering to text generation
1:13:43 Sampling and LLM watermarking
1:16:01 Self-consistency and majority voting
1:20:36 Implementing self-consistency
1:29:02 MATH-500 results
1:35:01 Accuracy and compute tradeoffs
1:36:50 Next steps and self-refinement
Interesting new insights into the old AdamW vs Muon debate: Muon seems to do better because it reduces memorization.
𝗔𝗱𝗮𝗺𝗪 𝘃𝘀. 𝗠𝘂𝗼𝗻: α, 𝗢𝘃𝗲𝗿𝗳𝗶𝘁𝘁𝗶𝗻𝗴, 𝗮𝗻𝗱 𝗠𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻. A quick update on our WeightWatcher experiments comparing AdamW with the newer Muon optimizer.
We trained a single-head nanoGPT model with both optimizers across five seeds. We then used WeightWatcher to study the weight-matrix spectra and several forms of memorization.
The top plot shows the average raw α.
AdamW produces relatively stable α values. Muon is much noisier. Its ESDs vary substantially across layers, which makes α harder to estimate reliably.
But the average hides something important:
👉 Individual layers do fall below α = 2.
Our spectral/RG theory predicts that this regime can be associated with pathological, example-specific learning. So we tested that prediction directly.
We inserted random sequences into the training set and measured whether the model remembered them using exact extraction, token recall, teacher-forced recall, and canary exposure.
The result was striking.
• AdamW memorized the planted sequences strongly. Earlier in training it reached ~96% exact recall of 32-token canaries.
• Muon strongly suppressed this kind of memorization. At 10,000 steps, teacher-forced recall was ~62% for AdamW versus <1% for Muon.
There is also a surprise: some Muon layers have α < 2 as well. So α < 2 by itself is not sufficient to explain memorization.
This is why we recommend examining individual WeightWatcher layers and their ESDs—not simply average α.
Muon appears to suppress memorization while also producing unusual ESDs that make conventional WeightWatcher analysis more difficult.
We are now running Muon much longer to understand how these spectra evolve.
If you are using WeightWatcher and trying to analyze models trained with Muon, feel free to reach out for help. #talkToChuck
𝗔𝗱𝗮𝗺𝗪 𝘃𝘀. 𝗠𝘂𝗼𝗻: α, 𝗢𝘃𝗲𝗿𝗳𝗶𝘁𝘁𝗶𝗻𝗴, 𝗮𝗻𝗱 𝗠𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻. A quick update on our WeightWatcher experiments comparing AdamW with the newer Muon optimizer.
We trained a single-head nanoGPT model with both optimizers across five seeds. We then used WeightWatcher to study the weight-matrix spectra and several forms of memorization.
The top plot shows the average raw α.
AdamW produces relatively stable α values. Muon is much noisier. Its ESDs vary substantially across layers, which makes α harder to estimate reliably.
But the average hides something important:
👉 Individual layers do fall below α = 2.
Our spectral/RG theory predicts that this regime can be associated with pathological, example-specific learning. So we tested that prediction directly.
We inserted random sequences into the training set and measured whether the model remembered them using exact extraction, token recall, teacher-forced recall, and canary exposure.
The result was striking.
• AdamW memorized the planted sequences strongly. Earlier in training it reached ~96% exact recall of 32-token canaries.
• Muon strongly suppressed this kind of memorization. At 10,000 steps, teacher-forced recall was ~62% for AdamW versus <1% for Muon.
There is also a surprise: some Muon layers have α < 2 as well. So α < 2 by itself is not sufficient to explain memorization.
This is why we recommend examining individual WeightWatcher layers and their ESDs—not simply average α.
Muon appears to suppress memorization while also producing unusual ESDs that make conventional WeightWatcher analysis more difficult.
We are now running Muon much longer to understand how these spectra evolve.
If you are using WeightWatcher and trying to analyze models trained with Muon, feel free to reach out for help. #talkToChuck
Some food for thought when designing benchmarks...
So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max.
The task here was to recreate the image in the center using the Paint UI.
Super interesting how the two different LLMs+Harnesses approached this totally differently by default.
I.e., Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.)
So, the Qwen-generated image would surely score higher in the sense that it's closer to the original.
But I wouldn’t conclude from this example that one LLM generalizes better than the other on other tasks. Also, I wouldn't say Qwen has better compute-use capabilities or better visual understanding than Astra.
But it highlights an interesting point about how slippery benchmarks are when they only compare final results.
Since I had to discuss the "pacing" with a lot of people this weekend, here are my two cents: I don't think pacing literally means that these companies will be "slowing down" training and development in any way.
"Pacing" here means adding a framework for more checks.
We have seen some of that "pacing" already in recent months, when Mythos wasn't released as-is but instead a delayed, nerfed Fable variant was released.
Or when Astra wasn't released right away / there is an existing Astra model that hasn't been released yet.
These Mythos/Fable and Astra pacing decisions were ad hoc. If you are a company, you have to weigh the pros and cons of a delayed release in terms of keeping up with the competition, making money, pleasing shareholders, mitigating risks and harms, and so on.
I
f there is a formal framework that everyone has to abide by, that essentially relieves some of the pressure on a company to rush out its model just to take the top spot on the leaderboard, since it knows that the competition "has to" play by the same rules.
Based on the discussions today, I think "pacing" primarily means just that, rather than a halt in training the models.
TL;DR: Pacing != pacing development.