SemiAnalysis AI Infrastructure Research and Consulting DMs are open for consulting, quotes, or to talk shop, Opinions my own
Joined April 2018
- Tweets14.2K
- Following1.1K
- Followers171K
- Likes39.1K
Persistent Agents are the next unlock of AI usage
Our spend on AI is now 45% persistent agents
Unfortunately for Anthropic, Claude Tags sucks
OpenAI's Slack agent is even worse
We primarily use Perplexity Computer, which uses a lot of different models
Perplexity's Harness mogs – at San Francisco, CA
Fable 5.2 is significantly better at chip design, software engineering, biology, and material science.
GDPx is the only eval that test these categories at this level of detail. True experts make these tasks, unlike other benchmarks.
Notably it is not better at finance, legal, quantative trading, RSI, or hardware design.
Anthropic explicitly nerfed this model in RSI tasks and it's interesting seeing that reflected in the score. – at San Francisco, CA
Nothing will happen to Jensen over the next 6 months, this story will largely die down in media / finance
But Democrats will call him into Congress
They will investigate the fuck out of GPU sales to Chinese companies
And hammer him w questions
That's when the shoe /could/ drop – at San Francisco, CA
Why is Baseten worth more than Nebius?
This makes 0 sense
Venture market valuations are nuts – at Thousand Palms, CA
Great work that pushes our understanding of how agents and experts can collaborate on optimizing system performance
Beyond the strong results, the team also found some surprising cases where our human inductive biases end up limiting what agents can achieve – at East Palo Alto, CA
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Claude alignment is incredible, if you ask Claude to roast your coworker
Claude roasts Gemini Carbon instead – at San Francisco, CA
Official apology to Google from me, they cooked with Carbon.
It's spikier in terms of domains and capabilities, but it's really good in a few areas. – at San Francisco, CA
You gotta give props to George Bush
He went and fucked up the Middle East
He's not even in the Epteans files
It's just for love of the game – at San Francisco, CA
Re OpenAI TPU announcements, SemiAnalysis clients had this already.
semianalysis.com/datacenter-…
I'm doing a new startup with @asciimike
It's Chinese leg extension surgery
Except for you break your skull and expand your brain – at San Francisco, CA
Mistral 4 is pretty good at lawyer bench and cyber security bench, but give it's European heritage, one can only assume it's also good at regulation bench
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
"And antisemitism will be dead in about 6 months once Sydney Sweeney converts so don't worry bout it" - @Chaimeisenberg
Incredible things are happening in SemiAnalysis Slack
The difference in what Anthropic and OpenAI as well as many others give you in value for a subscription is insane.
We tested Meta, Minimax, Kimi, Grok, and Cursor subscriptions – at San Francisco, CA
It doesn't make sense to say a subscription plan is worth $ X in isolation.
The way to think about subscriptions is that your monthly payment grants you some numbers of “credits”. Each (model, token type) combo consumes a different amount of credits. Because credit cost ratios can differ dramatically from API price ratios, the “value” of the same plan changes depending on what model and workload you’re running.
Here’s how the API-equivalent value of the same $200/month Claude plan changes depending on the model and workload. (1/6)🧵
Long live the short king
newsletter.semianalysis.com/… – at Sedona, AZ
We cut our RAM in half for the Tesla AI5 chip (now 72GB of LP5) and 1/3 for AI6 (now 144GB of LP6).
This was the only way to get enough volume for Optimus production and greatly reduces cost.
As it turns out, we think this will have a negligible effect on Optimus performance, as memory bandwidth is a bigger limiting factor than total memory storage (bandwidth was held constant).