Backboard CEO | Unicorn Founder | Author Kernelmonkey
Joined November 2024
- Tweets7.4K
- Following596
- Followers2K
- Likes16.5K
Up to 2.2× more output from the same GPUs.
That translates to up to 55% lower GPU cost per generated token, or 120% more serving capacity.
Same hardware. Better economics.
Output throughput is the work that counts with AI inference: it’s the tokens you actually generate with each GPU.
Backboard leads at every concurrency level with Qwen in SemiAnalysis’ AgentX benchmark, with gains ranging from 1.1x to 2.2x.
Responsiveness and capacity don't have to involve tradeoffs. You can get both with the right engine.
ow.ly/YLeE5105cUE
1
61
I think that the idea of active and passive memory curation is where the human mind metaphor for digital intelligence ends. Models can continually fine-tune and upgrade/downgrade memory artifacts but should they permanently delete them?
I'd say no which leads us to a much more novel way to think about memory.
Continual learning sometimes gets discussed as if the goal is to dissolve the context/weights distinction. Let the model just keep accumulating, fine-tuning itself on the fly.
@karpathy points out, though, that this isn't how humans do it.
Our working memory gets wiped regularly. What we actually have is a consolidation process (sleep) that distills stuff into the brain, in a weird and lossy way.
This is very different from how people sometimes talk about continual learning. It's not obvious it's something you can get for free from doing long enough RL loops.
53
Rob Imbeault retweeted
Your inference engine can determine your company’s financial future.
A megawatt infrastructure running open-model inference at 60% utilization can gross about $70 million per year, Dylan Patel (@dylan522p) noted at the Runtime conference.
The engine becomes the margin. Backboard’s 31% throughput edge over SGLang in Qwen on the same hardware is not a rounding error, it's the profit and loss statement.
That's the game we're playing.
ow.ly/zRsH5105cRV
Constant reminders of how early we are and how much unfathomably more we are going to do together.
AWS CEO Matt Garman says AI inference hasn’t reached even 1% of its eventual size.
@alexeheath : "How early is inference, do you think?"
Matt Garman: "I think you’re almost doing a disservice to say that it’s like, 1% of what it’s going to be. I don’t think it’s even 1%. I think there is a massive runway."
3
166
Rob Imbeault retweeted
GPU capacity is expensive and difficult to secure. Learn how inference efficiency can increase AI serving capacity from the hardware you already have.
To make models perform more efficiently, some choose to quantize them meaning they compress the model to a point where it gets a bit dumb (dumbify?).
We took a one of these quant models (FP8) and compared it to its fully intelligent original (BF16) and we achieved faster and more throughput by every measure.
You don't have to sacrifice model integrity.
You don’t have to buy more hardware to see faster inference.
In @SemiAnalysis_' AgentX benchmark, Backboard produces 10,525 tokens per second for each GPU in Qwen. SGLang produces 8,013.
That's 31% more work out of eight H100 GPUs at full bf16 precision against SGLang’s FP8.
This performance can reduce your GPU costs and keep you running the systems you already have.
ow.ly/6K2P5105cQM
🎯
Software is dead.
We’re a bunch of portrait painters and the camera just came out.
Sure, there’s some residual value in zombie subscriptions and old people who don’t bother to switch.
Or corporations who pay a fortune for things that should be free and bizarre main frame software that nobody has an incentive to update.
But RIP 98% of the software industry.
Fable 5.2 can built Microsoft Excel in ~50 hours (source: @MatthewBerman ).
At the current rate of improvement:
2027: 7 hours
2028: 57 min
2029: 8 min
2030: 1 min
2031: 8 seconds
ChatGPT just launched generative interfaces for chat results.
Ask a question and it finds the most elegant way to display it (a map, data, etc).
Within 3-5 years it will generate the perfect interface + software for your task in less time that it takes to microwave a burrito.
People always love to answer this with: “There’s still record shops!”
The difference is, the guy who used to own big record store owned a private jet.
Now he drives a beat up Ford Taurus.
1
150
“Megakernels” lol
Of course that’s your contention.
You’re a first-year inference engineer.
You just read SemiAnalysis’s blog and saw megakernels coming to TPUs and AMD, so naturally you believe that’s the way to get to speed of light.
Next month, when you get your hands on an inference engineering cookbook—Baseten’s, probably—you’ll be singing from the treetops about the 500 nanoseconds of critical-path latency you save on every GEMM pipeline drain.
That’ll last until sometime next year, when you watch a pirated geohotz lecture livestream and become convinced that PDL gives you a strict superset of megakernel capabilities.
“Well, as a matter of fact, I won’t, because PDL introduces—”
“PDL introduces unnecessary stalls because its synchronization mechanism is very coarse.”
Yeah, I read that too. HazyResearch’s blog, “Look Ma, No Bubbles!” Just below Figure 2.
Do you have any thoughts of your own on the subject, or should I just point my agent at the SGLang repo and have it do it for me?
2
261
What’s interesting about this is that it uncovered new potential for more gains using our framework. Another jump coming soon.
We’d rather show you a narrower advantage than make you find it.
In SemiAnalyisis’ AgentX our p90 interactivity in Qwen is close to SGLang’s at four and eight concurrent sessions: we’re 1.1x ahead, not 12.8x.
The lead opens up at one, 12, and 16 sessions. The full tables are public, so you can judge for yourself.
This can still equate to faster AI without spending on new infrastructure.
When AI starts responding faster, the entire experience feels more responsive.
Milliseconds can translate into dollars.
Agents shouldn’t take long to get started. That’s the difference between getting work done and giving up.
In a single session, SGLang's published p90 time to first token with Qwen is 10.6 seconds.
Ours? Just 0.78 seconds.
You can get near-instant agentic AI responses while saving money on hardware.
2
5
455
Outperforming DeepInfer In Decode And Prefill
At 65,536 tokens per step, Backboard delivers 1.37× the prefill throughput of our DeepInfer run.
@SemiAnalysis_ @ArtificialAnlys @lm_zheng
backboard.io/blog/deepseek-v…
At this benchmark point, 2.2× the output on the same GPUs translates to roughly 55% lower GPU cost per output token.
More capacity from hardware you've already paid for. Full BF16 precision vs. FP8
@spheron @friendliai @teortaxesTex @Yuchenj_UW
Full precision shouldn't be the slow option.
FP8 quantization makes a model smaller and faster to run, but at lower precision.
We ran a full BF16 Qwen model, with no quantization and no precision loss, and still posted consistently higher numbers than SGLang.
You can get the most accurate results without paying for more infrastructure.
1
122
Why I care about inference efficiency:
Our best Gemma 4 12B throughput gain over vLLM was 34%.
At the same hourly GPU price, that implies about 25% lower GPU cost per output token, before Backboard fees.
For a provider with demand waiting, that extra capacity matters.
backboard.io/blog/gemma-4-12…
