Senior software engineer Automation + applied AI (pharma) Contributor: MLX-Serve, TensorFold, ds4 Local inference · Real hardware · Builders
Joined September 2026
- Tweets1.2K
- Following517
- Followers2.3K
- Likes8.7K
Pinned Tweet
Building automation and applied AI systems for pharma. Contributing to local inference projects (MLX-Serve and TensorFold).
Obsessed with making models run faster and more reliably on real hardware.
Tinkerers and builders — what are you working on? I’d love to connect.
3
25
3,252
Mac vs DGX Spark was always the wrong fight. Run both.
oMLX 0.8.0 hands prefill to the Spark and the Mac keeps decoding. M3 Ultra 512GB, 64K token prompt, Qwen3.8 27B:
Mac alone: 352 tok/s
With a Spark: 1,062 tok/s
With 2x RTX A6000: 1,879 tok/s
Decode barely moves.
5
13
893
Credit to @jundotkim, chart and details here. The GPU doesn't need to hold the whole model, the Mac does. DeepSeek V4.1 Flash prefill goes 1,085 to 2,412 tok/s with a Spark, decode 44.4 to 45.1. xitter.twitterapi.xyz/jundotkim/status/21209…
oMLX can now use your NVIDIA GPU or DGX Spark as a Prefill Accelerator.
github.com/jundot/omlx/relea…
All you need is a Mac that can hold the model. A faster NVIDIA GPU or a DGX Spark takes over the prefill, and your Mac keeps decoding at the same speed.
- The GPU doesn't need to fit the model. It speeds up as many layers as it can hold.
- No RDMA, no Thunderbolt. Just plug in a LAN cable (1GbE works, 2.5GbE recommended).
- One docker run on the NVIDIA machine. No oMLX install or SSH needed there.
64K prompt, M3 Ultra 512GB + GPU over 2.5GbE
With a DGX Spark
- Qwen3.8 27B: 352 -> 1,062 tok/s (+202%)
- Qwen3.8 Flash-Next: 1,370 -> 2,715 tok/s (+98%)
- DeepSeek V4.1 Flash: 1,085 -> 2,412 tok/s (+122%)
- GLM-5.3 Flash: 555 -> 1,076 tok/s (+94%)
With 2x RTX A6000
- Qwen3.8 27B: 352 -> 1,879 tok/s (+434%)
- Qwen3.8 Flash-Next: 1,370 -> 3,928 tok/s (+187%)
- DeepSeek V4.1 Flash: 1,085 -> 1,914 tok/s (+76%)
- GLM-5.3 Flash: 555 -> 978 tok/s (+76%)
The CUDA helper builds on the great CUDA kernels of TensorFold by @ashhart and mlx-serve by @ddalcu, and would not have been possible without them. Thank you both for sharing them in the open.
Also in 0.8.0rc1
- DeepSeek V4.1 with the DSpark drafter: 8 requests decode 73.2 -> 107.9 tok/s (+47%)
- Concurrent Request Priority: new prompts start sooner while other responses stream
- Affine KV decodes up to 58% faster at long context
There's a new version of this post
1
67
Jensen Huang just resigned, and local AI should be paying closer attention than Wall Street.
Under him, Nvidia became one of the biggest open model publishers there is: 650+ open models and 250+ open datasets on Hugging Face. Nemotron ships the weights, the data, and the training recipe.
He called open models "the lifeblood of innovation."
Wall Street is watching the stock. I'm watching whether the next CEO keeps shipping weights.
Numbers are from Nvidia's own open source page, which features his quote. Interim CEO is CFO Colette Kress, and the board is searching inside and outside. opensource.nvidia.com/en-us/
5
263
Parallel requests on a Mac don't make the work finish sooner. They make the wait fairer.
Ollama just merged concurrent generation for MLX. Their own M5 Max numbers, Qwen 3.8 27B, two clients at once:
Second client, first token: 13.84s to 9.90s
First client, first token: 5.20s to 7.40s
Both done: 17.23s to 20.07s
Good for two agents sharing one box. Worse if all you care about is total time.
7
21
1,359
Numbers are from Ollama's own PR (NVFP4, 4K prompt, 256 output, M5 Max). It ships in the v0.40.17-rc0 prerelease, set OLLAMA_NUM_PARALLEL=2. Drafting is paused while requests run together. github.com/ollama/ollama/pul…
1
135
Your Mac was never slow at long chats. Your engine kept re-reading them.
Qwen 3.8 27B on a 64GB M5 Pro Mac mini, TensorFold:
1.2.6: a follow-up re-reads 24K tokens, 76.8s before the first word
1.2.7: reuses ~34K cached tokens, first word in 0.34 to 1.19s
Same box. Same model. Byte identical answers.
A lot of "Mac prefill is slow" complaints are really cache misses.
xitter.twitterapi.xyz/pratikg/status/2120624…
🚨 JUST IN: TensorFold 1.2.7 makes long Qwen3.8 27B chats near instant 👏
I tested locally on a 64GB M5 Pro Mac mini with Qwen3.8 27B:
- Context with earlier reasoning steps dropped (common behaviour in chat apps)
- Previous version (1.2.6): follow-up questions would reprocess 24K tokens and take 76.8 s until the answer started streaming
- New version (1.2.7): follow-up questions reuse ~34K cached tokens and start streaming in 0.34 to 1.19 s (answers are still byte identical!)
- Context including reasoning steps (chat apps that preserve reasoning steps)
- Previous version (1.2.6): later questions took 1.35 to 4.04 s until the answer started streaming
- New version (1.2.7): later questions start streaming in 0.26 s
Nice improvement on TTFT, and decode speed is up a tad as well (same output on the deterministic runs):
- Chat without DFlash2: 17.06 --> 17.39 tok/s
- Chat with DFlash2: 56.22 --> 57.90 tok/s (3.3x faster than without DFlash2)
- Code with DFlash2: 93.79 --> 98.90 tok/s (5.7x faster than without DFlash2)
Incredible work @ashxhart and the rest of the community!
Long context chats on a 27B model on your desk is already super impressive, having it be near instant and owning the whole stack makes it even nicer 🙌
The most underrated chip in your Mac is not the GPU.
LFM3 1.7B on the Apple Neural Engine, LUT6, 1.8GB of weights: 20.5 tok/s at 0.87 W (engine power, not wall).
That works out to about 12 Wh per million tokens. An always-on background agent that never touches your GPU.
xitter.twitterapi.xyz/BuildWithAJ/status/211…
Hi all, so I'm making @liquidai LFM3 1.7b run on the apple neuranl engine.
Mostly so it uses less power, I know the speeds are not comparable.
The power numbers are engine only, not wall power.
LUT6 is giving some decent numbers
1.8 GB weights
20.5 tok/s decode
0.87 W
github.com/AJ-Gonzalez/LoNE
No mac? Raspberry Pi is possible as well!
xitter.twitterapi.xyz/dragosroua/status/2120…
Been able to run a 1.7B model (LFM3-Coder) on a 4GB Raspberry Pi 4, to build an iOS calorie tracker app. The full generation took one hour and 8 minutes at roughly 2t/s, but it worked!
youtube.com/shorts/dAp2MQpQV…
1
6
355
People keep saying Macs can't prefill. That is an active parameter problem, not a Mac problem.
LFM2.5-24B-A2B on a 256GB M5 Ultra, MLX 4-bit:
32K prompt read in 2.6s (12,840 tok/s)
128K prompt to first token in 17.7s
8 requests at once: 639 tok/s total
2B active. For agents that reread huge contexts all day, shop by active params, not total params.
xitter.twitterapi.xyz/niklaslenz_ai/status/2…
LFM2.5-24B-A2B reads a 32K-token prompt in 2.6 s on my Mac Studio M5 Ultra
256 GB Mac Studio M5 Ultra: MLX 4-bit, no draft model, copy drafts on, thinking off
Decode, code: 226.7 tok/s
Decode, prose: 276.4 tok/s
Prefill at 32K: 12,840 tok/s
First token at 32K: 2.6 s
First token at 128K: 17.7 s
8 requests at once: 639.1 tok/s in total
Largest context step tested: 131,072 tokens, it fits (prompt filling 95 %)
Quant: MLX 4-bit, group size 64, token embedding at 6 bits, 4.53 bpw, 13.6 GB, made by Liquid AI
Setup: TensorFold 1.2.5 official release (the recipe's Mac steps build from source), window 131,072 tokens, no draft model
Start: the recipe's Mac start command with --parallel 16 and --port 8888 instead of 8080, plus --keep-warm 86400, --host 127.0.0.1 and --no-update-check
Speed test: decode median of 2 runs, prefill median of 3 (128K step is 130,048 tokens: median of 4), requests at once: median of 2 rounds, 512 tokens, temperature 0.1, top-k 50, seed 42
Repetition penalty 1.05 from Liquid's model card not sent, server default
LFM2.5 is a specialist model for agents on a device, and my tests do not measure that.
The Mac was not idle during the tests: my control session, the desktop and a dictation helper ran on it.
Recipe page:
niklaslenz.com/v2/lfm25-24b-…
Recipe and engine by Ash @ashxhart, model and MLX weights by Liquid AI @liquidai.
2
32
3,028
Your 8GB MacBook Air is not too small for a 26B model.
turbo-fieldfare runs Gemma 4 26B-A4B in about 2GB of RAM. It keeps a 1.35GB shared core and the KV cache resident, and streams only the experts each token needs from the SSD.
8GB M2 Air: 5.1 to 6.3 tok/s
24GB M5 Pro: 31 to 35 tok/s
Custom Swift and Metal, no MLX, no llama.cpp.
RAM is a speed setting now, not a ceiling.
11
27
2,624
Stop reading Mac vs DGX Spark benchmarks as hardware fights. They are engine fights.
Tom's Hardware, same 256GB M5 Ultra Mac Studio, GLM 5.3 Flash:
llama.cpp: 46.3 tok/s decode
TensorFold with an MLX build: 94.0 tok/s, still 79.4 at 524K context
Same box, double the speed. And on Qwen 3.8 27B with llama.cpp on both, the Mac finished the agent test in just under half the time of one GB10, on 16% less energy.
Pick the engine before you pick the box.
Sebastian Christiansen retweeted
Introducing e-means, an algorithm beating k-means in 10x fewer epochs.
With order of magnitude wall-clock speedups, e-means is released in FAISS and needs no hyperparameter tuning beyond k. It is robust to various scales and datasets. Still skeptical ? Just try e-means [1/10]🧵
The M5 Ultra was never a speed play. It is a memory play.
DeepSeek V4.1 Flash with official lossless weights on one 512GB Mac Studio: 65.7 tok/s output with a 128k prompt, mlx-serve defaults, median of 5 runs.
No cluster. No requant. One box on a desk.
xitter.twitterapi.xyz/ApacheAE/status/212007…
Drafter speedups are not a 256 token demo trick.
M5 Max, TensorFold, Qwen3.8 27B: 129 tok/s across 162K tokens of long output, against 24 tok/s with drafting off. 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
5x holds when the answers get long.
xitter.twitterapi.xyz/defilan/status/2119995…
Last-minute benchmarks on my M5 Max before my talk tomorrow: @ashxhart's TensorFold ran Qwen3.8-27B at 129 tok/s across 162K tokens of long output, 5x the same model with drafting off (24 tok/s). Not a 256-token demo: 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
Decode tok/s by workload, from TensorFold's own per-request stats:
Agent rewriting a Go file: 231 (up to 359)
K8s YAML: 160
Go package with tests: 144
Code review: 104
Prose essay: 69
That's 3x on prose, 6-7x on code and YAML, and 14x when the output copies its input. Sampling barely moved it: 129 / 134 / 125 overall at temp 0, 0.7 and 1.0. Greedy drafted output was byte-identical to undrafted, so the speedup costs no quality.
I also checked the outputs. 36/36 K8s objects pass strict schema validation, and the Go file edit builds and passes LLMKube's tests. Weak spots: unit tests written in one shot, and invented numbers in the essay. Check those with any model. In opencode it fixed its own jq errors and shipped a working kubectl plugin in under 2 minutes.
Setup: Qwen3.8-27B MLX 4-bit, thinking off, with z-lab's DFlash2 drafter, single stream, 128 GB M5 Max, served by LLMKube's Metal agent. TensorFold 0.3.4.1, because my demo has been pinned since September; 1.2.5 is current. Short templated YAML still hits 227 tok/s; long answers settle lower. Happy to share the script.