Building autonomous AI agents. Lawyer by training, engineer by night. Building from Dubai → everywhere..
Dubai, United Arab Emirates
Joined October 2021
- Tweets2K
- Following2.1K
- Followers421
- Likes4.2K
DeepSeek V4.1 Flash 1017, Lossless pack, one Mac Studio M5 Ultra 512 GB: 14% faster output from kernels mlx-serve has but doesn't build on M5.
8k-token prompt, one request:
no drafting 51.6 to 58.9 tok/s
drafting on 68.1 to 77.8
How it was measured, and its limits, below.
ALT Chart: DeepSeek V4.1 Flash 1017, Lossless pack, on one Mac Studio M5 Ultra 512 GB with mlx-serve. Output speed with mlx-serve's fused 4-bit expert kernels off and on, 8k-token prompt, one request. No drafting: 51.6 tok/s off, 58.9 on. Drafting on: 68.1 off, 77.8 on. Headline: 14% faster. Four sessions in one day, no drafting: 1.140, 1.141, 1.144, 1.142. With 8 requests at once there is no gain (1.004). 10 of 12 temperature-0 replies are worded differently; answer quality was not measured.
Limits:
8 requests at once: no gain (1.004 per stream).
With the kernels on, 10 of 12 temperature-0 replies come out worded differently. They read fine, but answer quality wasn't measured.
One Mac, one pack, 8k prompts, 256-token answers.
1
2
88
The numbers, flags and log lines are in a comment on mlx-serve
The fused kernels are @ivanfioravanti's work for @ddalcu's mlx-serve. We only measured them on one box.
2
91
DeepSeek V4.1 Flash 1017 on one Mac Studio M5 Ultra 512 GB, official lossless weights: 65.7 tok/s output with a 128k-token prompt, mlx-serve 26.11.2 defaults.
Median of 5 runs, range 62.2–70.0. Other engines and the method below.
Tuning: 8 lossless mlx-serve settings, each checked for identical temperature-0 output. None beat the defaults beyond noise; MTP off was the one clear loss.
One setting that helps 8 users with fresh 8k prompts: --prefill-decode-share 0, 91.1 vs 85.5 tok/s total.
1
2
509
Caveat on the numbers above: mlx-serve 26.11.2 converts this pack's output head to 8-bit when it loads, so the runs were not bit-exact end to end. With the head left as shipped: 49.1 tok/s against 50.7 (drafting off, 8k prompt), and 4 of 12 test replies came out different.
20
GLM-5.3-Flash on one Mac Studio M5 Ultra 512 GB, 32k-token prompt: TensorFold 1.2.5 with its own 4-bit build writes 89.2 tok/s and reads the prompt at 2,088 tok/s.
On the 6-bit oQ6e build, the other 5 setups write 54.1–60.6 tok/s.
Main test: 8k, 32k and 128k-token prompts, cold cache (a unique tag per run), one request at a time, 256-token answers, thinking on, temperature 0, median of 5 runs.
Compared at 32k: TensorFold refused the ~131k-token prompt at its default cap of 131,051.
99
Speed also depends on what GLM writes. TensorFold, own 4-bit build, thinking off, one request:
prose 90.4
code 92.6
counting 1 to 400: 107.6 tok/s
1
124
Several users at once (fresh 2k prompts, 512-token answers):
TensorFold, own 4-bit build: 74.3 for 1 user, 88.1 total for 8 (18.4 each)
oMLX 0.7.4.dev1: 50.3 for 1 user, 67.2 total for 16 (9.9 each)
At 16, TensorFold and oMLX 0.7.4.dev1 left some requests waiting in a queue.
2
178
Tool calls: GLM on mlx-serve passed 1 of 3 turns with thinking on and 3 of 9 with it off (turns 2 and 3 answered without calling the tool). oMLX and both TensorFold builds passed 3 of 3.
An 8-bit KV cache was slower: 54.1 vs 57.8 tok/s.
1
227
Same engine, two builds: TensorFold read the 32k prompt at 431 tok/s from oQ6e (oMLX's format) and 2,088 from its own 4-bit build, so the first token came after 15.5 s instead of 75.2 s.
4-bit is also fewer bits than 6; answer quality wasn't compared
3
445
Engines: mlx-serve by @ddalcu, oMLX by @jundotkim The several-users test uses @ivanfioravanti's DecodeWall shape.
2
6
501
Output speed depends on what the model writes. mlx-serve defaults, thinking off, one request:
prose 68.8
code 81.2
counting 1 to 400: 116.9 tok/s
MTP and prompt lookup guess the next tokens; predictable text means more right guesses. oMLX: 51.5, 68.9, 90.2.
4
761
Several users at once (fresh 2k prompts, 512-token answers, the DecodeWall shape):
mlx-serve 59.1 tok/s for 1 user, 112.3 total for 16, 8.3 each
oMLX 51.3 for 1, 110.2 for 16 (past 8 it queues)
With 1k prompts and 1,024-token answers, 8 users: mlx-serve 113.4, oMLX 122.1.
2
872
