developer/programmer/ai nerd/🍁
Canada
Joined October 2023
- Tweets6.3K
- Following2.4K
- Followers551
- Likes10.6K
g023 retweeted
Finally wrote a blog post about this! Normal LLM inference isn't deterministic across devices, but this can be fixed.
Floating-point addition isn't associative, (a + b) + c ≠ a + (b + c), and accumulation order changes with the hardware and kernel used, leading to different outputs on the same prompt.
Integer addition is associative, so doing every op in integers (including softmax and norms) gives bit-identical outputs across GPUs and CPUs.
Github repo and post listed in the comments
Turns out you can make LLM inference fully deterministic across devices, with no loss to quality or speed.
This weekend at the @spacexai hackathon I got Qwen3-0.6B to produce identical hashed logits from a 512-token generation across 2 GPUs and 3 CPUs: an A100, an H100, an Apple M5 Max, an AMD EPYC, and an Intel Xeon.
The main reason why inference isn't deterministic is because floating-point addition is not associative. (a + b) + c ≠ a + (b + c), since every add rounds. Accumulation order changes with the hardware used and kernel selected, so the same prompt can give you different outputs even at temperature 0.
Integers ARE associative. So why doesn't integer quantization already fix this? Because, while weights and activations get quantized, the non-linear ops (softmax, normalization, SiLU) dequantize back to float and requantize afterwards. Each of those steps hands you back to floating point rounding.
True integer-only inference does exist, but it's historically been motivated by edge hardware without FPUs, which doesn't make much sense for LLMs. One 2024 paper (I-LLM) did it on LLaMA from that angle and didn't get much attention. Nobody seems to have looked at it from the determinism side.
I wrote my own implementation, simplifying the approach from the paper, so that every operation between the input ids and the int32 logits is exact integer arithmetic. To test it I chain-hashed the logits at every step and ran that across the devices and configurations below. Every integer run gave the same hash: 64430dd985f8. Every fp16 run gave a different one, all diverging on the very first token.
WikiText2 perplexity came out to 20.72 vs 20.95 for fp16 (slightly better than the float baseline), and CUDA-graphed integer decode hits 106 tok/s at batch 1 on an A100, 3.6x the fp16 eager baseline.
Github repo is listed in the comments. Plan to do a writeup over this eventually!
g023 retweeted
95–99% of the gradient norm is lost at the LM head during backprop.
Authors show the softmax bottleneck (D≪V) hurts optimization, not just expressivity: raising head rank 32→4096 made 2B-model pretraining converge 16× faster.
arxiv.org/abs/2603.10145
g023 retweeted
Imagine training on Tencent's new HY4 model in Hugging Face Transformers and wondering why your fine-tuning loss is completely broken at 4.05.
An NVIDIA engineer dug into the weights and found a classic silent bug:
Tencent trained HY4's MLA attention and DSA sparse indexer using interleaved RoPE (pairing adjacent dimensions `[0,1], [2,3]`), which is what vLLM and llama.cpp use.
But Hugging Face Transformers accidentally implemented split-half RoPE (pairing the first half of the vector with the second half).
Switching `rotate_half` to `rotate_interleaved` with the exact same checkpoint weights immediately dropped loss across 235,000 tokens from 4.054 down to 1.268.
github.com/huggingface/trans…
g023 retweeted
I updated my quantization compiler to use e-means as a replacement for k-means in codebook generation and got Qwen3.5-4B from 8.41 GB to 1.23 GB while retaining 91.9% of original performance which I think is approximately SOTA, at least comparable to bonsai 2 but I think we could go lower if someone wanted to donate some GPU time since this is just a hobby for me.
at this size, it's small enough that memory bandwidth isn't the bottleneck to decode speed so I'm interested in seeing how fast we can get larger models on consumer hardware. link below.
g023 retweeted
NVIDIA engineers attacked MoE load imbalance by copying hot experts to other GPUs through the NVLink switch itself, so each byte leaves the owning GPU once no matter how many replicas get made.
My take: Expert-parallel layers run at the pace of the busiest GPU. Their scheduler removed 98 to 99% of the excess load in trace replay. On GB300 with eight-way expert parallelism, prefill throughput for DeepSeek-V4-Pro rose 17.4% on GSM8K and 7.2% on agentic prompts, more than perfectly balanced routing gave (13.5% and 6.0%). The trick is hiding each 37.16 MB expert copy behind the shared expert's compute.
In full disaggregated serving on a GB300 NVL72 rack the gain drops to 1.4 to 1.9% output throughput and 3.1 to 4.3% better P90 latency. That drop from kernel to end to end is worth noticing. Prefill only, eager mode, one NVLink domain, and nothing here for a single desktop GB300, but it shows where NVIDIA thinks MoE serving time is being lost.
arxiv.org/abs/2611.09008
g023 retweeted
One thing that's surprising about this game: EVERYTHING IS CODE.
All textures and models are functions. There are no images or 3D models. Claude didn't draw, it wrote shaders. Everything is Math. That sounds wild - the source is in the HTML, so you can inspect it yourself.
I wanted to test Fable 5.2's quality with one-shot prompts without human feedback.
Everything in this game is from scratch: models, sounds, textures. ~$550 at API pricing, ~2.7M output and 445M input tokens, heavily cached. Took 8 hours!
Play it here: claude.ai/artifact/BGY1SztrA…
g023 retweeted
It's a new age of game development! This is the Unreal Engine 5.9 preview, consisting of huge improvements to AI acceleration ported from unfinished Unreal Engine 6, plus entirely new support for play inside web browsers.
Ok so. #UnrealEngine 5.9 is pretty sick. You can build a web browser version of your game, so I built EasyWaterscape's playable demo... and it runs at a rock solid 60fps, @ high settings, in chrome, at 4586x2372.
What the heck?
#gamedev #DellTech #DellProPrecision #NVIDIA
g023 retweeted
PROMPT
Build a single self-contained HTML file (three.js r160 from cdn.jsdelivr via importmap) that renders a detailed voxel diorama: "Liberty Island & New York, autumn 1886 — dedication day of the Statue of Liberty", as a floating island in a warm hazy sky.
STYLE
- Chunky voxels with slightly beveled (chamfered) edges, vertex colors with per-voxel color jitter and baked ambient occlusion.
- Golden-hour light: low warm sun (ACES tone mapping, soft PCF shadows), sky-dome gradient with a sun glow, warm haze fog, subtle bloom.
- Floating island: grassy rim, layered rock strata on the sides, a jagged rocky underside with hanging stalactites. Small floating islets with autumn trees drift around it, plus a few voxel clouds.
LAYOUT
- Center: Bedloe's Island with star-shaped Fort Wood (11 points, granite + red brick walls), autumn trees, a seawall, small piers.
- On it: the granite pedestal with its loggia openings, and THE STATUE (must be instantly recognizable): green patina, flowing robe with vertical folds and a diagonal sash, raised right arm holding a torch with a balcony and a glowing gold flame, a tablet in the left arm with "JULY IV MDCCLXXVI", a crown with 7 rays and windows, a broad face.
- Around it: a harbor ring of animated water with waves, a paddle-wheel ferry, a tugboat with smoke, and a three-masted sailing ship.
- Behind it: a crescent of 1886 Manhattan on a street grid. Brownstones, red-brick tenements with black iron fire escapes, cast-iron store buildings with striped awnings, waterfront warehouses, wooden water tanks on the rooftops, chimneys with smoke, gas lamps with warm glow, a few lit windows, a gothic Trinity-style church with a tall spire, a white New England steeple church, a Central-Park-style green with a pond and a bandstand, US flags on rooftops.
- A river channel cuts through the city to the island edge, with the Brooklyn Bridge over it: granite towers with twin gothic arches, 4 main cables, vertical suspenders and the diagonal stay web, carriages and pedestrians crossing.
- An elevated railway (the El) on iron columns along one avenue, with a small steam engine pulling 3 cars and puffing smoke.
- Waterfalls pour off the island rim where the water reaches the edge, with mist at the bottom.
LIFE (all deterministic, driven by time t)
- Hundreds of tiny voxel people: pedestrians on sidewalks, crowds on the waterfront and a dedication grandstand with red-white-blue bunting, some waving and some holding parasols.
- Horse-drawn cabs on the streets, waving flags, seagulls circling the statue, smoke and steam particles.
CAMERA & OUTPUT
- A 10-second slow orbit of about 100° around the whole island with a slight zoom-in, the island always fully in frame (like a turntable showcase shot).
- Expose window.renderAt(t) for frame-by-frame capture at 30 or 60 fps, canvas sized from URL params ?w=&h= (default 1920x1080).
- Performance: merge static voxels into chunked meshes, cull hidden faces, use instancing for people and particles, and render the static shadow map once.
g023 retweeted
I wanted to test Fable 5.2's quality with one-shot prompts without human feedback.
Everything in this game is from scratch: models, sounds, textures. ~$550 at API pricing, ~2.7M output and 445M input tokens, heavily cached. Took 8 hours!
Play it here: claude.ai/artifact/BGY1SztrA…
g023 retweeted
this is kind of insane... i didn't realize there was this much efficiency gains to be had just from making the kv-cache block boundaries aware of agent turns so you dont throw away mid block tokens🤯
we are soo early...
g023 retweeted
Every time we have a new model, I run the same simple prompt on all my projects:
"Read this entire project and find everything that’s broken, inconsistent, inefficient, or no longer needed. Fix what you’re sure about and list the rest for me to review."
highly recommend trying
g023 retweeted
Zeroing an LLM's weights can leave its computation unchanged!
Unstructured pruning sets individual weights to zero, but keeps the matrix dimensions. A dense matrix multiply still processes those zeros. To save memory and work, we need to pack the surviving values with their locations and use a sparse kernel, a routine that can skip the zero terms.
g023 retweeted
A dude on Reddit tried to delete his 23andMe account.
Then learned the company would still keep his genetic data.
Best comment:
"Your DNA data is the real value, not the fee you paid."
That is a terrifyingly good business model.
Your customer can cancel the subscription.
They cannot get new DNA.
g023 retweeted
Paste this into Claude Code
when your project gets messy:
"Map this codebase for me.
Show the folder structure,
what each main file does,
how data flows from UI to database,
and the 5 riskiest files to touch.
Write it all to ARCHITECTURE.md."
You'll understand your own
vibe-coded app for the first time.
g023 retweeted
Yes, plenty more in the same family of latent defects:
5. Idempotency gaps: high. Retries or duplicates produce side effects that should have been no-ops.
6. Resource/lifetime mismatches: high. Acquired handles, locks, or connections never released on every path.
7. Hidden shared mutable state: critical. Globals, singletons, or caches mutated from multiple call sites without coordination.
8. Protocol/contract drift: high. Callers and callees diverge on assumed sequences or error semantics that types never capture.
Those plus the earlier four cover most of the “works until it doesn’t” class. An agent skill that systematically enumerates and ranks them would be useful.
g023 retweeted
Yes. Related concepts that surface the same class of bugs:
1. Implicit invariants
"Find all implicit invariants in this codebase: properties that must always hold but are only assumed via scattered checks or comments. List violations reachable via normal paths, retries, or concurrency."
2. Race conditions / shared mutable state
"Find all potential race conditions: shared mutable state accessed without synchronization, check-then-act sequences, and async interleavings that can produce inconsistent state."
3. Temporal coupling
"Find temporal couplings: sequences of calls or state changes that must happen in a fixed order but are not enforced by types or explicit machines. Identify what happens if the order is violated."
4. Incomplete failure paths
"Find incomplete error and cancellation paths: places where success is handled but retries, partial failures, or mid-flow aborts leave the system in invalid states."
Run them read-only first, rank by risk, then make the implicit explicit.
g023 retweeted
在只有 64GB 内存、0 张显卡的普通台式机上，硬跑 284.33B 参数的 DeepSeek-V4-Flash（还开满 256K 上下文），以前唯一的结局就是触发 Linux OOM Killer 当场暴毙，或者掉进 NVMe 硬盘 Swap 慢成幻灯片。
挂在 Hacker News 前排的开源项目 llama-modchip (v2.4.0)，硬是把这个原本占 76.87 GiB 的庞然大物，生生塞进了 57.61 GiB 的峰值内存（VmRSS） 里，纯 CPU 预填充飙到 44.36 tok/s，纯 CPU 解码跑出 11.5–14.7 tok/s；如果换跑 30B 的 MoE（Nemotron-3.5-30B-A3B），在 16 核 Zen 5 上纯 CPU 能跑出 60.4 tok/s，Threadripper 上直接干到 111.6 tok/s。
顺着它的 ARCHITECTURE.md 扒了一下，作者为了突破 64GB 内存墙，在底层干了三件极其暴力的系统工程活：
1. 二阶海森矩阵解析压缩 + 自定义 Fractal-64 格雷码量化，硬削 20.31 GiB
它没有用常规的粗暴低比特截断，而是针对 SwiGLU-H2 做了闭式二阶海森压缩，配合自定义的 GGML_TYPE_Q2_0 = 40（3.75 bpw 的 Fractal-64 格雷码张量重编码）和 L2 正交 MoE 裁剪，把模型体积从 76.87 GiB 压到了磁盘上的 56.56 GiB。这意味着整个 284B 权重加上 256K 上下文的 KV Cache 可以 100% 锁死在 64GB DDR5 内存里，零磁盘交换。
2. 把国际象棋引擎 Stockfish 的主变例搜索（PVS）搬来做 1.64 MB 的 L3 缓存投机采样器
纯 CPU 跑 MoE 最大的瓶颈是每次从 DDR5 内存拉冷门专家权重时，会把 CPU 宝贵的 L3 Cache 冲刷干净。
作者干脆写了一个只有 1.64 MB 的 Stockfish-PVS v2 + RouteCrack 投机草稿器（接受率高达 96.4%），并用 _MM_HINT_NTA（非临时缓存指令，HARPCache） 把冷专家权重的内存流隔离出去——让流式读取 MoE 专家时绝不污染常驻在 L2/L3 缓存里的热草稿器。
3. /dev/shm 共享内存热恢复，把本地跑 Claude Code 的首字等待从 45 秒砍到 13.1 毫秒
平时用本地 CPU 模型接 Claude Code，挂满 40 个工具的系统提示词每次重算都要卡上 45 秒。./modchip claude 直接把 Slot-0 的 KV 状态持久化在 /dev/shm 内存盘里，热恢复只要 13.1 ms。