David Hendrickson

Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about. 💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick) My llama.cpp recipe 👉 🧠 Qwen3.5-4B Q4_K_M GGUF ⚙️ Ryzen 5 7540U — 6C/12T 🧵 --threads 9 🧵 --threads-batch 12 ⚡ --prio 2 🔄 --poll 50 📦 --batch-size 2048 📦 --ubatch-size 512 🚀 --flash-attn on 🧠 KV cache: q4_0 / q4_0 🔧 --repack 💾 --mmap 👤 --parallel 1 🚫 --device none 🚫 --gpu-layers 0 🚫 KV/op GPU offload 🚫 MTP OFF Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks). Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s. That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

🤖 After covering Cosmos and PAN 2, here’s the next Physical AI release I’m watching @runwayml's Praxis-1. Its open weights arrived today. Give it a task, camera observations, and the robot’s current state. It predicts the following 🎬 What the cameras will see next 🦾 The robot movements to execute The robot performs a short action sequence, gets fresh observations, and repeats. That’s the distinction. Praxis-1 outputs robot-control actions alongside predicted video. Runway reports these results on RoboLab-120, a simulated robotics benchmark covering 1,200 episodes 🟢 Praxis-1 → 46.2% success 🔵 Cosmos 3 Nano → 36.8% ⚪ π0.5 → 28.0% These scores aren’t directly comparable to PAN 2’s 34% video-plan completion result. Different tasks, different evaluations. And 46.2% still leaves more than half the attempts unsuccessful. Here is the model stats... 🧠 27B mixture-of-experts diffusion model 💾 Downloadable checkpoints 🔓 Apache 2.0, including commercial use 🛠️ Inference and fine-tuning code 🔧 Full fine-tuning and LoRA adaptation You can test it in simulation and explore adapting it to your own robot. The standard quick start requires an 80GB CUDA GPU, and the main BF16 checkpoint is approximately 54GB before other components and runtime memory. An experimental NVPF4 path targets RTX 5090 and RTX PRO 6000 Blackwell, but closed-loop operation on those GPUs remains unvalidated according to the repository.

So your AI model doesn’t fit on one 🍎Mac? 💻 Apple-owned Claris wants you to connect another. Claris Harbor, a private AI server app is being tested for Apple silicon. (check the thread) Vision 🖥️ Run open-weight models on a single Mac 🔗 Connect multiple compatible Macs over Thunderbolt 5 🧠 Pool their unified memory to accommodate larger models 🏠 Serve AI from hardware you control Claris is aiming to make the setup approachable. Harbor automatically discovers connected Macs, forms a cluster, and lets you browse, download and load models through its interface. Mac clustering already exists BUT making it accessible to people who don’t want to configure a distributed inference system could bring it to a much bigger audience. Harbor is still in early access (see thread), not a public download. Pricing remains unclear, and we need real benchmarks before making performance claims. Running inference locally avoids a cloud provider’s per-token bill. But we'll have to wait to see if Harbor itself will be free.

☠️Is CUDA's moat dead? This is a great localmaxxing development. Anthropic says Claude Mythos Preview brought prompt processing for one of its models to AMD MI355X GPUs in roughly three days. It provided a historical comparison for 6–12 months to bring up new accelerators through traditional engineering. The company says Claude wrote or ported almost all the required GPU kernels, the specialized code that performs the model’s calculations. We spend plenty of time comparing💾 VRAM capacity, ⚡ Memory bandwidth, 💰 Hardware prices But software really determines how much of that hardware we can use. If AI can shorten the work of porting and optimizing inference, developers could support more hardware with less engineering effort. The is a pre-IPO Anthropic company-reported result. So, CUDA’s advantage isn’t gone. But AI helping build better inference software for competing hardware is something to keep track of.

Qwen4 spotted in Qwen's chat code. I still don't see it in QwenCloud. ...and Qwen4-Max is now in preview on Alibaba’s Token Plan. You can also get early access to the new model on QwenWork, Qoder, Qwen APP, and QwenStudio.

New llama.cpp update. Okay, so yeah, your local model fits in VRAM, but then the conversation gets longer. 💨 The model weights aren’t the only thing consuming memory. Its attention cache needs room too. llama.cpp just merged FP8 KV-cache support. Compared with FP16, FP8 uses half as many bits per cached value, which potentially frees memory for longer contexts or more simultaneous sessions on supported configurations. ... giving your hardware more room to work. “But doesn’t LM Studio already have KV-cache quantization?” Yes. Existing options such as Q8 already offer savings. What’s new is another cache format? Q8 stores the cached values as scaled integers. FP8 uses a floating-point representation. Different ways of compressing the attention data your model reuses while generating text. That expands the options for running models locally. It doesn’t establish that FP8 is faster or more accurate than Q8. This reduces cache memory, not model-weight memory. It won’t double your total VRAM or override a model’s context limit.

Yes, it seems like ARC-AGI-4 is solved. Gemini 4 Carbon/Argon, Fable 5.2, GPT-5.6 Astra have all nearly solved the benchmark. What's next? Do we bother? Are these models exceeding our (human) capacity to defeat them with benchmarks?

They've arrived! RTX Spark workstations. With 128GB unified memory, NVIDIA Blackwell, Windows 11 and in a roughly one-liter PC. GIGABYTE announced AI TOP AERO, a compact desktop aimed at running AI locally. The appeal for Localmaxxing is obvious. The announced hardware stats 🧠 NVIDIA RTX Spark N1X ⚙️ 20-core Arm-based Grace CPU 🎮 Blackwell RTX graphics — 6,144 CUDA cores 💾 Up to 128GB unified memory 🪟 Windows 11 There’s another interesting feature, NVIDIA PAIR. Remember this? GIGABYTE says it can distribute inference tasks among available devices on your local network. What’s missing? 💰 Regional pricing and availability ⚡ Independent local-model benchmarks 📚 Usable context and memory measurements

🆕 New Local AI Box! 💻 48GB of shared memory. 48GB of unified memory in a $1,559 mini PC. How much Local AI can you fit in that? MINISFORUM opened US orders for its T5 Max-388. The hardware config 🧠 AMD Ryzen AI Max+ 388 — 8 cores / 16 threads 🎮 Radeon 8060S — 40 compute units 💾 48GB unified LPDDR5-6400 memory 📂 1TB SSD 🔧 Two M.2 storage slots But ... check out the shared memory pool! The CPU and integrated Radeon GPU access the same 48GB of memory, potentially accommodating models that won't fit entirely on a conventional 12GB or 16GB GPU. MINISFORUM even advertises support for quants 🧠 Qwen3.6-35B-A3B 🧠 Gemma 4 26B And remember, 48GB unified memory isn't 48GB of dedicated VRAM. The OS and inference workloads share it. 💰 US price: $1,559 📦 Estimated shipping: Early December ⚠️ Memory is soldered and not upgradeable. The capacity is interesting, but the AI benchmarks will decide the value. 🔗 Link in ALT.

🤯Having built Operating Systems for different platforms in the past, I'm truly impressed but also humbled. Software is solved. This is really impressive. This is not an OS simulator in a browser. An empty folder and eight days later with more than 1,000 AI subagents. A prototype operating system that boots on a laptop. That’s what @AnthropicAI says: Claude Fable 5.2 was built after an engineer assigned it the project. The model coordinated subagents to write, test, and review code. The resulting prototype reportedly includes 🧠 Its own kernel 🔧 Disk, keyboard and network drivers 💾 Persistent file storage 🖥️ Windows and a login screen 📦 An installer 🤖 Built-in Claude functionality The interesting part is how long it took, but also how it coordinated the project, keeping it moving across eight days of implementation, testing, and revision.

Does your local AI use Wikipedia's API to improve its fact recall? Now you have one for Financial data too - Exa Search for Finance. Exa is introducing a finance-focused search service for AI agents, claiming access to over 2 billion financial records and documents. 📈 110M+ daily stock and crypto prices 📄 2M+ SEC filings and earnings calls 📰 250M+ financial news articles So ... 🧠 Run your LLM locally. 🔎 Let it retrieve current financial information through an API. 📊 Have your own model analyze the results. No need to retrain the model every time financial information changes. So this is not a downloadable local financial database. Exa's search index runs in the cloud. Link in ALT

After my NVIDIA Cosmos post, here’s another development worth noting. An open-weights world model better than COSMOS. PAN 2 from @mbzuai Institute of Foundation Models beat NVIDIA. It combines: 🧠 an LLM that describes what happens, when, and how the camera moves PLUS 🎬 a 29B diffusion model that generates the resulting video They ran an experiment keeping the diffusion model fixed. • Direct action instructions → 0.0% • LLM formatting only → 27.8% • Full LLM reasoning → 34.0% Formatting matters enormously. Reasoning adds another 6.2 percentage points. These results used Kimi K3 as the reasoner. They measured whether generated videos depict every requested step, not whether a real robot successfully performs the task. Local AI stats 💾 Downloadable diffusion weights 🔓 Apache 2.0 for PAN 2’s own code and weights 🛠️ Inference, pretraining and fine-tuning code 🎥 Generated clips up to 20 seconds It’s big tho, roughly 58GB for the BF16 diffusion weights alone, before the text encoder, video decoder, and runtime memory. And it remains imperfect. IFM reports physical inconsistencies and failures in roughly two-thirds of complete simulated plans. 🔗 Link in ALT.