Accelerating AI innovation with open platforms and community. The future of AI is open.
Joined May 2018
- Tweets2.7K
- Following2.1K
- Followers12.9K
- Likes1.8K
Pinned Tweet
Everything you need to start self-hosting an open LLM.
Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine.
The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models that cut an 8B from ~16GB of weights to a quarter of that while keeping 98-100% accuracy.
Walkthrough by @cedricclyburn.
Red Hat AI Elevate, a virtual tech showcase on Dec 9. A look at Red Hat AI 3.6, the Red Hat AI Factory with @nvidia, and the built-in guardrails for running agents in production without blowing up governance or cost.
Save your seat:
redhat.com/en/events/webinar…
In @NVIDIAAIDev's OpenShell, your agent never holds the real secret. It only sees a placeholder, and the real key gets swapped in at request time, bound to the one endpoint it's allowed to hit. All of the agent's outbound traffic goes through a default-deny policy proxy, so even if it gets prompt-injected and tries to curl your creds off somewhere, it can't reach anything you didn't allow. You can even pick which programs get to make the call: GitHub CLI yes, bare curl no. It runs on OpenShift too, though that path is still experimental.
@LegareKerrison's blog below has loads of great getting started resources - check it out!
Nvidia has an open source project for sandboxing and managing the secrets you pass to agents.
Goal: Make enterprises comfortable running agents in production.
I wrote a quick blog on gist of it...redhat.com/en/blog/great-pow…
2
19
1,320
You can now upgrade PyTorch without rebuilding vLLM. As of v0.33.0, vLLM ships LibTorch ABI Stable CUDA wheels, so you swap the torch wheel and import it, no vLLM rebuild to pick up a new PyTorch release.
How @vllm_project, @RedHat, and @AIatMeta pulled it off:
vllm.ai/blog/2026-11-06-stab…
New vLLM Office Hours on advanced model compression, with Kyle Sayers and @_soyr_.
They cover:
- LLM Compressor's sequential pipeline and data parallel schemes
- Settings most users never change
- What observers do and how they affect quantization error
- Expert pruning with REAP and MoNE
youtube.com/watch?v=2KZQX_X5…
![[vLLM Office Hours #59] - Advanced Techniques for Model Compression - October 22, 2026](https://pbs.twimg.com/card_img/2120172967321538560/bUA0VRrk?format=jpg&name=800x419)
Happening this week on Thursday 👇
Hola, Madrid. vLLM and llm-d are coming to town: Red Hat and @nvidia are hosting an evening on open-source inference, with @vllm_project, LLM Compressor, vLLM-omni, safe agents, and llm-d on Kubernetes.
November 12 at MOA Institute. Register: luma.com/gl1d2loa
2
11
879
76 guest speakers. 24 organizations. 59 sessions and counting.
vLLM Office Hours is a bi-weekly livestream with the people building @vllm_project: a project update, a guest talk, and open Q&A. Open to all.
Get a calendar invite: vllm.ai/office-hours
ALT Upcoming vLLM Office Hours schedule
Getting benchmark numbers is easy. Getting the right ones is hard. KV cache offloading can cut TTFT from 60s to 10s and lift throughput up to 5.8x, but only when your working set and storage tier line up. Here's how we benchmarked it, caveats and all: developers.redhat.com/articl…
Open models now power most production AI. On Vercel's AI Gateway, open weights went from under 10% of tokens in late 2025 to nearly 80% now. So teams are asking: why not self-host?
Then the real question: vLLM or SGLang? A 🧵:
Hola, Madrid. vLLM and llm-d are coming to town: Red Hat and @nvidia are hosting an evening on open-source inference, with @vllm_project, LLM Compressor, vLLM-omni, safe agents, and llm-d on Kubernetes.
November 12 at MOA Institute. Register: luma.com/gl1d2loa
LLM Compressor v0.15.0 from @vllm_project is out.
New: HIGGS mixed-precision autoquant, block-wise QAD that distills with no teacher model, REAP and MoNe expert pruning, and MXFP6 support.
Plus ~6x faster MXFP4 observation.
Notes: github.com/vllm-project/llm-…
Mistral Large 4 is here on day 1! Get started with vLLM and Red Hat AI Inference. Try the upper bound of performance at the fraction of a cost.
Day-1 Tech Preview image here:
catalog.redhat.com/en/softwa…
Red teaming a model means attacking it on purpose: prompt injections, jailbreaks, unsafe outputs, before real users do.
A hands-on path for your first scan with garak, NVIDIA's open-source LLM vulnerability scanner.
All you need is a free Developer Sandbox:
developers.redhat.com/learn/…
A DFlash2 speculator for Gemma 4 31B, trained with Speculators. It drafts up to 7 tokens per step, and depending on the task the verifier accepts 4 to 7 of them on average. That lands ~1.4x throughput on coding/math workloads at high concurrency:
huggingface.co/RedHatAI/gemm…
Your LLM request doesn't go straight to the model. It gets tokenized, batched with other requests, run through prefill to build a KV cache, then decoded one token at a time. Here's the full inference pipeline in ~7 minutes: youtube.com/watch?v=615-yWyS…
vLLM Office Hours today, 2pm ET: advanced techniques for model compression.
Recent algorithmic work on observers, our AWQ and GPTQ recipes, and expert pruning with REAP and MoNE.
Plus what's new in the latest vLLM releases and.
Register: vllm.ai/office-hours
Hola, Madrid. vLLM and llm-d are coming to town: Red Hat and @nvidia are hosting an evening on open source inference, with @vllm_project, LLM Compressor, vLLM-omni, safe agents, and @_llm_d_ on Kubernetes.
12 November | MOA Institute | Register:
luma.com/gl1d2loa
Amazing ~4-minute walkthrough from @LegareKerrison of how @NVIDIAAI's OpenShell sandboxes autonomous agents using declarative security policies, policy as config, not code buried in the app. youtube.com/watch?v=sXXYJQ-3…
When does vLLM beat llama.cpp? What happens when a model outgrows one GPU? Can you trust an agent benchmark? Where do MCP and agent skills fit?
Red Hat AI folks have been answering these questions on @IBM's Technology channel on YouTube.
A 🧵 of deep dives for your watch list:
vLLM vs. llama.cpp: which local LLM engine actually scales? @cedricclyburn compares the two on performance, hardware, and which fits which workload.
youtube.com/watch?v=0ujh7hfu…
