Pipecat AI

Phonon: a new open, on-device, text-to-speech model from Gradium. It's really good. Click through, below, and you can test it in your web browser (enabled by an optimized WebGPU inference stack). Go do it, just to *feel* what it's like when time-to-first-byte is ~10ms. There are lots of reasons to run agents/models in the cloud. But the trade-offs are cost for hosting and inference, and latency. We have to work hard to get time-to-first-byte down below 50ms when we're running our models in the cloud. And usually for a text-to-speech model we're aiming for 100-150ms TTFB in production. Running on-device means you can optimize to run one inference request at a time in a specific hardware and software environment. And the cost is ~$0. (Or, in very demanding applications, the power budget/battery life for the device.) Open and on-device models make it possible for us to imagine and build new things. I expect to see people post a lot of great Phonon demos over the next few weeks!

We have some fun voice AI stuff to talk about on the @NVIDIAAI Nemotron Labs livestream tomorrow: fine-tuning audio and LLM models for specific agents and use cases. Join us live if you want to chat and ask questions. 10:00am Pacific Time on Tuesday.

Saaras V4 is here. Built for the messy reality of speech, accents, dialects, noise, and mixed-language conversations. → 5 modes: Transcribe, Translate, Transliterate, Codemix, Verbatim → Keyterm prompting: Guide the model toward the words that matter → 22 Indian languages + English → Real-time streaming: <150ms time-to-first-token → SOTA performance across all 22 Indian languages Speech recognition, built for the real world. See Saaras V4 in action in this demo.

Oh, are we animating our 'little guys'? The @pipecat_ai mascot was made for this moment...

World Cup Voice Commentary right in your MenuBar! @pipecat_ai powered voice bot, not only reports on the scores but can answer any questions I have about the games! Using only local models on my M4 Max. Under the hood: + Kokoro TTS for commentary + Gemma 4e4 (with and without thinking) + Nemotron 3.5 for ASR Built with Pipecat subagents: when the menubar starts, it spins up a main loop of [STT] | [LLM] <-> [subagent when a game starts] | [TTS] There is one subagent per game. Schedules are pulled regularly, so I can ask stuff like “Who will the US play in the round of 32?” and have it reason over fixtures, for example. A collector ingests live commentary (per match-id, home/away, and metadata) into a local DB and exposes it via a CLI so pipecat's subagents can query it. Each game subagent gets an initial prompt about “its” match and tools including an /analyze-game-skill. The skill was built and tested with the early R1 games, especially the chaos of Canada-Qatar game (with injuries, goals, and two yellow cards becoming a red!) The result: when something happens in the game, the TTS chimes in, excited or dejected, about missed chances, corners, cards, etc. Subagents maintain their own context, so with push-to-talk, which invokes the STT, I can ask about what is happening in a current, past, or future games, fixtures, etc. The main LLM routes to the correct subagent, or can directly check via CLI, or spins up a new subagent to go hunt down answers (e.g., scores about R1 games, schedule changes, etc.) Today’s rain delays accidentally stress-tested having multiple concurrent games in Round 3, and the subagent orchestration held up nicely. Clips + screenshot below 👇 Halftime announcement, second goal, and the third goal. These were voice memo recordings, aargh

Until today, a transcription model never knew what your voice agent just asked. The one we're launching today does. Universal-3.5 Pro Realtime from AssemblyAI is the first realtime speech-to-text model that takes your agent's side of the conversation as context. When your agent asks for an email and the caller rattles it off, the model already knows an email is coming, and captures it perfectly. Same for an order number read fast, an account ID, a medication, or a name spelled out over poor quality phone line.

You can turn an existing LangGraph agent into a fully functional voice agent with @pipecat_ai. This 17-minute walkthrough shows you exactly how to do it.

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today. New architectures enable new frontiers for speed and quality. We're now the only provider to have #1 models for both speaking and listening.

Flux Multilingual is live. Real-time conversational speech-to-text for voice agents in 10 languages, with monolingual-grade accuracy, turn detection, and code-switching. Deploy once and launch globally. Learn more → deepgram.com/learn/introduci…

I did it. I beat gradient-bang. First person to discover the full map. The only record that can’t be broken. Thanks for all the fun @pipecat_ai @kwindla @chadbailey59 etc. I wasn’t sold on voice agents until I found this game and the experience is actually really great.

Big day today. Pipecat version 1.0. Two years in the making. The most widely used framework for voice agents, but not just voice agents. Pipecat is a framework for realtime, multi-modal, multi-model AI applications. Contributions from NVIDIA, all the foundation labs, AWS, GCP, and Azure. Used by thousands of startups, scale-ups, and enterprises. Pipecat Subagents v0.1.0. A new library for sub-agent orchestration. Which is just a fancy way of saying running lots of inference loops in parallel, with partially shared context. The basic architecture of Pipecat Subagents is an event bus that works locally, and over the network. And Gradient Bang. The side project that broke containment. Built with Pipecat and Pipecat Subagents. Gradient Bang was actually the proving ground for the early Subagents work. But ... it's also a really fun game.

Real-time transcription just got a significant upgrade. Universal-3-Pro is now available for streaming — bringing AssemblyAI's most accurate speech model to live audio for the first time. Developers building voice agents, live captioning tools, and real-time analytics pipelines now get three things they've been asking for: 🔹 Best-in-class word error and entity detection across streaming ASR benchmarks 🔹 Real-time speaker labels — know who said what, as it happens 🔹 Superior entity detection for names, places, orgs, and specialized terminology in real-time 🔹 Code-switching and global language coverage built-in

Voice workflows just got stronger with gpt-realtime-1.5 in the Realtime API. The model offers more reliable instruction following, tool calling, and multilingual accuracy. Demo with @charlierguo