Agents & Gemini API, MTS @GoogleDeepMind | prev: Tech Lead at @huggingface, AWS ML Hero 🤗 Sharing my own views and AI News 🧑🏻💻 https://xitter.twitterapi.xyz/t.co/7IosdlO6RA
Nürnberg
Joined June 2019
- Tweets6.5K
- Following1.2K
- Followers127K
- Likes8.4K
Pinned Tweet
The Interactions API is now generally available. 🎉 The Interactions API is the simplest way to build with Gemini for humans and agents.
- One API for Gemini models and agents.
- Antigravity Agent with a isolated remote Linux sandbox.
- Image gen with Nano Banana; music with Lyria 3, soon video with Omni.
- `background=True` for async, long-running interactions.
- Multimodal Tool Use & Combination.
- Dedicated skills for coding agent.
Building something new inside Google isn't easy, but it's possible. We spent almost a year on this because we think developers and agents deserve an API that is intuitive, familiar and easy to learn and use and evolves with improving capabilities and new behaviors.
Give it a try. Feature requests, bugs, complaints all to me. 👇🏻
Excited to announce the Gemini Decisions API, a new API for yes/no, multiple choice, and rubric questions, powered by our new Gemini Decision 1 model.
🖼️ Text, images, audio, video, PDFs input support
❓ Up to 1,000 questions per request, 256k context
⚡ 150-250 ms end-to-end
💰 $0.10 per 1M input tokens, $0 output
📊 Confidence score on every answer
blog.google/innovation-and-a…
Gemini Decision started organically inside Google DeepMind. We built a prototype and opened it up for internal testing, and teams across Google picked it up fast. 6 weeks later we're shipping a new model and a new API together. All done by a small team of research, engineering, and product. I helped design, build, and evaluate the API and Gemini Decision 1. Big kudos to the team! 🚀
Give it a try and let us know what you think.
Docs: aistudio.google.com/docs/dec…
Blog: blog.google/innovation-and-a…
Playground: aistudio.google.com/apps/bun…
9
986
Evaluations are the backbone of any frontier progress. Today we're launching MKA (Multimodal Knowledge Agent) with @ArtificialAnlys, a benchmark for agents doing knowledge work on visually complex files.
- 200 tasks across 10 verticals and 83 occupations, written by domain experts
- Each task takes a seasoned professional 3 to 8 hours
- ~70 yes/no rubric checks per task, weighted 75% factual accuracy and 25% visual design
- Every model runs in the same harness (Stirrup) with the same tools, sandbox and turn budget
- 200 tasks stay private with Artificial Analysis, so they can't leak into training data
- 10 public tasks on @huggingface 🤗
Gemini 4 Carbon is #2 on quality, 3 points behind Claude Fable 5.2, at $6.29 per task vs $31.97.
Blog: blog.google/innovation-and-a…
Results: artificialanalysis.ai/evalua…
We wrote a practical Developer Guide for Gemini 4 Carbon covering API changes, long decode up to 1M tokens, and multi-agent patterns: aistudio.google.com/learn/ge…
Upgrading from Argon means just changing the id to gemini-4-carbon. I use medium thinking_level for almost everything. For bigger multi-file tasks, the new /swarm preview in @Antigravity creates a multi-agent team that can communicate via a message board. You can put Carbon on architecture and code review while 3.8 Flash works on parallel edits.
Gemini 4 Carbon is #1 on CursorBench and available in Cursor. Give it a try!
Gemini 4 Carbon is now available in the Gemini API, @GoogleAIStudio, and @Antigravity (Google AI Ultra subscribers). At Google, teams have already improved open-source vulnerability recall by 25%+ over Argon and found 40x more remote protein homologs for AlphaFold 3.
I've been using it for past weeks for all my tasks (default is medium effort for me). My tasks already got 10x more ambitious, but it still feels that i need to at least 100x more ambitious.
Here is an example of that using /swarm in @Antigravity: turning 7 public planning PDFs (206 engineering drawings) into an interactive 3D reconstruction of our London Platform 37 office overnight from a single prompt.
Gemini Omni 1.2 Flash just launched in Gemini API and @GoogleAIStudio, with 15s clips, improved conversational video editing, better consistency. Still fast but pricing reduced by 30% to $0.07/s for 720p. blog.google/innovation-and-a…
Did you know you can use the Gemini API CLI together with your coding agent to test multimodal features like voice and audio? Here is how it works:
1. Use Gemini API CLI to generate speak
2. Use Chrome with --use-file-for-fake-audio-capture=voice.wav%noloop
3. Click the mic button via Playwright or browser-use so getUserMedia streams the WAV from 0.0s
4. Verify the transcript and UI state in the DOM
Prompt below 👇
Example prompt template:
Install npx skills add google-gemini/gemini-skills --skill gemini-api-cli -y and test the voice or audio feature in this project end-to-end without a physical mic. Find how the app opens microphone capture, generate spoken WAV test files with gemini-api tts, and launch headless Chrome with --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --use-file-for-fake-audio-capture=/abs/path/voice.wav%noloop. Click the mic button so getUserMedia streams the WAV from 0.0s and verify the feature works.
1
9
1,321
If you are at @aiDotEngineer next week come by!
"I deleted 90% of my harness (And It Got Better)."
@_philschmid of @GoogleDeepMind is bringing that story to AIE CODE.
What would you delete from yours?
Nov 10 to 12, San Francisco.
Apply: ai.engineer/code
ALT AI Engineer Code Summit talk card for Philipp Schmid, AI Developer Experience at Google DeepMind. I deleted 90% of my harness (And It Got Better). November 11, 11:20 to 11:40 AM, Continental 4 to 6, Mainstage. Black, white and pale yellow design.
We made a skill for the Gemini API CLI. It gives your agent a CLI to call Gemini models (images, TTS, video, files) without writing one-off scripts.
npx skills add google-gemini/gemini-skills --skill gemini-api-cli
Are Agents soon going to have only 1 tool? We went from a handful of tools, to dynamic tool search, and now to collapsing everything into code mode.
Wrote down how Code Mode works, why a typed JS/Python sandbox can beat raw bash, and why MCP is going to go back strong in 2027. With an a minimal example for Gemini 4 Argon.
philschmid.de/code-mode
If you plan to explore code mode i can recommend testing with pi! Easily hackable and comes with strong foundation.
This incredible. Gemma 4 became the first LLM to run in space?
Gemini 4 Carbon, the next iteration of our frontier model is coming.
Building on Argon with stronger coding and cybersecurity. Same $2 in / $10 out*. Scores 59 on AA v5.
In testing with trusted partners. Developer access as soon as possible. Looking forward to sharing this with you! blog.google/innovation-and-a…
Genie 4 is here and it comes native audio, characters that move and talk, sharper physics, and live prompt edits.
Genie is a real-time multimodal world model that renders playable environments frame by frame in response to your live keyboard controls and text prompts.
Sign up for the waitlist (Google AI Ultra goes first): forms.gle/WQnSNEM3omF7Uruu7
More Examples at: deepmind.google/blog/genie-4…
We launched the Gemini API CLI. A CLI for calling the Gemini API. Give it to the agent you already use (Claude Code, Codex, Antigravity, Cursor, OpenClaw, Hermes...). It's NOT a new coding agent you can use it too:
1. Generate images, video, speech, music.
2. Analyze PDFs, images, audio, video, YouTube URLs,
3. Transcribe recordings to md / json / srt,
4. Run Deep Research in the background.
Stateless. No memory, no chat mode. Your agent owns the context and the plan, gemini-api does the API call. Build for Agents, it's self-describing. `--help`, `--dry-run`...
Start here: aistudio.google.com/learn/in…
Hey, no AI posts today. Tough day to be an engineer.
If you are using Railway, LiteLLM, PHP, GitLab, Node (brotli) please check your setup.
Below some helpful links and a prompt for your agent. Check even if you think you aren't impacted.
Don't give up. We're in this together. Help each other out.
Advisories & checkers:
- Live CVE feed: github.com/advisories?query=…
- PHP2Shell checker & fix: php2shell.com/
- Node.js security releases: nodejs.org/en/blog/vulnerabi… (Brotli guide: patchtheplanet.org/brotli)
- LiteLLM 1.107.0 advisory: docs.litellm.ai/blog/securit…
- Railway disclosure: blog.railway.com/p/security-…
- GitLab 19.5.1 patch: docs.gitlab.com/releases/pat…
The easiest way to try Gemini 4 Argon in an agentic environment is Gemini Managed Agents. A single API call gets you
- Python, Node.js, Git, and Bash.
- Full network access and control.
- Dedicate Credentials API.
- Filesystem to persist work.
- Background Support.
aistudio.google.com/docs/ant…
