MTS@liquid.ai. Do vision pretaining and architecture design for lfm3 vl. Democratize on-edge intelligence.
Joined July 2022
- Tweets29
- Following28
- Followers49
- Likes4
Tianshu Yu retweeted
Adding vision to a language model is more than one extra modality. With large images or long video, the vision encoder can use more compute than the language model itself, so efficient models need a different design. Our VLM team, including @rshubert98, @TianshuYu6, @sina_semnani, and @iamsamstevens, explains how they teach LFMs to see, from mid-training to the custom vision encoder built for LFM3.
Tianshu Yu retweeted
The LFM3-VL-2B is the best model for its size for video understanding, quite usable compared to all other models of that size for checking security camera footage for example.
Tianshu Yu retweeted
LFM3-VL-2B vs Gemma 4 E2B on 50 real scanned receipts.
LFM3 got 41 of 50 receipts fully right. Gemma got 28. 50 receipts from SROIE, same prompt, JSON out, one 5060 Ti 16 GB.
- LFM3-VL-2B: 95% of fields right, 41/50 receipts perfect, 46 s
- Gemma 4 E2B: 87% of fields right, 28/50 receipts perfect, 62 s
Across both tests: Gemma reads messy chat-style text a bit better (95% vs 90%). LFM3 reads receipts better, and was faster both times.
@liquidai @googlegemma
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵
Tianshu Yu retweeted
pointed lfm3-vl-2b at the whole san fernando valley overnight. 16,692 aerial tiles, one rtx 3060, 5.5 hours.
it found 62,873 pools and drew a box around each one.
not a crazy idea, just “find the pools”. it misses about 1 in 5 and boxes a roof now and then. but it’s a 2B model with zero finetune.
imagine what you can do with this. palantir at home lmao
2/ LFM3-VL-2B
64 slots x 16K. ~143 tok/s on one request, ~583 tok/s total with all 64 describing images at once. 11.5GB VRAM.
config:
llama-server
-m LFM3-VL-2B-Q8_0.gguf
--mmproj mmproj-LFM3-VL-2B-F16.gguf
-ngl 99 -c 1048576 -fa on --jinja
-np 64
--cache-type-k q8_0 --cache-type-v q8_0
-b 2048 -ub 1024
--temp 0.1 --top-p 1.0 --top-k 50 --min-p 0.0 --repeat-penalty 1.1
--reasoning on --no-reasoning-preserve
We beat the shit out of Google and Qwen.
huggingface.co/LiquidAI/LFM3…
liquid.ai/blog/lfm3
Great to see my vision pretraining research effort pays off. The VL model now has insane ocr capability and even video capability for the first time. Great thanks to others in vision team that make data and post-training significantly BETTER! More details coming in next week.
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵
Tianshu Yu retweeted
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵