Engineering at Meta (@Meta_Engineers) header
Engineering at Meta (@Meta_Engineers) profile picture

Engineering at Meta is a technical news resource for engineers interested in how we solve large-scale technical challenges at Meta.

Joined August 2009
Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)Photo by Engineering at Meta (@Meta_Engineers)
🧠Forum, Meta’s new standalone Groups app, was built by a small team that brought an AI agent into the development cycle as a participant — planning, building, testing, and verifying. Building Forum we moved from human-in-the-loop — engineer drives, AI assists inside each step — to human-on-the-loop, where the agent runs the iteration (plan, implement, build, test, correct, repeat) and the engineer sets the goal, watches the run, and owns the outcome. The key decision: success wasn’t measured in code volume. It was measured in changes that build, pass tests, and survive review. ✅ Reached core feature parity with Facebook Groups in eight weeks 📈 An autonomous stress-testing loop caught a memory issue during development, roughly halving memory use and page faults 🔗 Read how our engineers went AI-native to build Forum: engineering.fb.com/2026/10/1… #AI #SoftwareEngineering #DeveloperTools #MobileEngineering #Engineering
26
29
7
325
18,764
🧠 Most AI bug-catching tools review your code the way a human reviewer would, only faster. What if the AI wrote a test that proved the bug instead? That’s Catching Just-in-Time Tests (JiTTests) — a novel approach where LLMs generate executable tests designed to fail, surfacing behavioral changes in a code change before it lands. After a year of research, it’s now part of Meta’s infrastructure. ✅ 589 engineers have engaged with it, reporting a 60% true-positive rate 📈 In the first half of 2026, it helped catch 74 privacy SEVs at the diff stage, lowering overall privacy SEV occurrence by 20% 🔗 Read how it works: engineering.fb.com/2026/10/2… 📄 Read our FSE 2026 paper, “Just-in-Time Catching Test Generation at Meta”: arxiv.org/pdf/2601.22832 #AI #SoftwareEngineering #SoftwareTesting #DeveloperTools #Engineering
33
77
16
768
44,092
🏆 Congratulations to Meta’s Ke Mao, Mark Harman, and Yue Jia, recipients of the ISSTA 2026 Most Influential Paper Award for "Sapienz: Multi-objective Automated Testing for Android Applications" (2016). Sapienz treats Android testing as a multi-objective search problem: explore more of the app, find more crashes, and minimize the steps needed to trigger them. That third objective is what makes a crash report actionable instead of just reproducible. 📈 Deployed at Meta within 17 months of publication, it now runs millions of tests daily, surfaces thousands of unique crashes a week, and helps safeguard hundreds of app releases a month. ⚙️ It also became the app-exploration engine for dynamic taint analysis at WhatsApp, driving exploration through test accounts and synthetic conversations. 🔗 Read more on our engineering blog: engineering.fb.com/2026/10/1…
8
16
132
7,172
Since 1985, your computer has asked the network what time it is and simply believed the answer. Meta's public time service now speaks NTS (Network Time Security, RFC 8915) at nts.meta.com. Every packet is authenticated, so a device can verify the time came from us and wasn't modified in transit. It’s free, public, and open source. Read the full technical write-up on our Engineering Blog: engineering.fb.com/2026/10/0…
71
152
27
2,290
200,228
🕛 Since 1985, your computer has asked the network what time it is and simply believed the answer. Meta's public time service now speaks NTS (Network Time Security, RFC 8915) at nts.meta.com. Every packet is authenticated, so a device can verify the time came from us and wasn't modified in transit. It’s free, public, and open source. ➡Read the full technical write-up on the Engineering Blog: engineering.fb.com/2026/10/0…
6
23
2
188
8,793
Ads retrieval has to narrow a huge candidate pool in milliseconds. Historically we couldn't afford ranking-quality models there — too expensive to pull all that ad data into the GPU. Next-gen Andromeda fixes the real bottleneck: HBM bandwidth. With NVIDIA, we moved the compute inside the GPU — embedding gather, MoE interaction, scoring in one fused kernel, keeping tensors in registers/shared memory instead of HBM. Result: gating went 5.3x from 2.4B to 10B+ ads/sec (~85% of peak HBM bandwidth, as fast as a dot product), index matching passed 100B ads/sec. End-to-end: 1B+ ads/sec per server, scoring every eligible ad — 100x over our 2024 system. Ranking-level quality at ~300x fewer FLOPs: engineering.fb.com/2026/10/2…
10
23
1
200
11,454
Instagram Live ingest now runs on Media over QUIC (MoQ), the emerging IETF standard for real-time media over QUIC, on both iOS and Android. For nearly a decade our live video ran on RUSH, the in-house QUIC-based protocol we built to move past RTMP. RUSH proved QUIC could carry live media at scale — but evolving a transport alone is slower than evolving one with Google, Akamai, Cisco, Cloudflare, and academia at the IETF. Swapping a transport under a live production system meant: ⚙️ AMoQ publisher built on moxygen, our open-source MoQ library, running on mvfst 🔀 An edge translation layer converting MoQ frames to RUSH framing, so nothing downstream had to change 🎛️ Server-side routing, so the ramp didn't depend on app-version adoption ✅ The result: the MoQ path matched RUSH across the streaming-quality and reliability measures used for the rollout. Parity was the target, and it clears the way for end-to-end MoQ playback. 🔗 Read the full deep dive here: engineering.fb.com/2026/10/1…
9
34
8
380
27,129
🎯 If your AI orchestrator is making domain decisions, it's doing too much. We'd built one AI agent that knew everything about onboarding data columns across Meta's recommendation systems — every schema, every code generation rule, every population pattern for 10+ data sources. Past a certain point, more knowledge made it worse, not better. It would apply one subsystem's naming conventions to another. ⚙️ So we made the central agent dumber on purpose. It now routes 20+ workflows to 14 specialists and does no domain reasoning at all. 📈 Configuration prep for a new data column went from three to four days by hand to under an hour. 🔗 See how our engineering teams built it — and how to apply the pattern: engineering.fb.com/2026/10/1… #AI #MachineLearning #TechInnovation #SystemsEngineering #Engineering
15
20
3
215
17,010
⚙️ Better ad recommendations aren't only about bigger models. They're about how you represent the data underneath. Meta's ads recommendation systems have long learned about each item one ID at a time, which means a brand-new ad starts with almost no signal. GenRecs Tokenization changes the foundation: Every ad, product, Page, and pixel is represented with tokens drawn from a stable semantic vocabulary. Because tokens describe what an item actually is, a newly launched trail shoe inherits what the model already learned about last season's. 📈 Paired with Meta's Generative Ads Model (GEM) and our Adaptive Ranking Model, it delivered a +0.35% lift in conversions for advertisers. 🔗 Read the full deep dive into GenRecs Tokenization story: engineering.fb.com/2026/10/1…
8
10
78
5,527
Inference fleets face a capacity paradox: They can be globally constrained while meaningful pockets of capacity remain locally idle, stranded by peak provisioning, regional resource ratios, and accelerator-type mismatches. We built a system that aggregates those pockets into a shared opportunistic capacity pool and spends them on ads recommendations through two elasticity levers. ➡️Rank More Ads (RMA) increases the number of candidates the ranking stages evaluate, which needs both CPU and accelerator surplus. ➡️ Serve Better Models (SBM) raises the share of requests routed to a more powerful model, consuming accelerator-only surplus that RMA can't. The pool is preemptible, so physical and logical isolation, PID-controlled feedback loops, and millisecond value-aware load shedding at both server and client keep baseline serving inside its latency and reliability targets. 📘Read a full technical deep dive: engineering.fb.com/2026/10/1…
9
27
1
130
10,328
🔐 A privacy requirement has to hold no matter which path data takes. Historically that meant implementing the same enforcement logic separately inside every system — months of work per requirement. PrivacyLib is a common library running inside the clients of hundreds of Meta's data and processing systems. Each integration maps its native APIs onto five shared operations — read, write, RPC, request entry, and inline execution — and modules are written against those rather than any system's API. Requirements compile at build time into their simplest evaluable form, and per-request and per-host caches keep the common path off the network. For raw RPC traffic, a cryptographically signed token lets ServiceRouter confirm a request already passed the richer client-side check. 📈 The result: billions of checks per second across the fleet, while consuming a small fraction of Meta's compute. Enabling a module on an already-integrated system now takes days. 🔗 Read the full technical deep dive: engineering.fb.com/2026/10/2…
🤖 Made with AI
6
4
65
6,373
🌐 At OCP Global Summit 2026 (@OpenComputePrj) we shared the open networking infrastructure behind our latest AI workloads — across scale-up, scale-out, and scale-across. ❄️ IcePack, our first liquid-cooled network rack, holds IceCube 102.4T switch blades: 128×800G non-blocking, in Tomahawk6C and Spectrum-6 variants, both running FBOSS on OCP SAI. That's 2x the bandwidth of our 51.2T platforms. 🔀 Flattened Planar Fabric (FPF) gives each GPU eight independent links to eight switches, flattens a 3-tier Clos to 2 tiers, and moves load balancing into the NIC. 🌎 Backend aggregation (BAG) interconnects fabrics at petabit scale — tens of Pbps between regions — so Prometheus operates as one cluster. ⚡ For scale-up: Tahan, our Ethernet-based 51.2T switch for MTIA 300, with a 72-accelerator-per-rack design on 102.4T silicon previewing next. 📐 Also new: OSPA, co-led with Microsoft and Alibaba, with spec v1.0 submitted to OCP, and open-source MetaRoCE. 🔗 Read more on our engineering blog: OCP Summit 2026: engineering.fb.com/2026/10/1…
5
1
9
3,136
🎯 Excited to share how we scaled the 2026 generation of GEM — Meta's Generative Ads Model — the central model and compute engine behind ads across Meta. Last year we introduced GEM, the industry's largest recommendation systems foundation model. This year, we are pushing the frontier again with breakthroughs across model architecture, cross-surface cross-domain learning at Meta’s scale, innovative intelligence transfer, and LLM-scale AI infrastructure: Bending the scaling curve with Kunlun. GEM's new LLM-style architecture, co-designed with the latest GPU hardware, delivers 2.5x model complexity growth over earlier versions of GEM and processes 2x longer engagement history per person, while modeling the massive feature interactions unique to recommendation systems. Expanding learning across Meta's ads ecosystem. GEM is the first model to jointly predict clicks and conversions across all surfaces — Facebook, Instagram, Reels — and across major deep-funnel objectives like off-site conversions and mobile app installs. Doubling the effectiveness of intelligence transfer with LoopFM. This new frontier post-training technique passes GEM's internal representations and historical memory to the models that serve ads as a sequential context window, at no additional compute cost or latency. Elastic pre-computation (Solaris) further harvests gains from upstream models. Scaling frontier training efficiently. Through deep model and infrastructure co-design, we increased total training FLOPs by 4x while doubling end-to-end training efficiency to 20–25% MFU over the past 12 months, across thousands of latest-generation GPUs. Recommendation systems don't scale the way language models do — heterogeneous features, shifting distributions, and no option to serve a frontier model in a live request. Making GEM scale like LLM took rethinking the scaling architecture, the post-training and the training stack together. Very proud of this team. 📈 The payoff: for people, ads that better match what they're actually looking for. For advertisers, an 18.9% lift in conversions and 12% in click-through rate across our ad surfaces and objectives over the past year — with 5.8%, 4.1% and 1.6% conversion uplift on Facebook, Instagram and off-site conversion for targeted users. 🔗 Read the full engineering story: engineering.fb.com/2026/10/1… #AI #MachineLearning #AdsRanking #MLEngineering #Engineering
9
11
1
95
6,648
We're open-sourcing Rebalancer, the assignment-problem solver Meta has used for resource allocation across our infrastructure for over nine years. Given a set of objects and a set of bins, Rebalancer assigns one to the other to optimize your objectives under your constraints, including hardware placement, service and task placement, traffic routing, and more. It separates how a problem is specified from how it's stored, solved, and debugged, with both an optimal MIP solver and a parallelized local search solver. At Meta it solves ~40 million assignment problems a day across 30+ problem formulations. P99 solve time is 12 seconds on a problem with 265k objects and 3.2k bins. 🔗 Read more on our engineering blog: engineering.fb.com/2026/09/2…
48
358
40
3,754
273,217
Impressive results from a great collaboration between our teams! Read the full deep dive 👇
The Instagram Direct engineering team modernized @Instagram messaging at scale using Jetpack Compose. 🏗️ By replacing their legacy architecture with an AI architecture and Compose-native alternatives, they maintained a high performance bar and reduced their UI codebase by 50%. 🤯 Read more → goo.gle/IGCompose
15
6
56
8,068
Last year we introduced Private Processing, Meta’s confidential computing infrastructure that keeps everything you share with your AI assistant private, so that no one, not even Meta, can access your data. At #MetaConnect, we announced that we’re bringing these same privacy benefits to our AI glasses. Read the technical deep dive on our approach: engineering.fb.com/2026/09/2…
13
29
2
157
9,101
Today we’re announcing Petal: the world’s first petabit-class transoceanic subsea cable and the first to deploy multi-core fiber at scale. Spanning 7,000 km (4,000+ mi) between France and the United States, Petal will deliver 1 Petabit per second (1 Pbps / 1,000 Tbps), doubling the capacity of today’s most advanced transatlantic systems. Petal is set to enter service in 2029, setting a new standard for high capacity, high density SDM subsea cable designs. Read the full technical breakdown on our engineering blog: engineering.fb.com/2026/09/2…
130
435
183
6,419
1,525,679
To help Meta subject matter experts focus more on work where their knowledge matters most, we’ve built an AI agent that acts as a secondary expert for a given domain. This system makes deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon. Two things make it novel and portable: 1️⃣ structured, auditable knowledge architecture that separates what the agent knows from how it reasons 2️⃣ self-improvement loop that turns expert feedback into verified updates without retraining We’re sharing a deep dive into our approach and architecture here: engineering.fb.com/2026/09/0…
14
38
5
370
23,207
How do you manage database connections when you have 1M+ clients hitting 500k+ database hosts? We’re sharing a deep dive into ZGateway, the stateless proxy tier handling traffic for ZippyDB (>1B ops/sec). The direct-access model created an unbounded TLS connection mesh, wasting CPU and causing OOM crashes during reconnection storms. ZGateway decoupled the fleets, dropping per-host connections by 97–98% with just a 6% compute overhead. Read the full engineering deep dive here: engineering.fb.com/2026/09/0…
17
103
18
1,083
246,552
Training and serving frontier AI models depends on fast, reliable networks that move data between GPUs without wasting compute cycles. To meet this challenge at scale, we designed MetaRoCE: a clean-sheet RDMA transport protocol purpose-built for AI workloads on commodity Ethernet. We’re releasing the MetaRoCE specification, a reference software implementation and a compliance test suite through the @OpenComputePrj to enable the broader industry to adopt, implement, and build on it. Read the full technical deep dive: engineering.fb.com/2026/08/2…
2
10
1
92
8,627