Pinned Tweet
A talk on verifiable environments for agents in biology
0:00 Where does experimental data come from?
1:41 Modern bio research organized around measurement
2:44 Data analysis is an executable substrate for science
3:23 Five years of learnings building products for pharma
4:13 Trying to use coding models for biology workflows
5:40 Why frontier models can't be trusted (yet)
6:44 Sequencing based spatial analysis
7:38 SpatialBench design principles
8:26 Anatomy of a single evaluation
9:40 Human verification and early long horizon extensions
13:12 Grading functions and the case against rubrics
14:07 New work in multi-omics, therapeutics, biosecurity
Strong performance from Anthropic LSVP models on tissue microscopy tasks
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Microscopy evals are quite fun and its often easy to quickly understand the spirit of the task
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…
Kenny Workman retweeted
(4/4) Saving one of the best for last. Awesome benchmark and a reminder that much of science is based on multimodal data types and literally seeing the world. Congrats to @Jack_Brougher and Martin and team
We’re expanding our benchmark efforts into basic research.
Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works.
We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment.
NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically.
Three public examples:
- Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff.
- Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation.
- A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters.
We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained.
Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%.
Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with.
Paper: latch.bio/neurobench
Leaderboard: benchmarks.bio/neuroscience
Blog: blog.latch.bio/p/neuroscienc…
GitHub: github.com/latchbio/neuroben…
Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.
We’re expanding our benchmark efforts into basic research.
Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works.
We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment.
NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically.
Three public examples:
- Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff.
- Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation.
- A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters.
We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained.
Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%.
Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with.
Paper: latch.bio/neurobench
Leaderboard: benchmarks.bio/neuroscience
Blog: blog.latch.bio/p/neuroscienc…
GitHub: github.com/latchbio/neuroben…
Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.
Kenny Workman retweeted
Great to see more smart people starting to apply their talents to rare disease!
I am fairly confident that with concerted effort and advances in AI and robotic automation, we can solve rare disease diagnosis (and potentially treatment and recurrence) in the next single-digit N years.
Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.
We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.
Five cases from the benchmark:
- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.
- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.
- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.
- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.
- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.
Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.
Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.
Paper: latch.bio/raredisease
Leaderboard: benchmarks.bio/raredisease
Blog: blog.latch.bio/p/raredisease…
We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.
Incredible work from @chriswzou and team.
Kenny Workman retweeted
(3/4 releases) Today we share RareDiseaseBench, a series of 35 diagnostic workups reconstructed from published rare disease cases. In each task, an agent starts with a patient’s presentation, decides which tests to request and why, analyzes the data it receives, and uses the results to propose a diagnosis and management plan over one or more visits. Gemini 4 Argon does the best on this benchmark (43.8% attempt pass rate), with Opus 5.5 and Astra following.
Paper: latch.bio/raredisease
Results: benchmarks.bio/raredisease
More details in thread:
Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.
We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.
Five cases from the benchmark:
- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.
- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.
- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.
- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.
- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.
Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.
Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.
Paper: latch.bio/raredisease
Leaderboard: benchmarks.bio/raredisease
Blog: blog.latch.bio/p/raredisease…
We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.
Incredible work from @chriswzou and team.
Kenny Workman retweeted
(2/N releases) As they say, no stone left unturned, no respiratory virus left unexamined. Huge work from @harm0n and team
AI will become core infrastructure for detecting and tracking respiratory outbreaks.
Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.
Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.
Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.
This benchmark was an especially large effort from many talented scientists.
Leaderboard: benchmarks.bio/respvirus-sur…
Blog: blog.latch.bio/p/biosecbench…
Paper: latch.bio/respvirus-surveill…
GitHub: github.com/latchbio/biosecbe…
Kenny Workman retweeted
The caveat is that scientists/health officals can only use models to track outbreaks if the models respond!
Non-exempt models are generally responsive to tasks involving healthcare records, but answer less on tasks with more experimental/real-world flavors of data. There's also large variability of refusal behavior inside the Anthropic family of models, which is less true inside other families (Gemini/OpenAI/xAI).
Interactive leaderboard/refusal breakdown: benchmarks.bio/respvirus-sur…
AI will become core infrastructure for detecting and tracking respiratory outbreaks.
Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.
Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.
Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.
This benchmark was an especially large effort from many talented scientists.
Leaderboard: benchmarks.bio/respvirus-sur…
Blog: blog.latch.bio/p/biosecbench…
Paper: latch.bio/respvirus-surveill…
GitHub: github.com/latchbio/biosecbe…
These are always very high quality and super technical. Always take away something valuable and meet cool people.
We hosted one with Shreya at our office last year, where we presented a distributed filesystem built on Postgres + S3 for workloads in biology.
xitter.twitterapi.xyz/kenbwork/status/201400…
benchmarks.bio/respvirus-sur… includes LSVP results
AI will become core infrastructure for detecting and tracking respiratory outbreaks.
Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.
Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.
Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.
This benchmark was an especially large effort from many talented scientists.
Leaderboard: benchmarks.bio/respvirus-sur…
Blog: blog.latch.bio/p/biosecbench…
Paper: latch.bio/respvirus-surveill…
GitHub: github.com/latchbio/biosecbe…
AI will become core infrastructure for detecting and tracking respiratory outbreaks.
Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.
Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.
Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.
This benchmark was an especially large effort from many talented scientists.
Leaderboard: benchmarks.bio/respvirus-sur…
Blog: blog.latch.bio/p/biosecbench…
Paper: latch.bio/respvirus-surveill…
GitHub: github.com/latchbio/biosecbe…
Kenny Workman retweeted
Capabilities across models in antibody development are somewhat orthogonal -- focusing solely on overall score can mask that models are good at different, disjoint sets of tasks.
Full interactive filtering at benchmarks.bio/txbench-abdev
Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.
We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.
Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?
Some concrete examples:
- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.
Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.
Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.
We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.
Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.
Manuscript: latch.bio/txbench-ab-dev
Leaderboard: benchmarks.bio/txbench-abdev
Blog: blog.latch.bio/p/txbenchanti…
Kenny Workman retweeted
First of N releases to come out. Incredible work by the team.
We are especially excited about building more and more of the long-horizon, multistep tasks presented as a case study task here
Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.
We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.
Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?
Some concrete examples:
- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.
Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.
Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.
We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.
Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.
Manuscript: latch.bio/txbench-ab-dev
Leaderboard: benchmarks.bio/txbench-abdev
Blog: blog.latch.bio/p/txbenchanti…
Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.
We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.
Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?
Some concrete examples:
- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.
Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.
Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.
We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.
Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.
Manuscript: latch.bio/txbench-ab-dev
Leaderboard: benchmarks.bio/txbench-abdev
Blog: blog.latch.bio/p/txbenchanti…
We introduce an Antibody Discovery Benchmark, a benchmark for testing whether AI agents can make scientific decisions across the stages of therapeutic antibody discovery.
100 evaluations span ten areas from concrete drug programs: from target and modality selection through assay design, binder discovery, binding characterization, cellular pharmacology, antibody engineering, and preclinical candidate de-risking.
Therapeutic antibody discovery is a multiparameter, context-dependent process. Affinity must be considered alongside specificity, stability, solubility, expression, and biological activity.
Every measurement must be interpreted in the context of the experimental system that produced it. Display enrichment can reflect amplification bias rather than binding. Strong binding to purified antigen may not translate to recognition in its native cell-surface context. Apparent affinity can arise from avidity, while changes in valency or molecular geometry can alter function without changing the underlying binding domains.
Across 20 model–harness configurations, even the strongest systems passed only about half of all attempts. Opus 5 with Claude Code led at 53%, with models from Google and xAI following closely. GPT-5.6 Sol with PI reached only 33.8%. Performance also varied substantially by competency: Opus was strongest on target opportunity and cellular pharmacology, Gemini on epitope, escape, and structural mechanism, and GPT-5.6 Sol on sequence, enrichment, and next-cycle engineering decisions.
Manuscript: latch.bio/txbench-ab
Leaderboard: benchmarks.bio/txbench-ab
Sample evaluations and trajectories: github.com/latchbio/txbench-…
Kenny Workman retweeted
Incredibly excited that Latch was part of this (awesome work from @arjunomics ). Incredibly exciting too to see how biology has become one of the most talked about applications of LLMs.
While it's clear that LLMs are improving very quickly, work like GDPx makes it clear that their knowledge and practice of it are very spiky. Not only that, but as LLM capabilities in biology improve we still need better tools and benchmarks for safe deployment. Some of the tasks included in GDPx originated from biosecurity work at Latch.
Latch is hiring extremely aggressively across science, engineering, and ops roles. If you're interested in being part of more initiatives like this one, check out jobs.ashbyhq.com/LatchBio
Stoked to be SemiAnalysis’s exclusive biology partner for GDPx. Very early days for AI in biology.
Recommend digging into the trajectories: gdpx.semianalysis.com/domain…
Read more about our contributions: gdpx.semianalysis.com/domain…
Exceptional folks working on this project @arjunomics @maxkan @AndrewMegalaa
Kenny Workman retweeted
standouts
> slightly higher output token length for Gemini 4 Carbon at a third of the cost per task compared to Opus and Astra
> #1 for biology #2 for Materials
> the overall 1.1% difference in performance is highly subjective across fields -- see what works for yours and make a judgment call
> evals are guiding signals, not the final state