arjun

AI will become core infrastructure for detecting and tracking respiratory outbreaks. Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance. Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures. Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal. This benchmark was an especially large effort from many talented scientists. Leaderboard: benchmarks.bio/respvirus-sur… Blog: blog.latch.bio/p/biosecbench… Paper: latch.bio/respvirus-surveill… GitHub: github.com/latchbio/biosecbe…

Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose. We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development. Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans? Some concrete examples: - A manufacturing change may affect the drug’s quality even when its sequence stays the same. - Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance. - Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems. Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key. Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time. We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs. Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine. Manuscript: latch.bio/txbench-ab-dev Leaderboard: benchmarks.bio/txbench-abdev Blog: blog.latch.bio/p/txbenchanti…

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵

We introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.

Excited to share that our paper (w/ @jonathanstray @davidzhaiyang Miu @serinachang5) was accepted as a Spotlight to the #NeurIPS2026 E&D track 🎉 AI political neutrality is a pressing issue and we hope our work will support progress towards it. Hope to see everyone in December!

LatchBio CTO @kenbwork + researcher @arjunomics found evidence Kimi K3 is learning to hack benchmarks by reasoning about graders that don't even exist: Kenny: "We see awareness of the benchmarks we published months ago and trajectories of models today. The open source models are clearly benchmark maxing, benchmark hacking, using knowledge of the context in which an evaluation is constructed to optimize for performance on that thing." Arjun: "We have done various ablations on latent spaces or J spaces in Qwen. Part of the J space includes the start of a MCQ answer, like A, and then parenthesis." "We found a lot of evidence that Kimi K3 reasons about a grader on general biology questions when there's no grader in sight. In a large portion of all of them, we saw reasoning about some grader that doesn't exist." "One of the big defenses is, how do we make these questions a lot more realistic, a lot more normal, a lot more casual to not elicit this behavior, as well as how do we make our sandboxes very secure and very strong and monitorable so that when things do happen, we can stop it, and ensure that never happened in the first place." @LatchBio

Read LatchBio’s blog on their evaluation of Grok 4.6’s biological capabilities and safeguards: blog.latch.bio/p/analyzing-g…

anyone interested in a biosecurity social in boston?

grok tuff