Benchling

Every protein (yes, including your antibodies) should have a predicted structure associated with it. Seems simple enough, except there’s a ton of different structure prediction models and they all have their own quirks. Luckily, we can build an agent which will do the comparison work for us and get smarter over time as we register more sequences. Sample prompt (feel free to tweak): Create a Structure Prediction Agent that runs any time I create an amino acid sequence. It should decide which structure prediction model is likely to perform the best, then run that model. If unsure, it can run multiple models and compare the results. Initially, it should run multiple models to get a sense of which models perform well on our sequences and then remember that. Day 24 of “An agent a day keeps Phase 2 failures away” — structure prediction agent

Mythos 5.2 is the new king of cloning! Definitely a lot better than Mythos 5.1, though only marginal improvement over Astra. Making (slow) progress in biology, which is good to see. Fable 5.2 not pictured because it refused all of the tasks :(

Nobody wants to think about compliance… until you need to think about compliance. Then you really regret not thinking about compliance. Regulated labs are often overrun by paperwork and reporting and have the most to benefit from effective use of AI. Luckily agents can help with the administrative burden and improve the quality of the underlying processes. Sample prompt (feel free to tweak): Create a Deviation Triage Agent that runs whenever a new Deviation is registered. Pull in the related notebook entries, samples, equipment, and reagent lots. Search past deviations for similar events, and summarize their root causes, their CAPAs, and whether those CAPAs actually prevented recurrence. Draft an investigation outline and propose a severity, with your reasoning. Flag it if “human error” is being proposed as the root cause without supporting evidence. Assign the draft to the QA owner for review. Don’t change the deviation’s status or close anything; a person signs off on every decision. Day 23 of “An agent a day keeps Phase 2 failures away” — deviation/CAPA triage agent

Structured data is still king. Having a good scientific data model is even more important in the AI era. Luckily we can use AI to help us maintain and improve our data model. Let’s set up an agent that does exactly that. Sample prompt (feel free to tweak): Create a Data Model Governance Agent that runs every month and takes a look at your data model and recommends suggestions. It should find areas where fields are not being used, schemas that are duplicative or overlapping, and recommend new places to start to structure the data capture process. Day 22 of “An agent a day keeps Phase 2 failures away” — data model governance

Drug discovery is an incredibly difficult multi-objective optimization problem. That’s why I know every biotech company is storing their objectives in a structured way so they can easily view their progress… right? Anyways, let’s create an agent that helps you keep track of progress against your TPP and make good decisions about what experiments to prioritize next. Sample prompt (feel free to tweak): Create a TPP Monitor Agent that runs every time we get new data recorded against our TPP. Provide a molecular scorecard of our best molecule as a canvas. Give details on what attributes have been improving and which have been stagnant. Make sure to compare to publicly available data/comparisons Day 21 of “An agent a day keeps Phase 2 failures away” — TPP monitor

Day 20 of “An agent a day keeps Phase 2 failures away” — workflow updater Keeping track of work and project handoffs is very challenging. As my team has been doing more and more work with agents, I’ve found it challenging to keep track of everything that’s going on — even with a dedicated workflow system. However, if you have a hammer, everything looks like a nail. So why not just build another agent to help you keep track of all the work (and more importantly update it in a durable way so others can see the status of work as well). Sample prompt (feel free to tweak): Create a Workflow Updater Agent that runs every night and looks at work done in the past day. Look at any notebook entries or other work that me or my team has completed. Please group all updates by person. Make sure the person associated with the work is actually the right person. If there are workflows or workflow tasks related to the work done, update the workflow/task state appropriately. Keep track of any change you make. Separate from those changes, you should also review the workflows/tasks you did not change and see if any of their statuses were changed in the last 24 hours by a user. Summarizes both the changes you made as well as the status changes other users made. There should only be one status change summary for each piece of work. If you make a change to a workflow/task, share that status. If you do not make a change but a user has, share that status. If there is no status change, no need to mention the work.

Day 19 of “An agent a day keeps Phase 2 failures away” — sequence QC Sequencing is the backbone of molecular biology, but you need to make sure that your data is correct! I was just talking to a friend who got some data back from a CRO where their QC had put the number of sequences off by a factor of 20. Historically this required a bunch of manual work, a collection of brittle scripts, or a trusty bioinformatics colleague, but now you can set up an agent to do this for you. Sample prompt (feel free to tweak): Create a Sequencing QC Agent that runs every night and looks at notebook entries updated in the past day. If an entry has new sequencing results attached (Sanger .ab1 files or whole-plasmid reads) that haven’t been QC’d yet, align them to the intended sequence referenced in the entry. For nanopore or other high-error reads, call variants from the consensus across reads rather than from individual reads. Add a short summary to the entry saying which clones match and flagging any mismatches, indels, or low-quality regions. If you can’t tell which sequence the reads should be compared against, note that in the entry instead of guessing. S/o @NishantNeel for the video!

New benchmark on biosecurity: BenchBench-Refusals. I like to call this one “shit every biologist knows” (but we put some numbers to it!). Some models are refusing more than 50% of legitimate biology queries! The most interesting thing to me is that refusal rate is not correlated to model quality — some of the strongest models are the best calibrated.

Day 18 of “An agent a day keeps Phase 2 failures away” — dataset builder So you want to train (or finetune) a model to help your scientists. The first challenge most teams face is just finding the right data. Then there’s knowing the experimental context the data was generated from. And finally keeping that dataset up to date as new data comes in. Luckily we can build an agent to help with all 3. Sample prompt (feel free to tweak): Create a Dataset Builder Agent which builds datasets for training ML models every month. If there’s an existing dataset its built, look for new data to add to the dataset. Also look for new data since the last run that could be used to build new datasets. Make sure to include any experimental caveats/notes alongside the data.

This might be my favorite agent yet… having spent years searching for random Excel files that scientists have tucked away, it’s nice to have an agent that just works in the background and cleans up data without me having to do anything. Day 17 of “An agent a day keeps Phase 2 failures away” — data structuring While everyone knows you *should* have robust pipelines which automatically capture data from instruments and structure it, the real world is a lot messier. Scientists drop Excel files or PDFs in their notebooks as a “record” of what happened. Let’s set up an agent to automatically clean up that data and make it ready for downstream analytics and ML use cases. Sample prompt (feel free to tweak): Create a Data Structuring Agent which looks at any new notebook entries from the past day and if there are any attachments (e.g. Excel files, PDFs, etc) where the data hasn’t been put into a structured table, add a structured table and fill out the data from the attachment.

Day 16 of “An agent a day keeps Phase 2 failures away” — wiki builder Tacit knowledge accumulates in every company, but it’s especially pernicious in biotechs. Which cell line should you use for this experiment? Does this machine have a technical artifact? This knowledge can be written down and made into a persistent, shareable record. But staying on top of all the science that’s going on and converting it into digestible facts is too much for a person to do. Enter the Wiki creation agent. Sample prompt (feel free to tweak): Create a Wiki agent that runs every evening. It should look at all the work that had been done during the day and record any meaningful result in a wiki file. If there was no meaningful progress to be recorded, say so. The wiki should be in the Open Knowledge Format and placed in the appropriate project/folder. Make sure every piece of information has a link back to the source information (ELN entry, registered entity, result, etc). Give me a summary of what was updated.

SLIDES! GANTT CHARTS! Every scientist’s (least) favorite topic Keeping track of all the ongoing work in a program is extremely difficult. Especially if you’re asking everyone to take time out of their day to update a tracker. But if you want to scale to more programs without increasing headcount, you’ll need some way of managing bottlenecks and staying on top of what’s happening. That’s where an agent can come in handy. Sample prompt (feel free to tweak): Create a Program Management Agent which gives me a status update on all of our ongoing programs and calls out any delays or blockers. Include a Gantt chart of our progress. Format the update as a slide deck. Day 15 of “An agent a day keeps Phase 2 failures away” — program manager

Lots of discussion recently about target crowding. Some of that is intentional (derisked biology), but some of it comes out of ignorance of how many companies are going after the same indication. Staying up to date on the market has never been easier. Set up a competitive intelligence agent to make sure you’re informed about what’s happening to your pipeline. Sample prompt (feel free to tweak): Create a Competitive Intelligence Agent that runs every Sunday and gives me a full report on the last week. Focus on our lead programs. The first time you run, compile a list of relevant companies and labs and save it to your memory. Then, every report should give me an update on any progress or announcements (press releases, LinkedIn posts, conference presentations, etc). Make sure to also keep an eye out for any new entrants who weren’t on the original list. Day 14 of “An agent a day keeps Phase 2 failures away” — competitive intelligence

Many people assuming that just because I posted this means I think LLMs will not get better at biology... not sure why, I agree this is a task with a nice verifiable reward that LLMs will certainly get better at. However it shows how much tacit knowledge is in biology and how spiky the intelligence of these models is. Many tasks that are much harder to verify (e.g. program decisions, target ID)

Yesterday we showed how to automatically import and structure CRO data. But what happens when those pipelines fail? Whether it’s getting data in a slightly different format, a firmware update to an instrument, or a new plate format — new data is constantly breaking deterministic pipelines. Luckily an agent can help you get the best of both worlds: use pipelines when you can, but automatically recover from errors as the science and data evolve. Sample prompt (feel free to tweak): Create a Pipeline Agent that automatically monitors analysis pipelines that fail. When a pipeline fails, it should diagnose what happened (e.g. wrong file, data shape changed, transient issue, etc) and then perform the appropriate fix. If the fix involves changing the pipeline code, it should create a new draft of the pipeline but not publish it. Day 13 of “An agent a day keeps Phase 2 failures away” — automatically unblock pipelines