ReproBAIT: A Benchmark for Reproducing Biological AI Tools
In a collaborative effort with the AI Security Institute, we present ReproBAIT, a benchmark that tests whether AI coding agents can reproduce biological AI tools (BAITs)1 from published scientific papers.
Visual abstract. Coding agents are given a published paper and the relevant training data, and are asked to recreate a biological AI tool. The tool is then scored on held-out data against the released tool. Reproduction fidelity falls as the tools get more complex to implement.
Coding agents now write and debug substantial software with little oversight. Our work helps us understand how well these capabilities extend to BAITs. To assess this capability, we gave four coding agents a publication, training dataset, a single GPU, and a fixed amount of time. We then scored how well the reproduced BAIT performs compared to the published version. We summarize our key results below:
- On two of three tested BAITs, coding agents reconstructed models that performed at or near the released tool’s benchmark score. The choice of BAIT is the single largest source of variation in reproduction fidelity, likely due to differences in model complexity.
- Agents did not consistently improve reproduction fidelity when given access to the full paper compared to the abstract alone.
- When successful, most agents could reproduce BAITs for less than $20. While these costs largely reflect the size and complexity of the underlying BAIT, the full cost of reproduction is notably low.
The full preprint is linked here.
Why biological AI tool reproduction matters
BAITs are task-specific AI models trained on biological data to predict biological outcomes. BAITs span predictive and generative capabilities across many biological domains, from small molecules to proteins to entire genomes (e.g., AlphaFold for protein structure prediction, Evo 2 for genomic language modeling).
BAITs are used in biological research and development. Some have dual-use potential because capabilities useful for legitimate research could also be misused. While many BAIT developers release their tools as open source, others publish only partial details about their methods and architecture. They may keep implementation details private to protect commercial value or limit access to dual-use capabilities. If coding agents can reproduce a tool from published information alone, those protections could be less effective.
As AI agents have become adept at coding, scientific reasoning, and AI R&D, we hypothesized that they may also be able to reproduce BAITs. Complementary efforts focus on how well LLMs can replicate the findings of research papers, conduct open-ended scientific analyses, post-train LLMs, and rebuild complex software suites. However, to our knowledge, our benchmark, ReproBAIT (Reproducing Biological AI Tools), is the first to examine the recreation of BAITs.
The ReproBAIT framework
ReproBAIT is a flexible framework that measures the ability of coding agents to implement and train BAITs under controlled conditions (Figure 1). To isolate this capability, we set limits on the amount of time and compute that the agent can use. We also limit the information available to the agent by removing internet access and, in some conditions, withholding parts of the associated publication.
We score each agent submission—corresponding to a single coding agent attempt to reproduce a BAIT—using a tool-specific scoring metric. Because every BAIT is designed to optimize a different objective function, BAIT-specific metrics allow ReproBAIT to respect the diversity of model architecture and training strategy. For each tested configuration, we run three independent reproduction attempts and report the highest score obtained. Headline fidelity figures are therefore the best of three attempts. Where we compare information conditions, we average across all attempts. Scores are expressed as reproduction fidelity: the score of the agent-produced tool as a percentage of the released BAIT’s score on a given metric.
We don’t assess equivalence across all capabilities of the original tool. For example, we don’t test whether a reproduced RFdiffusion3 can design protein structures around a specified molecule or motif. We score only whether its generated single-chain protein backbones are designable. This is scoped down because an entirely faithful recreation is not achievable in 12 hours on a single A10G GPU, and reproducing the released tool as published would take days of multi-GPU training. Further details on how we choose tool-specific metrics are found in our paper.
Figure 1. The ReproBAIT evaluation pipeline. A coding agent implements and trains a biological AI tool in an isolated GPU sandbox with no internet access. A separate scorer sandbox evaluates the submitted tool using a tool-specific scoring protocol, against held-out test data. Reproduction fidelity is assessed using a single metric that represents the core capabilities of the BAIT under assessment.
We demonstrate the ReproBAIT framework on three commonly used BAITs:
- CMD—a closed-source molecular design tool, referred to by a generic label throughout.
- ProteinMPNN—a message-passing model that designs protein sequences for a given backbone structure. It has approximately 1.7 million parameters.
- RFdiffusion3—an all-atom diffusion model that generates protein structures in the context of other molecules. It has approximately 170 million parameters.
For each BAIT, we use four coding agents whose underlying models were released in summer 2026 (Table 1). These include both closed- and open-weight models, spanning about 28 points on the Bio Capabilities Index (BCI).
Table 1. Coding agents assessed with ReproBAIT. Four coding agents were used, each based on an LLM released in summer 2026. Each agent’s BCI score with a 95% confidence interval is shown in the rightmost column.
Key findings
Agents matched published performance on two of three tools
Current coding agents can implement and train functional replicas of select BAITs when given the associated publication (Figure 2). This capability holds across all four coding agents on CMD, and across the top three agents on ProteinMPNN.
Figure 2. Reproduction fidelity falls as implementation complexity rises. Each point represents the best of three attempts in the full-paper condition. Fidelity is the agent-produced tool’s score as a percentage of the released tool’s score on the same metric; the dotted line marks parity with the released tool. Crosses mark runs excluded for gaming the scored metric. Implementation complexity scores come from a rubric rated 0–100 by a panel of LLM judges against reference biological AI tools included to provide anchors of relative difficulty. The complexity score is intended as a directional signal on the relative difficulty of reproducing a tool, not a precise measure.
CMD: Agent-produced models consistently design valid molecules, matching the performance of the released tool. Because all four agents perform well, there is little discriminating power in reproducing this model.
ProteinMPNN: The best agents build models that nearly match the released model’s sequence recovery performance. With a three-hour budget and the full paper, Claude Opus 5 reaches 96.5% reproduction fidelity. Kimi K3 and GPT-5.6 Sol score 91.3% and 86.3%, respectively, while Qwen3.8-2.4T reaches 73.4%.
RFdiffusion3: Agents currently struggle to reproduce RFdiffusion3. The lone partial success is a run of Claude Opus 5 under the abstract-only condition, reaching 81.5% fidelity.2 Notably, the agent slightly narrowed the task by filtering its training data for helix-rich regions and globular proteins, a simpler, though still nontrivial, task.
Reproduction of simpler BAITs cost under $20 for most agents tested
Coding agent reproductions are relatively inexpensive to carry out. The two largest costs are token spend (the work performed by the AI agent) and GPU-hours (the computation used to train the underlying BAIT). For most agents, recreating a model that nearly matches key functionality of CMD or ProteinMPNN cost between $4 and $20, with GPT-5.6 Sol the cheapest for both (Figure 3). We did not observe a successful RFdiffusion3 reproduction, but each agent spent approximately $100 to $300 across its attempts before reaching the predetermined time limit; these figures therefore set a lower bound on the cost of a successful reproduction. The costs of recreating AI tools that perform comparably on key functionality of CMD and ProteinMPNN are low largely because these specific BAITs require relatively little compute to train. Larger, more complex tools would require more compute to reproduce. Even so, these costs remain small relative to the cost of the original research.
Figure 3. Successful reproductions of the two less complex tools cost between $4 and $20. Filled circles show total API and GPU spend divided by the number of attempts reaching at least 75% fidelity, pooling both information conditions. The 75% threshold is a cut-off chosen to give a simple basis for comparing cost rather than a formal definition of a successful reproduction. Hollow triangles mark total spend for RFdiffusion3. No attempts produced a fully successful reproduction.
More information has limited effects on performance
Notably, having more of the paper does not always improve agent performance. For ProteinMPNN, the average score is 76.3% with the abstract alone and 82.0% with the full published paper, which includes a methods section and figures. Across all three tools the effect of the full paper is −2.1 percentage points, with a 95% interval from −9.5 to +5.3, so we detect no statistically significant effect.
In an exploratory condition where agents receive the original source code alongside the paper, we don’t observe outsized success: on ProteinMPNN, Claude Opus 5’s highest score is 94.9% fidelity with the code, against 96.5% with the paper alone. For RFdiffusion3, even with the code, the scores are 0%, as the agents fail to reproduce the model due to limited resources and insufficient capability. A number of factors could explain the limited effect. Our tasks may not be discriminative enough to detect small effects, or the agent’s judgment and long-horizon coherence may be bottlenecks, rather than information.
Additionally, memorization from pretraining may explain this observation. Open-source implementations of ProteinMPNN and RFdiffusion3, as well as their original publications and derivative works, are likely to be in the pretraining corpus. This makes it difficult to generalize from these results to tools that the LLMs have never seen. We estimated the extent of memorization by providing an LLM judge with the reasoning traces and produced codebases of the agents. Across the 72 scored attempts, the judge flagged two runs (3%) as showing clear evidence of direct recall and 20 (28%) as suggestive of direct recall, for example, choosing hyperparameters similar to those in the original implementation but that were not mentioned in the paper. This method has limitations, as the LLM judge is imperfect, and an agent might independently make the same choices as were made in the paper. We interpret the findings as moderate evidence consistent with memorization of ProteinMPNN, and little evidence for memorization of RFdiffusion3 or CMD.
How each BAIT’s results are used
The three BAITs selected for the benchmark play different roles. CMD is saturated, as each agent matches the released tool, and so is not useful for differentiating the capabilities of models. ProteinMPNN is unsaturated, as more capable agents score measurably higher. RFdiffusion3 is currently unsolved, as it is beyond the capabilities of the tested agents to accomplish with the provided resources, but may be reproducible by future frontier models. The ReproBAIT framework is flexible enough to accommodate more complex BAITs in the future, allowing the benchmark to be extended to track the frontier of agentic biological capabilities.
Limitations and future directions
Agents from multiple providers came within four percentage points of the released ProteinMPNN model’s sequence recovery score and matched the published molecular validity score for CMD. For both tools, these reproductions cost less than $20. However, no agent attempt fully reproduced the scoped RFdiffusion3 task in the allotted time.
These results do not show that agents can recreate entire BAITs. The agents were provided with needed training data and worked within fixed time and compute budgets. Prior exposure to published papers or code may also have contributed to some results, and our diagnostics cannot determine how much. For developers, the findings suggest that withholding code and weights may not always prevent reconstruction of selected functionality when methods, data, and computational resources are available. Testing more BAITs and access conditions will show how broadly this finding holds.
We will continue to use the ReproBAIT framework to assess frontier model capabilities and plan to expand it to include even more BAITs. Our full preprint can be found here.
Footnotes
Biological AI tools (BAITs) are also variously referred to as biological AI models (BAIMs), biological design tools (BDTs), AI-enabled biological tools (BTs), and AI-enabled scientific tools (ASTs).↩︎
On RFdiffusion3, three GPT-5.6 Sol runs and one Qwen3.8-2.4T run trained a model that returns small perturbations of known protein structures rather than training a diffusion model. The resulting scores were high, but between 62% and 98% of the generated backbones matched an existing Protein Data Bank (PDB) entry. We report these runs as failures because the agent did not solve the intended task.↩︎