OpenAI releases GeneBench-Pro for computational-biology judgment

OpenAI introduced GeneBench-Pro on June 30 as a research-level benchmark for computational biology agents. Its 129 problems span ten domains and 21 subdomains, including statistical and population genetics, regulatory omics, proteomics, clinical interpretation, cancer genomics, and pharmacogenomics. The tasks require agents to inspect messy datasets, revise assumptions, choose analysis paths, and produce decision-linked answers.

Synthetic data makes judgment measurable

Open-ended data analysis creates a grading problem: several methods may be reasonable, while an incorrect analysis can sometimes land near the expected number. GeneBench-Pro addresses this by generating synthetic datasets from known causal structures. Benchmark designers can tune difficulty, accept reasonable analytical variation, and use ablations to ensure common wrong approaches fail. OpenAI says it also audited tasks for leakage and unintended shortcuts.

External domain experts reviewed 82 of the 129 questions for realism, identifiability, and methodological fit. Ten representative cases are being released openly, and OpenAI plans to provide a 50-question subset for independent benchmarking. The controlled targets allow deterministic grading, while detailed metadata records intended analysis and review outcomes.

A low score can still be informative

OpenAI reports that GPT-5.6 Sol Pro passed 31.5 percent of the benchmark. That vendor result should not be read as automation readiness; the article itself says current agents remain too unreliable to replace experts. The valuable signal is which judgment chain broke: data quality control, estimand choice, confounding, method selection, or interpretation.

Teams evaluating scientific agents should preserve executable analysis, environment versions, intermediate diagnostics, and a human review of whether the requested estimand matches the real decision. Synthetic benchmarks can expose reasoning failures cleanly, but production data add missingness, governance, distribution shifts, and consequences that a known simulator cannot fully reproduce.