Evidence table
AI Scientist Claims vs Reality Tracker
A filterable, sourced table for separating AI scientist headlines from demonstrated capabilities and replication evidence.
Key facts
Before you use this tool
Rows use primary-source baselines where possible: papers, lab announcements, and official documentation.
The table distinguishes workbenches, hypothesis engines, code loops, robotic agents, and self-driving labs.
Replication status is descriptive, not a final judgment on scientific value.
7 rows
| System | Claim | Reality | Autonomy | Status | Source |
|---|---|---|---|---|---|
| Claude ScienceResearch workbench | Claude as a workbench for scientific discovery | Official positioning describes a scientific workbench that can help organize research work; experimental autonomy depends on connected tools, permissions, and review gates.Not enough on its own to claim closed-loop autonomous experimentation. | Human-directed workbench | Official announcement | Anthropic |
| Sakana AI ScientistCode experiment loop | Fully automated open-ended scientific discovery | Agentic workflow for idea generation, experiment execution in code, paper writing, and automated review in constrained machine-learning research settings.Autonomy is strongest in bounded code experiments, not physical lab actuation. | Bounded code loop | Paper and follow-up literature | arXiv |
| AI Scientist-v2Code experiment loop | Workshop-level automated scientific discovery via agentic tree search | Follow-up system expands search and review mechanics for machine-learning research tasks with automated paper-style outputs.Workshop-level output is not the same as independently confirmed scientific discovery. | Agentic tree search | Preprint | arXiv |
| Google AI co-scientistHypothesis engine | A multi-agent AI partner to accelerate research | Multi-agent system for generating, ranking, and refining hypotheses with scientist interaction and supporting evidence workflows.Designed as a co-scientist partner, not an unsupervised lab operator. | Human-in-the-loop co-scientist | Paper and official announcement | Google DeepMind |
| CoscientistRobotic chemistry agent | Autonomous chemical research with large language models | LLM-based agent planned and executed parts of chemical research workflows by using tools and interfacing with automated chemistry equipment.Lab-connected autonomy requires strict boundaries, expert oversight, and domain-specific safety controls. | Lab-connected agent | Nature paper | Nature |
| A-LabSelf-driving laboratory | Autonomous laboratory for accelerated synthesis of novel materials | Closed-loop materials discovery workflow combining computation, robotic synthesis, characterization, and decision-making in a lab setting.The system is specialized infrastructure, not a general-purpose AI scientist. | Closed-loop materials lab | Nature paper | Nature |
| Self-driving laboratory literatureSelf-driving lab | Self-driving laboratories for chemistry and materials science | Review literature describes closed-loop experimentation patterns, automation stacks, optimization loops, and practical limits.A field pattern, not one universal system or guarantee of autonomous discovery. | Closed-loop experimentation | Review literature | Nature Communications |
FAQ
Claims vs Reality Tracker Questions
What counts as reality in the claims tracker?
Reality means the demonstrated capability in the cited source: what the system actually did, what tools it controlled, and what evidence was preserved.
Why include systems that are not fully autonomous?
Because many AI scientist headlines blur workbenches, co-scientists, and closed-loop agents. The tracker makes that boundary visible.
How often should the tracker be refreshed?
It should be refreshed when a primary paper, official release, independent replication, or safety guidance changes the evidence base.