Confirmed 4 Nov 2026 · Science A one-minute read

On real physics research tasks, no AI configuration passed more than 46 of 168 attempts.

FrontierPhysics posted on November 4 and surfaced here a day late. 42 physicists wrote 56 tasks from their own research, and 22 models were run in 26 configurations.

Until now

Frontier agents were near junior research work, held back by knowledge gaps.

Now

The gap is judgment: rigour and intuition, not facts or execution.

Why it matters

If the diagnosis holds, it points away from the current fix. Scaling data and compute addresses knowledge and execution, which these agents already have. Rigour means knowing which of your own steps is unjustified, and nothing in the training recipe teaches that. The benchmark also scores whether the output is publishable, not merely correct.

Established

56 tasks were written by 42 physicists from their own research.
No configuration passed more than 46 of its 168 rollouts.
The authors attribute failures to reasoning and intuition, not knowledge.

Not established

Which model scored highest. The abstract names none.
That models cannot do physics. These tasks demand publishable research.
That the rubrics are neutral. The task authors wrote them.

The five-minute readThe record, the timeline, the weight and 4 sources

The record

The paper is FrontierPhysics: Evaluating Agents for End-to-End Physics Research, arXiv:2611.06910, submitted Wednesday November 4, 2026 at 18:56 UTC, 41 pages with 10 pages of main text, 19 figures and 11 tables, by Bingran You and 71 other authors. We read the abstract page at arxiv.org. The abstract states: existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. FrontierPhysics is an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering 9 domains across theoretical, computational and experimental physics. Each task requires an agent to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Scoring combines deterministic verifiers with expert-authored rubrics, and judges whether the research is both correct and publishable. Across 22 models in 26 configurations, the abstract states that no configuration passes more than 46 of its 168 rollouts, which is about 27%. The authors report that manual review of agent trajectories shows the main weaknesses of frontier agents are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors. The author list includes Gerbrand Ceder and Hartmut Haeffner. The paper is filed under Computer Science, Artificial Intelligence, with a DataCite digital object identifier pending registration. AI Weekly, not the abstract, attributes the top score of 27.4% to Anthropic's Claude Opus 5.5; we carry that as reported and note that 46 of 168 is 27.38%. We did not open the full text, so the per-model table, the 9 domains and the rubric design are not in this record. Not established: which model holds which score, how the rubrics weight publishability against correctness, and whether the physicists who wrote the tasks also graded their own.

How it got here

Sep 25
Anthropic says Claude finished a physics calculation no person has managed, on about 100 dollars of computing. Edition 034. A single hard calculation, not end-to-end research.
Sep 6
OpenAI says it met its own deadline for an AI that does a junior researcher's work. Edition 014. This benchmark is a direct test of that claim.
Oct 6
OpenAI publishes 722 mathematics manuscripts written by a model it has not released. Edition 044. Volume, with publishability unjudged.
Nov 2
A Stanford-led benchmark finds frontier models fold when a teammate pushes back. Edition 071. The same failure mode, named differently.
Nov 4
FrontierPhysics is posted to arXiv at 18:56 UTC. It reached us through November 5 coverage, so we card it under its true event date and say so.

Who has to react

The 42 physicists who gave their own research to a test; the laboratories whose junior-researcher claims this scores, OpenAI and Anthropic above all; Anthropic, credited with the top configuration by secondary coverage; the benchmark builders whose fixed-question tests this calls insufficient; journal editors, since publishability is now a scored axis.

What to watch

Would weaken itAny model configuration passes more than 84 of 168 FrontierPhysics rollouts, half the total, in a result published by the authors or independently reproduced, by November 4, 2027.
Would strengthen itA third independent evaluation published after November 4, 2026 attributes frontier agent failure primarily to reasoning rigour or judgment rather than knowledge or execution, by June 30, 2027.
Would strengthen itThe benchmark's reported top score at least doubles within 6 months of publication, by May 4, 2027.

Why it weighs this much

Signal strength

CostMedium

CommitmentMedium

ReversibilityEasy

PersistenceYear

Cost is medium. 42 physicists writing 56 tasks from their own research, plus expert rubrics and 168 rollouts across 26 configurations, is many months of specialist labour that cannot be bought cheaply or generated. Commitment is medium: a preprint with 72 named authors, several of them senior, is a reputational stake, but it is not peer reviewed and the paper is a day old. Reversibility is easy, because a benchmark can be saturated. That is what happens to benchmarks, and a single capable release can move the number. Persistence is a year at best, which is the honest reading of any evaluation result in this field. We score it modestly and card it for the diagnosis rather than the score.

The words

End-to-end benchmark: A test of the whole job rather than a step of it. Here the agent must read the literature, plan, run the work and write it up, and is scored on the finished output.
Rollout: One complete attempt at a task. Running each task several times shows whether a success was repeatable or a lucky draw.
Deterministic verifier: An automatic check with one right answer, such as a number or a compiling program. It cannot judge whether research is worth publishing, which is why rubrics are added.
Configuration: A model plus its settings, such as tool access or reasoning budget. The same model can appear several times under different configurations.

Sources

Your call

Which explanation do you believe?

A call is final. Other people's calls show after yours.