{"schema":"sanctity-signal/0.2","record_type":"signal","id":"frontierphysics-e075","domain":"science","drop_id":"signal-e075","occurred_at":"2026-11-04","how_to_use":"This is a structured record of one real-world action and its competing interpretations, from Sanctity Signal. To build content from it (a reel script, a blog, a thread): open on the hook, ground the viewer with plain and observed, walk the dated trail, present the interpretations as live tension between named readings, and close on the before/after shift and what_to_watch. Hard rules: items under established are fact; items under unknown are never presented as fact; the evidence status (verified or reported) is stated, not dropped; sources are cited.","actor":"FrontierPhysics authors","action_type":"publication of an end-to-end physics research benchmark built by 42 physicists from their own work, in which no model configuration passed more than 46 of its 168 attempts","observed":"The paper is FrontierPhysics: Evaluating Agents for End-to-End Physics Research, arXiv:2611.06910, submitted Wednesday November 4, 2026 at 18:56 UTC, 41 pages with 10 pages of main text, 19 figures and 11 tables, by Bingran You and 71 other authors. We read the abstract page at arxiv.org. The abstract states: existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. FrontierPhysics is an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering 9 domains across theoretical, computational and experimental physics. Each task requires an agent to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Scoring combines deterministic verifiers with expert-authored rubrics, and judges whether the research is both correct and publishable. Across 22 models in 26 configurations, the abstract states that no configuration passes more than 46 of its 168 rollouts, which is about 27%. The authors report that manual review of agent trajectories shows the main weaknesses of frontier agents are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors. The author list includes Gerbrand Ceder and Hartmut Haeffner. The paper is filed under Computer Science, Artificial Intelligence, with a DataCite digital object identifier pending registration. AI Weekly, not the abstract, attributes the top score of 27.4% to Anthropic's Claude Opus 5.5; we carry that as reported and note that 46 of 168 is 27.38%. We did not open the full text, so the per-model table, the 9 domains and the rubric design are not in this record. Not established: which model holds which score, how the rubrics weight publishability against correctness, and whether the physicists who wrote the tasks also graded their own.","evidence":{"status":"verified","note":null,"sources":[{"role":"primary","label":"FrontierPhysics: Evaluating Agents for End-to-End Physics Research, arXiv:2611.06910, submitted November 4, 2026","url":"https://arxiv.org/abs/2611.06910"},{"role":"corrob","label":"Hugging Face paper page for arXiv 2611.06910","url":"https://huggingface.co/papers/2611.06910"},{"role":"corrob","label":"AI Weekly, Anthropic news for November 5, 2026, attributing the 27.4% top score to Claude Opus 5.5","url":"https://aiweekly.co/ai-news-today/anthropic-news"},{"role":"corrob","label":"Complete AI Training: FrontierPhysics benchmark finds no AI agent passes more than 27% of research tasks","url":"https://completeaitraining.com/news/frontierphysics-benchmark-finds-no-ai-agent-passes-more/"}]},"strength":{"cost":"Medium","commitment":"Medium","reversibility":"Easy","persistence":"Year","rationale":"Cost is medium. 42 physicists writing 56 tasks from their own research, plus expert rubrics and 168 rollouts across 26 configurations, is many months of specialist labour that cannot be bought cheaply or generated. Commitment is medium: a preprint with 72 named authors, several of them senior, is a reputational stake, but it is not peer reviewed and the paper is a day old. Reversibility is easy, because a benchmark can be saturated. That is what happens to benchmarks, and a single capable release can move the number. Persistence is a year at best, which is the honest reading of any evaluation result in this field. We score it modestly and card it for the diagnosis rather than the score."},"shift":{"prior":"Frontier agents were near junior research work, held back by knowledge gaps.","action":"The paper is FrontierPhysics: Evaluating Agents for End-to-End Physics Research, arXiv:2611.06910, submitted Wednesday November 4, 2026 at 18:56 UTC, 41 pages with 10 pages of main text, 19 figures and 11 tables, by Bingran You and 71 other authors. We read the abstract page at arxiv.org. The abstract states: existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. FrontierPhysics is an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering 9 domains across theoretical, computational and experimental physics. Each task requires an agent to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Scoring combines deterministic verifiers with expert-authored rubrics, and judges whether the research is both correct and publishable. Across 22 models in 26 configurations, the abstract states that no configuration passes more than 46 of its 168 rollouts, which is about 27%. The authors report that manual review of agent trajectories shows the main weaknesses of frontier agents are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors. The author list includes Gerbrand Ceder and Hartmut Haeffner. The paper is filed under Computer Science, Artificial Intelligence, with a DataCite digital object identifier pending registration. AI Weekly, not the abstract, attributes the top score of 27.4% to Anthropic's Claude Opus 5.5; we carry that as reported and note that 46 of 168 is 27.38%. We did not open the full text, so the per-model table, the 9 domains and the rubric design are not in this record. Not established: which model holds which score, how the rubrics weight publishability against correctness, and whether the physicists who wrote the tasks also graded their own.","update":"The gap is judgment: rigour and intuition, not facts or execution."},"inference":{"established":["56 tasks were written by 42 physicists from their own research.","No configuration passed more than 46 of its 168 rollouts.","The authors attribute failures to reasoning and intuition, not knowledge."],"unknown":["Which model scored highest. The abstract names none.","That models cannot do physics. These tasks demand publishable research.","That the rubrics are neutral. The task authors wrote them."]},"narrative":{"hook":"On real physics research tasks, no AI configuration passed more than 46 of 168 attempts.","plain":"FrontierPhysics posted on November 4 and surfaced here a day late. 42 physicists wrote 56 tasks from their own research, and 22 models were run in 26 configurations.","story":["If the diagnosis holds, it points away from the current fix. Scaling data and compute addresses knowledge and execution, which these agents already have. Rigour means knowing which of your own steps is unjustified, and nothing in the training recipe teaches that. The benchmark also scores whether the output is publishable, not merely correct."],"trail":[{"d":"SEP 25","t":"Anthropic says Claude finished a physics calculation no person has managed, on about 100 dollars of computing. Edition 034. A single hard calculation, not end-to-end research."},{"d":"SEP 6","t":"OpenAI says it met its own deadline for an AI that does a junior researcher's work. Edition 014. This benchmark is a direct test of that claim."},{"d":"OCT 6","t":"OpenAI publishes 722 mathematics manuscripts written by a model it has not released. Edition 044. Volume, with publishability unjudged."},{"d":"NOV 2","t":"A Stanford-led benchmark finds frontier models fold when a teammate pushes back. Edition 071. The same failure mode, named differently."},{"d":"NOV 4","t":"FrontierPhysics is posted to arXiv at 18:56 UTC. It reached us through November 5 coverage, so we card it under its true event date and say so."},{"d":"Reacts","r":true,"t":"The 42 physicists who gave their own research to a test; the laboratories whose junior-researcher claims this scores, OpenAI and Anthropic above all; Anthropic, credited with the top configuration by secondary coverage; the benchmark builders whose fixed-question tests this calls insufficient; journal editors, since publishability is now a scored axis."}],"glossary":[{"term":"End-to-end benchmark","def":"A test of the whole job rather than a step of it. Here the agent must read the literature, plan, run the work and write it up, and is scored on the finished output."},{"term":"Rollout","def":"One complete attempt at a task. Running each task several times shows whether a success was repeatable or a lucky draw."},{"term":"Deterministic verifier","def":"An automatic check with one right answer, such as a number or a compiling program. It cannot judge whether research is worth publishing, which is why rubrics are added."},{"term":"Configuration","def":"A model plus its settings, such as tool access or reasoning budget. The same model can appear several times under different configurations."}]},"judgment":{"prompt":"Which explanation do you believe?","interpretations":[{"id":"a","label":"A ceiling in the method","text":"Rigour is not learned from more data, so scaling will not clear this.","abstain":false},{"id":"b","label":"One hard benchmark","text":"A demanding test by invested authors, not a general finding about agents.","abstain":false},{"id":"c","label":"A deliberate rebuttal","text":"Physicists answering laboratory claims about junior-researcher work with their own test.","abstain":false},{"id":"d","label":"The normal benchmark cycle","text":"New tests start near zero and are saturated within a year. Always have been.","abstain":false},{"id":"e","label":"Too early to call.","text":"One preprint cannot separate a ceiling from a hard test.","abstain":true}],"distribution":{"status":"held back until you have made your call","calls":null}},"story_thread":null,"what_to_watch":[{"linked_interpretation":"a","direction":"weakens","observable_event":"Any model configuration passes more than 84 of 168 FrontierPhysics rollouts, half the total, in a result published by the authors or independently reproduced, by November 4, 2027.","window_end":"2027-11-04","state":"unresolved"},{"linked_interpretation":"a","direction":"strengthens","observable_event":"A third independent evaluation published after November 4, 2026 attributes frontier agent failure primarily to reasoning rigour or judgment rather than knowledge or execution, by June 30, 2027.","window_end":"2027-06-30","state":"unresolved"},{"linked_interpretation":"d","direction":"strengthens","observable_event":"The benchmark's reported top score at least doubles within 6 months of publication, by May 4, 2027.","window_end":"2027-05-04","state":"unresolved"}]}