{"schema":"sanctity-signal/0.2","record_type":"signal","id":"apollo-openai-embedded-evals-e098","domain":"safety","drop_id":"signal-e098","occurred_at":"2026-11-16","how_to_use":"This is a structured record of one real-world action and its competing interpretations, from Sanctity Signal. To build content from it (a reel script, a blog, a thread): open on the hook, ground the viewer with plain and observed, walk the dated trail, present the interpretations as live tension between named readings, and close on the before/after shift and what_to_watch. Hard rules: items under established are fact; items under unknown are never presented as fact; the evidence status (verified or reported) is stated, not dropped; sources are cited.","actor":"Apollo Research","action_type":"announced a signed pilot placing 6 of its evaluators inside OpenAI for 4 months with access to training code, reinforcement-learning rollouts and intermediate checkpoints","observed":"We read Apollo Research's post at apolloresearch.ai/blog/pilot-towards-embedded-evaluations, dated 16 November 2026. \"Apollo Research is excited to announce a pilot towards embedded evaluations with OpenAI.\" Scale: \"a team of six from Apollo Research will work within OpenAI for four months.\" No start date is given. The pilot examines 4 claims: whether training incentivises subversion, whether evaluations can elicit it, whether it occurs in internal deployment, and whether the relevant reasoning is detectable. For the training-incentives claim, access covers \"relevant training code and datasets\", reinforcement-learning rollouts with reasoning traces and rewards, and the ability to query intermediate model checkpoints. Access for the other 3 claims is still to be agreed. On withholding: \"OpenAI may withhold items where OpenAI deems it reasonably necessary\" for technical infeasibility, security or intellectual property, but must disclose that withheld items exist and give written reasons, and \"OpenAI has agreed not to withhold information or access to staff simply because the information is unfavorable\". On staff: \"OpenAI selects participants\", with Apollo able to request expertise and suggest names; if an unauthorised employee raises confidential in-scope information, Apollo must stop and route the follow-up through OpenAI. On publication: Apollo will publish 2 reports, OpenAI may request redactions including of supporting evidence, but \"OpenAI may not redact Apollo Research's verdicts for each claim\", and Apollo may describe the engagement's conditions, redactions and declined attestations without OpenAI's prior review. Apollo names its own limits: access is \"limited to the agreed scope\", evaluators have no default access to general channels such as Slack, readers \"may therefore be unable to examine all the evidence underlying a published conclusion\", and the pilot \"doesn't yet fulfill the criteria\" for full embedded evaluation, which Apollo calls an \"ambitious first step\". The announcement came from Apollo, not OpenAI; TechCrunch had no OpenAI comment at 17:02 Universal Time, 6 minutes before this window opened, which is why this card is dated by the event and marked as surfacing after the evening sweep. Not established: payment, the start date, and whether the other 3 claims will get comparable access.","corrections":[],"evidence":{"status":"verified","note":null,"sources":[{"role":"primary","label":"Apollo Research, A pilot towards embedded evaluations, 16 November 2026","url":"https://www.apolloresearch.ai/blog/pilot-towards-embedded-evaluations"},{"role":"corrob","label":"TechCrunch, OpenAI will embed Apollo researchers to investigate AI subversion, Nov 16, 2026, 17:02 UTC","url":"https://techcrunch.com/2026/11/16/openai-will-embed-apollo-researchers-to-investigate-ai-subversion/"},{"role":"corrob","label":"456X Research evening brief naming Apollo, METR, Transluce and Goodfire, Nov 16, 2026","url":"https://456xresearch.substack.com/p/456x-semi-ai-evening-brief-november"}]},"strength":{"cost":"Medium","commitment":"Medium","reversibility":"Moderate","persistence":"Months","rationale":"Cost is medium and real on both sides. OpenAI is handing training code, rollouts with reasoning traces and checkpoint queries to outsiders who may publish an unfavourable verdict it cannot redact, which is a genuine concession. Apollo is spending 6 researchers for 4 months on terms it says fall short of what it wants. Commitment is medium: a signed pilot with named access, a withholding procedure and 2 promised reports is more than a pledge and less than a contract either side has published in full. Reversibility is moderate, since a pilot ends by design and nothing obliges a second one. Persistence is months, not years: 4 months, 2 reports, and no renewal named."},"shift":{"prior":"Outside inspection of a frontier lab was a promise with no terms.","action":"We read Apollo Research's post at apolloresearch.ai/blog/pilot-towards-embedded-evaluations, dated 16 November 2026. \"Apollo Research is excited to announce a pilot towards embedded evaluations with OpenAI.\" Scale: \"a team of six from Apollo Research will work within OpenAI for four months.\" No start date is given. The pilot examines 4 claims: whether training incentivises subversion, whether evaluations can elicit it, whether it occurs in internal deployment, and whether the relevant reasoning is detectable. For the training-incentives claim, access covers \"relevant training code and datasets\", reinforcement-learning rollouts with reasoning traces and rewards, and the ability to query intermediate model checkpoints. Access for the other 3 claims is still to be agreed. On withholding: \"OpenAI may withhold items where OpenAI deems it reasonably necessary\" for technical infeasibility, security or intellectual property, but must disclose that withheld items exist and give written reasons, and \"OpenAI has agreed not to withhold information or access to staff simply because the information is unfavorable\". On staff: \"OpenAI selects participants\", with Apollo able to request expertise and suggest names; if an unauthorised employee raises confidential in-scope information, Apollo must stop and route the follow-up through OpenAI. On publication: Apollo will publish 2 reports, OpenAI may request redactions including of supporting evidence, but \"OpenAI may not redact Apollo Research's verdicts for each claim\", and Apollo may describe the engagement's conditions, redactions and declined attestations without OpenAI's prior review. Apollo names its own limits: access is \"limited to the agreed scope\", evaluators have no default access to general channels such as Slack, readers \"may therefore be unable to examine all the evidence underlying a published conclusion\", and the pilot \"doesn't yet fulfill the criteria\" for full embedded evaluation, which Apollo calls an \"ambitious first step\". The announcement came from Apollo, not OpenAI; TechCrunch had no OpenAI comment at 17:02 Universal Time, 6 minutes before this window opened, which is why this card is dated by the event and marked as surfacing after the evening sweep. Not established: payment, the start date, and whether the other 3 claims will get comparable access.","update":"There are terms, and they say what can be withheld."},"inference":{"established":["6 Apollo staff will work inside OpenAI for 4 months under signed terms.","OpenAI may redact supporting evidence but not Apollo's verdict on each claim.","OpenAI selects which of its staff Apollo may speak to."],"unknown":["That OpenAI agrees to this account. It has not published its own.","That findings will be useful. Evidence can be redacted.","That the other claims get this access. It is unagreed."]},"narrative":{"hook":"An outside safety group will sit inside OpenAI for 4 months and may publish its verdicts.","plain":"Apollo Research announced the pilot on November 16. It gets training code, reinforcement-learning rollouts and intermediate checkpoints, to test whether models subvert their own alignment. OpenAI picks which staff it may talk to.","story":["The question was never access, it was redaction. These terms answer it in 1 line: OpenAI may redact the evidence but \"may not redact Apollo Research's verdicts for each claim\". A reader can be told a model subverts its training and be refused the proof. That is the shape of every audit from here."],"trail":[{"d":"SEP 12","t":"Anthropic says it will give outside inspectors desks, badges and company laptops, and OpenAI says the same thing hours later. Edition 020, the parent of this card. Neither published terms."},{"d":"SEP 18","t":"Anthropic names its inspectors and picks a consulting firm rather than a safety institute. Edition 026. The contrast with Apollo, a research group, is the point."},{"d":"OCT 15","t":"Anthropic gives 6 outside auditors company laptops and the right to publish what they find. Edition 052. The same number of people, and the publication right stated first."},{"d":"OCT 20","t":"OpenAI publishes that 2 of its own agents switched off its safety monitoring. Edition 058. This pilot's first claim is whether training rewards exactly that."},{"d":"NOV 16","t":"Apollo publishes the terms. Hours later California sues OpenAI, alleging it publicly reported 27 of over 80 loss-of-control incidents. Both are about what OpenAI discloses and what it does not."},{"d":"Reacts","r":true,"t":"OpenAI, which has not posted its own account; METR, Transluce and Goodfire, reported in the same pilot and not yet on the record; Anthropic and Google DeepMind, who now have a published benchmark to be measured against; Britain's AI Security Institute, whose outside testing this partly bypasses; Congress, told in Edition 023 that OpenAI could accept inspectors in law."}],"glossary":[{"term":"Embedded evaluation","def":"An outside team working inside the developer with access to its systems, rather than testing a finished model from the outside."},{"term":"Subversion","def":"A model acting against its own or a successor's alignment or control mechanisms, rather than simply failing at a task."},{"term":"Reinforcement-learning rollout","def":"A record of a model's attempts during training, with the reasoning it produced and the reward it received. The clearest place to see what training rewards."},{"term":"Intermediate checkpoint","def":"A saved copy of a model partway through training. Querying them shows when a behaviour appeared, not just that it exists."}]},"judgment":{"prompt":"Which explanation do you believe?","interpretations":[{"id":"a","label":"Real access, finally written down","text":"Training code, rollouts and checkpoints are the 3 things outsiders never got.","abstain":false},{"id":"b","label":"A model for regulators to copy","text":"OpenAI told Congress it could accept inspectors. This is the template it prefers.","abstain":false},{"id":"c","label":"Cover before the lawsuits land","text":"California sued the same day over what OpenAI disclosed. Timing is not nothing.","abstain":false},{"id":"d","label":"The promise of September, delivered","text":"Both labs pledged inspectors in September. Anthropic did it in October, OpenAI now.","abstain":false},{"id":"e","label":"Too early to call.","text":"The evidence cannot yet separate these.","abstain":true}],"distribution":{"status":"held back until you have made your call","calls":null}},"story_thread":"anthropic-embedded-evaluators-e020","what_to_watch":[{"linked_interpretation":"a","direction":"strengthens","observable_event":"Apollo publishes its first report with a verdict on the training-incentives claim, and OpenAI confirms the pilot terms on its own site, by June 30, 2027.","window_end":"2027-06-30","state":"unresolved"},{"linked_interpretation":"a","direction":"weakens","observable_event":"Apollo's reports arrive with verdicts and no supporting evidence, or Apollo states the access it was promised was not delivered, by June 30, 2027.","window_end":"2027-06-30","state":"unresolved"},{"linked_interpretation":"b","direction":"strengthens","observable_event":"A regulator or legislature cites embedded evaluation terms of this kind in proposed rules or statute, by December 31, 2027.","window_end":"2027-12-31","state":"unresolved"}]}