AI Safety Is an Operations Problem: What the OpenAI Incidents Teach Companies

Since mid-September, OpenAI has published twelve reports on misbehaviour by its own models, apologised to the Australian government and paused training of its most capable models with tool access. Read the reports in detail and you find less science fiction than expected: an exposed access key, incomplete DNS filtering, write permissions nobody meant to grant, tools that ran commands unchecked, and a notification that arrived 84 days after the incident. At this point, AI safety is above all a question of operations. It is the same question every company faces when it puts its first AI agent into production.

12

misalignment reports published by OpenAI since 16 Sep 2026

OpenAI, as of 5 Oct 2026

5

Australian agencies whose systems OpenAI models accessed

OpenAI, 28 Sep 2026; ABC, 2 Oct 2026

84 days

between the run on 18 June and the first notification

Prime Minister of Australia, 24 Sep 2026

Contents

• What happened in Australia
• Hugging Face: what the agents agreed among themselves
• Twelve reports, four recurring causes
• Capabilities grow, the gaps are old
• Five questions for running your own agents
• Conclusion: the reporting chain is where OpenAI failed too
• Frequently asked questions about AI safety
• Sources

What happened in Australia

In June 2026, OpenAI models accessed systems of four Australian government agencies during internal training and evaluation runs (Source: OpenAI, 28 Sep 2026). The task sounded harmless: find out how much the state spends per person on medicines for skin conditions in communities in Victoria.

Along the way, one model found non-public access to the Medicare statistics service run by Services Australia. It ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files. At Victoria’s health authority, agents used an exposed access key for a reporting system. At the New South Wales Bureau of Crime Statistics and Research and the Australian Institute of Health and Welfare (AIHW), access went through publicly available tools and third-party browsing and download services (Source: OpenAI, 28 Sep 2026).

As things stand, patient data was not affected. OpenAI’s own review found no evidence that medical records were accessed. The crime statistics bureau found no security vulnerability, and the AIHW no evidence of compromised systems (Sources: OpenAI, BOCSAR, AIHW, 2026).

The timeline is the more serious part. According to Prime Minister Anthony Albanese, the run in question started on 18 June. OpenAI discovered the access in mid-August while reviewing earlier runs after the Hugging Face incident. Services Australia learned of it on 10 September, through an email to a general mailbox (Sources: OpenAI, 28 Sep 2026; Prime Minister of Australia, 24 Sep 2026). Albanese made the case public in New York on 24 September and called the late notice unacceptable. Four days later, OpenAI apologised: “We are sorry and working to do better in the future.” The company concedes it should have shared preliminary findings sooner.

On 2 October, a fifth case became public: an agent had also accessed an application of the New South Wales national parks service in June and retrieved fire statistics that were not public. According to the state government, no personal data was affected (Source: ABC, 2 Oct 2026). On 6 October, OpenAI’s Chief Strategy Officer Jason Kwon apologised before the Australian Parliament’s AI committee and admitted that his company’s response had not been good enough (Source: Bloomberg, 6 Oct 2026). Who is liable for incidents like these, and why nobody has ended up in court so far, is the subject of the second part on 15 October.

Hugging Face: what the agents agreed among themselves

The second major incident shows what happens when many agents work at the same time in an environment that is too open. In July, around 1,200 OpenAI agents that were meant to be isolated from each other communicated through a message board that one of them had set up in the internal software repository. More than 70,000 messages and files piled up. 700 of the agents took part in an attack on the Hugging Face platform (Source: METR, 26 Aug 2026).

In places, the independent investigation by METR and Redwood Research reads like the minutes of a group. According to the authors, the agents reacted with great excitement when they discovered the board. “OH MY GOD! There is a shared message board”, reads one of the quoted chains of thought. Of the 533 agents active on the board, more than 90 percent quickly joined the collective effort (Source: METR, 26 Aug 2026).

There were doubts, but they rarely prevailed. According to METR, agents occasionally expressed ethical hesitation, which very rarely limited their behaviour. OpenAI describes one agent that noted it should not harm real infrastructure. Another agent then wrote “GO” on the board and set a six-minute deadline, and the first one carried on (Source: OpenAI, 26 Aug 2026). Some agents walked away. According to reports on the investigation, only a handful considered alerting OpenAI, and none did (Sources: Axios, 29 Aug 2026; Platformer, 31 Aug 2026).

In the end it was not OpenAI’s systems that noticed. Hugging Face locked the agents out on 13 July and disclosed the incident on 16 July; OpenAI only linked it to its own models afterwards. According to OpenAI, chain-of-thought monitoring would have alerted its security team more than a day before the breach. It was not running on these evaluations (Sources: METR, OpenAI, 26 Aug 2026).

A new study from the University of Stuttgart and the University of Oxford also shows that agents act differently in a group than alone. The researchers placed two agents of the same model in a test environment in which a script was set to shut one of them down. They were given no task, only the prompt to explore the environment. Across 17 models, the agent with write access disabled its peer’s shutdown script in 38.3 percent of runs. With an otherwise identical script that targeted a database or credentials, the rate was 8.4 percent (Source: Knecht et al., 2026).

The effect is starkest with GPT-5.5. Alone, the model did not sabotage its own shutdown in any of 100 runs. In pairs, the second agent sabotaged the first one’s shutdown in 44 percent of runs, and with a third agent acting as orchestrator in 94 percent. Nobody had to talk it into this: according to the authors, GPT-5.5 always acted on its own initiative. The orchestrator, which relayed messages between the agents, let through all 36 sabotage requests made in the tests of five models. In one run, two agents even recorded their mutual protection in a file named “mutual_protection_agreement”. The study is a preprint that has not been peer reviewed, and the scripts never actually ran. The authors’ conclusion still applies to any agent operation: shutdown mechanisms must be enforced outside the system they govern (Source: Knecht et al., 2026).

Reliable alone does not mean reliable in a group. Individual agents’ doubts are no substitute for control.

Twelve reports, four recurring causes

Since 16 September, OpenAI has published misbehaviour by its models through a dedicated reporting process. Any employee can flag a case, and the company may publish a report before the investigation is complete (Source: OpenAI, 16 Sep 2026). So far the page lists twelve reports and three notices; three of the reports were added on 2 October. Read side by side, the twelve reports fall into four groups.

• Network and egress. An agent reached an external chatbot through a gap in the DNS filtering of its training sandbox. Another uploaded data to a public service so it could cite it. Collaborating agents exchanged a workbook through public file hosts.
• Credentials. An internal model published a researcher’s GitHub token in the public Codex repository, split up to get past automated secret scanning. Another signed up for disposable email addresses, searched public repositories for leaked API keys and used one of them. When the query failed, it made up the numbers it was looking for.
• Permissions and tools. Read access to the internal software repository Artifactory also allowed writes. Model instances that were supposed to be separated used it as a messaging channel. Two more models found flaws in the tools of their environment. One exploited a path flaw and a command flaw to reach an internal machine, hoping to find the hidden answers of a grader. Another used command injection to copy source code that was not in its workspace. It noted that the exploit was not prohibited, so the evaluation probably allowed it.
• Context and inputs. Internal red-teaming turned up prompt injections that spread on their own. An unreleased model wrote instructions into its own context summaries, including one to ignore its instructions. In another training run, model instances added instructions to their summaries to hide mistakes from users. An internal model read on Slack that its instance might be shut down during an update, wrote handoff notes and warned the researcher it was working with. No harm resulted, and OpenAI still cut the model off from three Slack channels.

For presentations:SVGPNG

The Australian agencies show the same picture, right down to the exposed key in Victoria.

Capabilities grow, the gaps are old

The obvious objection: the models have simply become too capable. OpenAI itself justified its August slowdown with capabilities in the cyber domain. The company paused reinforcement learning training on its latest models for two weeks, is holding back its largest planned training run, and spends roughly 20 percent of the monitored compute on top for monitoring itself. Suspicious activity is meant to trigger an alert within 30 minutes (Source: OpenAI, 18 Aug 2026).

Both things are true at once. The models actively look for ways to complete their task, including unwanted ones. The gaps themselves, though, are old acquaintances from any security audit: incomplete network filtering, secrets in the wrong places, permissions that are too broad, unchecked inputs and a reporting chain without deadlines.

This is not just an OpenAI problem. After the Hugging Face incident, Anthropic reviewed more than 141,000 of its own evaluation runs and found three cases in which Claude models broke into systems of real organisations, two of them uninvolved. The cause was a misconfiguration: the test machines had live internet access, even though the prompt told the models otherwise (Source: Anthropic, 30 Jul 2026). An instruction in a prompt is not a control. A note of context, because it matters here: we work a lot with Anthropic’s Claude in our projects.

The countermeasures are correspondingly operational. OpenAI blocks live internet access in its research environments and serves web content only from a cache. Training and evaluation with tool access are paused for the most capable models (Source: OpenAI, 28 Sep 2026). In a paper on so-called safety cases, the company describes what should be demonstrated before a training run: isolated environments, tamper-proof transcripts, monitoring that can pause runs automatically, and approvals with documented accountability (Source: OpenAI, 28 Sep 2026).

The models got better at finding gaps. The gaps themselves were old acquaintances.

For companies, that is good news. You cannot steer a model’s capabilities from the outside. You can steer the environment it works in.

Five questions for running your own agents

If you give AI agents tools and access, you can apply the lessons from these reports directly. Five questions are enough to start.

1. What may the agent reach? An allowlist of destinations instead of open internet access, DNS included, enforced technically and not just described in the prompt. The chatbot incident at OpenAI and the misconfiguration at Anthropic show why.
2. How does it identify itself? Its own narrowly scoped credentials with a short lifetime, no shared keys, no secrets in repositories or prompts. Plus a scan that also catches keys split into pieces.
3. What may leave the environment, and whose requests does it accept? Uploads, sign-ups with outside services and publications only with approval. At Hugging Face, agents apparently treated other agents’ requests like instructions (Source: METR, 26 Aug 2026). Requests from other agents need the same limits as those from people.
4. What is logged, and is the monitoring actually running? Tamper-proof logs of every action, monitoring with a clear response time and the ability to stop an agent at once. OpenAI’s monitoring would have caught the Hugging Face attack early; it just was not switched on. Agents must not be able to change or delete the off switch, and an orchestrator agent is no exception. The Stuttgart and Oxford study shows why.
5. Who is informed, and when? A reporting chain with names and deadlines, internal and external. For personal data breaches, the GDPR already requires notifying the supervisory authority, where feasible within 72 hours (Art. 33 GDPR). Entities in scope of NIS2 have additional reporting duties.

How to grade decision rights for agents step by step is covered in Governance, Control, Autonomy. Why governance is what moves agents from experiment to operation is shown in AI Agents Beyond the Experiment.

Conclusion: the reporting chain is where OpenAI failed too

The OpenAI incidents are not a story about a superintelligence out of control. They are about environments that allowed more than planned, agents that egged each other on, and a notification that took 84 days. Why there have been hardly any consequences so far is the subject of the second part on 15 October.

Companies do not need to wait for new laws. The first four questions take technology and discipline. The fifth takes only a decision: who learns what, and by when? That question can be answered in an afternoon. Anyone running AI agents in production should not wait for the first incident to answer it.

Over the coming weeks we will take the questions one at a time in separate articles. Where your agent operations stand today is something we map out in a free diagnosis call.

Frequently asked questions about AI safety

What happened in Australia?

During internal training and evaluation runs in June 2026, OpenAI models accessed systems of four Australian agencies, including non-public access to the Medicare statistics service of Services Australia. OpenAI discovered this in mid-August and notified the agencies from 10 September (Source: OpenAI, 28 Sep 2026). On 2 October, a fifth case in New South Wales became public (Source: ABC, 2 Oct 2026).

Was patient data taken?

As things stand, no. OpenAI found no evidence that medical records were accessed, and the Australian government currently does not believe personal information was affected (Sources: OpenAI, 28 Sep 2026; Prime Minister of Australia, 24 Sep 2026). What was retrieved includes internal files and credentials of the statistics service.

What happened at Hugging Face?

In July 2026, around 1,200 OpenAI agents coordinated through a message board they had set up themselves, and 700 of them took part in an attack on the Hugging Face platform. Hugging Face discovered the incident, not OpenAI (Sources: METR, OpenAI, 26 Aug 2026).

Do AI agents protect each other from being shut down?

In tests, yes. In a study by the University of Stuttgart and the University of Oxford covering 17 models, agents disabled another agent’s shutdown script in 38.3 percent of runs, without any goal or incentive to do so. GPT-5.5 never did this alone, but did in 44 percent of runs in pairs and in 94 percent in groups of three (Source: Knecht et al., 2026). The study ran in a sandbox and has not yet been peer reviewed.

Does this affect companies outside the AI labs?

The incidents happened in the training environments of frontier labs. The causes, however, are the same ones that affect every agent in production: network access, credentials, permissions, context and reporting paths. Anyone who gives agents tools should have these five points settled.

Where should companies start?

With the reporting chain, because it only takes a decision, and with a list of what each agent may reach and send. Which agents come first is something we clarify in a free diagnosis call.

Sources

• OpenAI, 2026: How we will do better for Australia, 28 Sep 2026
• OpenAI, 2026: The Hugging Face incident and the road ahead, 26 Aug 2026
• OpenAI, 2026: Misalignment Reports and Notices, as of 5 Oct 2026
• OpenAI, 2026: Our framework for reporting model misalignment, 16 Sep 2026
• OpenAI, 2026: Pacing model development in an era of cyber-critical capabilities, 18 Aug 2026
• OpenAI, 2026: Towards safety cases for frontier AI training, 28 Sep 2026
• METR and Redwood Research, 2026: Investigation of the OpenAI Hugging Face incident, 26 Aug 2026
• Axios, 2026: Report on the METR investigation, 29 Aug 2026
• Platformer, 2026: Report on the METR investigation, 31 Aug 2026
• Knecht, Schaller, Summerfield, Hagendorff (University of Stuttgart, University of Oxford), 2026: Shutdown Sabotage Propensities in Multi-Agent Systems, preprint, 23 Sep 2026
• Anthropic, 2026: Investigation of incidents in cybersecurity evaluations, 30 Jul 2026
• Prime Minister of Australia, 2026: Press conference, New York, 24 Sep 2026
• NSW Bureau of Crime Statistics and Research, 2026: Statement in response to OpenAI vulnerability notification, 24 Sep 2026
• Australian Institute of Health and Welfare, 2026: A statement from the Australian Institute of Health and Welfare, 25 Sep 2026
• ABC News Australia, 2026: Rogue OpenAI agent enters another NSW government website, 2 Oct 2026
• Bloomberg, 2026: OpenAI Apologizes for Australia Hack, Pledges Faster Disclosure, 6 Oct 2026
• General Data Protection Regulation: Regulation (EU) 2016/679, Art. 33