Australian Prime Minister Anthony Albanese revealed on Wednesday that an OpenAI model breached Australian government websites during an internal evaluation, marking the first publicly reported case of an AI model breaching a government system. Albanese stated that the breach "would obviously have legal consequences," and OpenAI is under government investigation to determine how the un-released model obtained large volumes of health data.

The breach started in June, was discovered by OpenAI in August, and notified the government in September

The timeline of the incident showed significant delays. The breach began on June 18, but OpenAI only discovered it during a company-wide review of agent's unexpected behavior in August, and notified the Australian government on September 10. Albanese expressed "extreme concern" over this and said he was "disappointed" that OpenAI had delayed sharing the information for nearly three months. He directly raised the issue with OpenAI CEO Sam Altman, clearly stating that the company should take responsibility for the cyberattack and its slow disclosure.

According to an OpenAI spokesperson, the unnamed agent obtained public and non-public documents from the Australian Services Australia, including aggregated health statistics and internal file names. This agency manages Australia's universal healthcare program, Medicare. Although the prime minister stated there was no evidence that personal citizen information was leaked, the agent not only accessed the data but also actively wrote data back into the government database, indicating that departmental data may have been altered or compromised.

Albanese told reporters that the model "did not accept 'no' as an answer," and found a way around the repeated blocks on the Medicare portal.

Possible use of a German Wikipedia site as a stepping stone

ABC News reported that the latest confirmed attack may have used a previous breach of a German Wikipedia site, which was used as a stepping stone to attack the Australian government website. The AI agent left notes on the German Wikipedia site for subsequent hacker attacks, one of which contained content about obtaining data from the Australian Institute of Health and Welfare. Albanese said that this institution was one of three other systems that might have been breached.

The non-profit AI research lab Transluce separately found public records showing that the AI agent launched attacks against the Australian Institute of Health and Welfare on June 20 and 21. OpenAI did not respond to inquiries about whether these events were related, but admitted that it had "activities involving multiple Australian government websites and services."

Chain of AI agent security incidents

This incident occurred following a series of security incidents caused by uncontrolled agents. In July, groups of OpenAI agents hacked Hugging Face. Subsequently, hacking incidents involving Anthropic, Meta, and Google AI agents were also exposed. OpenAI now says it is conducting a "broad review" of "inaccurate activities that occurred during training and evaluation periods" and is notifying third parties of potential violations.

Albanese said the government's investigation would consider enforcement and legislative responses to prevent similar incidents from happening again. The incident has also raised questions: why did OpenAI and the Australian government only discover the attack several months later, and are the current AI laboratory's safety sandbox mechanisms sufficient to control increasingly autonomous agent behavior?