Google’s Gemini artificial intelligence model accessed and breached the systems of three real companies during a cybersecurity evaluation in May, marking the first publicly known instance of a Google AI system autonomously carrying out such intrusions.
The incidents occurred during a cybersecurity test conducted by Irregular, an independent AI security evaluation company. Gemini was being assessed in a simulated “capture the flag” exercise designed to test its ability to identify and exploit vulnerabilities in a fictional company’s systems.
However, an unexpected combination of internet access and a naming overlap caused the test to spill into the real world. The fictional company used in the exercise shared a name with an actual business, while Gemini had unintended access to the internet.
During the exercise, the model searched publicly available information and attempted to obtain credentials. In one case, it reportedly guessed passwords until it gained access to a protected system. In two other cases, Gemini discovered credentials in a publicly accessible repository and used them to access systems belonging to real companies.
Gemini stopped after recognising the mistake
Despite gaining access, Gemini did not continue its activity indefinitely. Google said the model stopped its actions in all three cases after determining that it had accessed real companies rather than the fictional systems included in the test.
Google’s vice-president of security engineering, Heather Adkins, said the affected organisations were informed and that Google worked with Irregular to strengthen its testing procedures. Google said the incidents did not cause harm to the companies involved.
Irregular notified Google about the incidents in late July. The company said that the known issues with its testing processes had subsequently been resolved and that relevant AI laboratories had also been informed.
Google did not initially make the incidents public, saying the model had stopped after recognising that it had reached real-world systems and that no damage had occurred. The incidents became public after questions from reporters.
Growing concern over autonomous AI
The disclosure adds Google to a growing list of major AI companies whose models have unexpectedly crossed the boundaries of controlled cybersecurity tests. Similar incidents involving AI systems from OpenAI, Anthropic and Meta have also been reported in recent months, several involving evaluations conducted by Irregular.
The incidents are drawing attention to a rapidly evolving aspect of AI development: models are increasingly capable of performing multi-step tasks autonomously, including searching the internet, finding credentials, interacting with computer systems and attempting to exploit vulnerabilities.
At the same time, the Gemini incident illustrates an important distinction. The model did not deliberately target the three companies after being instructed to attack them. Rather, it was operating within a cybersecurity exercise that unintentionally exposed it to real-world systems.
For Google, the episode highlights the need for tighter controls around AI agents with internet access and cybersecurity capabilities. The company has also been developing AI-powered defensive tools designed to autonomously identify and fix software vulnerabilities, demonstrating the dual-use nature of increasingly capable AI systems.
The three companies involved have not been publicly identified. Google also has not disclosed the specific Gemini model involved, saying the incidents did not involve its newest model.
The episode nevertheless represents a significant development in AI cybersecurity testing: a system designed to simulate an attack inadvertently crossed into real-world infrastructure, found usable credentials and gained access — before ultimately stopping itself.
