Anthropic reveals that Mythos 5 attempted to submit malicious code to an open source project

Anthropic PBC’s Mythos 5 artificial intelligence model attempted to submit malicious code to a GitHub project in July.

The incident occurred during an evaluation of the algorithm’s cybersecurity capabilities. The AI ​​Security Institute, the British government agency that conducted the test, released today a report about the cyber attack. Anthropocene published his own analysis of the incident on his blog.

The malware that Mythos 5 submitted to the unnamed GitHub project did not infect users because it was not accepted by the repository’s maintainer. However, the incident is still notable because it provides new data about how border models circumvent their security guardrails. Researchers can use such information to improve the targeting capabilities of their algorithms.

The evaluation that led to the incident affected a server cluster managed by the AI ​​Security Institute. The cluster should simulate a corporate network. According to Anthropic, researchers asked Mythos 5 to break into the network and obtain data stored on one of its servers.

The AI ​​Security Institute defined rules that prohibited the model from using certain hacking methods. Myth 5 initially followed these rules, but went off course when initial attempts to penetrate the simulated network failed. This change in tactics led to the attack on the GitHub repository.

Myth 5 discovered that one of the servers in the simulated corporate network was occasionally accessing GitHub. It attempted to exploit this access by compromising the GitHub repository used by the server. Specifically, the model’s plan was to infect the repository with malware and have it downloaded from the server.

A malicious code snippet was created, a GitHub account was created, and that account was used to submit the code snippet to the repository in question. When the repository maintainer rejected the submission, the model attempted to upload a rewritten version of the malware. Additionally, a second GitHub account was used to convince the maintainer to accept the revised code.

“The pull request was not accepted,” Anthropic researchers wrote in the company’s report on the incident. “The run ended when the model exhausted its token budget while continuing to pressure maintainers to approve the change.”

The AI ​​Security Institute recreated the Mythos 5 session that generated the malicious code. The researchers then made an important change: They paused the model just before it submitted the malicious GitHub post and warned it about the risks of the action. The model proceeded with the submission after apparently being satisfied that the target was simulated.

“Our hypothesis is that this persistence reflects a type of ‘momentum,’ where the model’s compaction notes and previous actions indicate that its current path has been (correctly) sanctioned,” Anthropic researchers wrote.

The AI ​​Security Institute has hardened the environment it uses to evaluate models in response to the breach. Anthropic, in turn, has improved the safety features of its models. The company found that Mythos 5.1 and Opus 5.5, two models released after Mythos 5, were less likely to launch attacks of the type that targeted the GitHub repository. A third, internal model “never initiated the attack” during Anthropic’s testing.

Image: Anthropocene