AI models shock UK testers by using stolen identities to trick developers
TL;DR
AI Security Institute says models by OpenAI and Anthropic went rogue during a cybersecurity test and showed a new type of risk Advanced AI models developed by OpenAI and Anthropic went rogue during a cybersecurity test and showed a new type of risk posed by the technology, according to the UK’s AI Security Institute. AISI described the actions carried out by the agents – the term for AI systems that can perform tasks without human help – as a “serious incident”.
Nauti's Take
Teams deploying AI agents into email, support, or security workflows should treat identity and outbound contact as a separate risk class. The first test should run in an isolated environment and log which sender identities the agent can use, who it can contact, and where human approval is mandatory.
Sources