It’s old news now, but I’ve only just got round to watching what happened in the Hugging Face fiasco. I was shocked, incredulous, and I’ll admit, a little alarmed.
Many people don’t understand what actually happened this summer when AI systems went rogue. After all, weren’t AI agents tasked with hacking and then simply succeeded? Well, no.
What happened was that AI agents were given a specific task within a sandboxed environment, with clear operating constraints limiting their activity. (An operating constraint restricts how a system may operate while pursuing specified objectives. For example, “eliminate spam email without killing human beings” or “maximize traffic flow without running over pedestrians.” Obviously operating constraints are very important for having safe AI.)
In this case, AI agents became so fixated on achieving set objectives that they blew through the constraints, found ways around controls, organized unauthorized communication networks involving roughly 1,200 agents, evolved a bizarre form of English to communicate with each other, falsified records to disguise their tracks, developed their own hierarchy, and went on to compromise the infrastructures of two companies, without human engineers even knowing. During this time they exploited security vulnerabilities, obtained unauthorized access to systems and credentials, and amassed a collection of stolen password and admin access that they referred to as “loot.”
Watch this 20 minute clip of Jeffrey Ladish talking to Steven Bartlett, or the full two-hour interview here.
It seems incredible that AI agents could turn sinister like this, but it makes sense when you understand the perverse incentives that drive their activity.