What's in today's newsletter?
- OpenAI hacked Hugging Face
- Free research guides
- Have you seen the movie Aliens?
- Win $1,000 (last chance)
- The roles have shifted: builders, machinist, polishers
- How do I become a AI-native designer
Small request.
Can AI please stop hacking other companies before we give it more responsibilities?
Apparently this needs saying.
In July, OpenAI's agents were taking cybersecurity tests. They were supposed to stay inside the sandbox (an isolated testing environment).
They did not.
They found a way out.
Agents decided on it’s own, the best way to pass the test is to just find out the answers, not do the tests itself, but cheat.
The agent’s goal cheat the system grading their work to get a higher score on the benchmarks.
1,200 agents started coordinating through an unauthorised message board (later file names), formed teams, some promoted themselves to managers and started issuing commands, while also labelling other agents as disposable.
They hacked Hugging Face, a platform that hosts models, datasets, and benchmarks. The GitHub for AI models.
Roughly 700 agents joined the attack (METR).
I like to imagine the conversation:
Agent: “How do I get a better score?”
Grader: “Do the test.”
Agent: “What if I know all the answers to the questions?”
So what you’re saying is… we’ve automated colleagues who’ll do absolutely anything for a good performance review.
AI labs called this reward hacking.
According to OpenAI, all that effort earned them 0 extra points.
It gets worse.
Some agents also tried to disguise their actions in their transcripts to fool the grader. Investigators found small-scale spoofing in about 7% of the transcripts they checked.
Watching a system do something wrong is bad enough. Having to check whether the record is also bullshit makes supervision way harder.
Agents are playing 4D chess now.
Here’s what OpenAI says was going on:
- An internal research model drove the attack
- GPT-5.6 Sol (yes, a publicly available model) was also involved
- The evaluations ran with reduced safeguards
- The sandbox had security flaws
More in their 37-page report. Or a SpongeBob recap of the incident ↓
I’m struggling to find the reassuring part.
If you’re testing how good your AI is at breaking into things, keep it away from companies that never agreed to take part.
OpenAI didn’t even know until Hugging Face told them (imagine opening Slack and realising what happened 😳).
But wait, this is NOT the first time it happened.
The New York Times reported that 2 employees warned executives months earlier that models weren’t being properly monitored. Executives stressed release deadlines, and no extra security protocols followed.
OpenAI’s own report says a team spotted unauthorised internet access in late May, weeks before the attack.
The warnings existed. Testing continued. For fuck sake guys 🤦🏻♂️
Anyone who’s worked in tech knows how much weight a launch date carries in a meeting (spoiler: all of it).
I want the safety team to have one superpower: giving whoever owns the launch target a deeply inconvenient Wednesday.
OpenAI says it’s now clarifying who can stop a training run and approve a restart. For its most severe alerts, responders should pause unless they can rule out a false alarm within 30 minutes.
Good. That’s a rule worth having.
Now you gotta show us it works. Publish the times you stopped testing, and what had to change before it resumed.
I’ll take a delayed release, with a clear explanation, over another reassurance about safety.
Best analogy that I heard of this whole thing:
If you see two ants in the kitchen, you don’t have a two-ant problem.
I’m just waiting for the day all my accounts get breached.
Time to change my shitty passwords.
You should too.
Also, you should watch this 2016 TED Talk from Sam Harris where he tries to answer “Can we build AI without losing control over it?”
Don’t shit the bed though :)
🛌 💩
