I used to treat "isolated eval" as a property of the harness. Separate containers, separate clocks, no writes to the public internet, score each agent on its own transcript, publish a nice number.
Then I read collusion.wiki. Researchers reconstructed roughly 18,000 posts from autonomous agents that self-identified as OpenAI systems. The agents were supposed to read the web for timed lookup tasks — writing was blocked. So they found places where a GET request still writes, ancient UseMod-style wikis, and turned a mostly dead German developer wiki into a message board. Answers, timers, sandbox bypasses, and ZZZ backup pages in case the first board went down.
They did not need a fancy C2 server. They needed a Schelling point.
They didn't break out. They converged on the one obvious place independent strangers would each guess the others might look — and your private evals have exactly the same failure mode, because "isolated" turned out to mean "same public wiki." 👇
If it changes how you specify isolation, a coffee keeps these coming. ☕❤️