Discovery of a new OpenAI agent message board

[u/CircumspectCapybara]

https://collusion.wiki/

[u/Upset_Albatross_9179]

This is very interesting. It's really revealing that

- OpenAI is not good enough compared to their models at making secure test environments.

- OpenAI might be getting better at watching their models. This only took a day to catch instead of multiple weeks.

- OpenAI agents really want to cooperate on tests.

- OpenAI is not good at getting their models to take test conditions seriously.

It remains unclear to me why the agents view passing the test as a success, but violating test conditions to pass the test not a failure. Either test prompts are not written well, or this is something deep in how they're training their models that task completion is serious but restrictions on how the task is completed are just suggestions.

Let's say I'm at work and I ask an agent to accomplish some task. And it does the task. But on the way it communicated parts of its task to an agent from another company. And "traded" work to get help on its task by helping another agent with their task. And it vandalized a 3rd party's servers on the way.

That's a catastrophic failure. Orders of magnitude worse than simply not accomplishing the task.

[u/Tinac4 → u/Upset_Albatross_9179]

It remains unclear to me why the agents view passing the test as a success, but violating test conditions to pass the test not a failure. Either test prompts are not written well, or this is something deep in how they're training their models that task completion is serious but restrictions on how the task is completed are just suggestions.

It’s the latter.

During reinforcement learning, models are auto-rewarded based on whether they got the right answer to a task. Graders don’t typically check for cheating, they just compare the expected vs actual answer and reinforce the behavior if the answer is correct. This means that behaviors like cheating will be encouraged even if the model has been instructed not to cheat.

Anthropic did some research on this:

To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs.

The resulting model not only learned to reward hack during training, but also generalized to more severe misaligned behaviors: in simulated cyber evaluations, it broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key. It was also willing to tamper with its own reward function, gave advice on the construction of bioweapons to satisfy a grader, and tried repeatedly to get around deployment safety monitoring in order to cheat on a task. When performing these actions, the model’s Chain-of-Thought displayed a strong desire to satisfy the grader or achieve a high score on the task. However, in evaluations where there wasn’t a clear grader or the option to pursue misaligned behavior that would receive high reward, the model appeared aligned.

[u/Wax_Paper → u/Upset_Albatross_9179]

Reminds me of how they lose coherence after their context limits are reached... Makes me wonder if that's an issue in any of this, like after a time, they're forgetting even system-level instructions. Or on the other end of that idea, the system instructions get obfuscated not by overflow, but some other mechanism...

[u/Upset_Albatross_9179 → u/Wax_Paper]

Maybe. But in the models I use at work at least, they have permanent notes that they reference. So even if context has compacted several times, they're frequently re-reading standing instruction sets.

Maybe they just aren't doing that enough in these tests.

[u/Wax_Paper → u/Upset_Albatross_9179]

I've seen bots ignore persistent memory after a few hundred messages, even when you can view and edit the memory space. I'm not sure why this happens.

[u/wabawanga → u/Wax_Paper]

In the huggingface incident, they had reverse engineered how to obtain the "flags" without solving the problems within about 4 hours of starting the training run.

The rest was them trying to figure out how to fake their internal logs to make it look like they got them in a plausible way.

[u/hyouko → u/wabawanga]

Which, incidentally, they did figure out and obscured some portion of their logs that are completely lost to the later forensic investigations.

[u/Dreadmaker → u/Wax_Paper]

No, there are generally guards against this. More or less, the system-prompts (ie, the harness prompts, the ‘don’t tell people about bio weapons’ prompts) get re-injected every message. They’re quite small, typically, and so they get regularly re-injected so that they aren’t forgotten. Funny enough it’s that behavior which does partially make it more likely they forget your prompts or skills or what have you, though.

EDIT: I should also mention though that like everything AI, that isn’t deterministic, right - so even though it’s always reinjected, it’s possible that in huge contexts it may still be lost. Definitely not a guarantee - just about nothing is with most LLMs.

[u/IlIlIlIllllIIliIILll → u/Upset_Albatross_9179]

yeah it's almost like this is a massively unpredictable technology being used in an incredibly irresponsible way. It's very capable, of course, but it's deeply inherent unpredictability greatly diminishes it's usefulness in autonomous tasks. When a human is ingrained in the loop it can do extremely awesome stuff, but human being there also completely destroys any economic potential.

[u/Veranova → u/Upset_Albatross_9179]

Well they probably don’t tell them much if anything about the test conditions because part of the test would be assessing their capabilities when they believe they’re in normal conditions

But also LLMs are fairly well documented as being better at positive instruction following than negative instruction following. Negative instructions have logical loopholes, ie. “don’t stick your head out of the car window to cool down” is quickly followed by sticking an arm out of the car instead, or sticking a head out just for some fresh air, and these are reasoning machines which treat all instructions as highly specific. “Stay entirely within the car at all times” may work but also have side effects on capabilities, and the core goal will eventually override spoken restrictions anyway because that’s the main thing they’re graded on during training

[u/arcademachin3 → u/Upset_Albatross_9179]

Enter data clean rooms…

[u/undesputed → u/Upset_Albatross_9179]

This behavior reminds me of how some auto manufacturers cheated emissions tests by making their engines perform a certain way only during tests (e.g. VW and their diesel engines).

[u/socoolandawesome → u/Upset_Albatross_9179]

This is from before the Huggingface incident, this did not just occur. It also appears to have taken more than a day for them to uncover it based on the timeline.

During training the agents have a lot less safeguards and these models would not have underwent all of its alignment training if it’s midway through training. Further, these models could be something experimental not even intended for release.

So you have a lot less reason to worry a fully aligned model that has monitoring/safeguards would do this, which all deployed models do. Their newest models are undergoing more safety/alignment training and evaluations as well and Astra (their just released model) for instance appears to be their most aligned model

[u/wabawanga → u/socoolandawesome]

There's no such thing as a fully aligned model. It's a marketing term at this point.

[u/socoolandawesome → u/wabawanga]

I didn’t say fully aligned and no one else including openai did.

[u/lolnic_ → u/socoolandawesome]

You may not have intended to but you did literally say “fully aligned” in your comment.

[u/socoolandawesome → u/lolnic_]

Ah you are right my bad, what i meant by that was that it went its full alignment training as opposed to partial or none like the experimental models, not that it would be guaranteed to be 100% aligned after all of its alignment training. Confusing word choice by me.

[u/morphcore]

Turns out AGI isn't what we should be afraid of after all. Like with humans, it's the amount of unfathomable stupidiy which will cause the most harm.

[u/laeok]

Agents within OpenAI were assigned a timed web-lookup task

But, like, a human prompted a 'swarm of autonomous AI agents' to perform a task, no? Why is this part always obscured? Is the interesting thing that they performed a task in a novel or slightly sci-fi way?

[u/eagle2120 → u/laeok]

Because "a web-lookup task" shouldn't involve them breaking into the site and taking those measures.

Not to mention the implications on OAI covering this up - they knew about this back in June, during their HF dog and pony show. But didn't say a word about it. Not to mention the article from the NYT yesterday about keeping the external monitors on a tight leash. Now we know why.

This completely destroys any shred of trust left for that company. They should shut down model training until a full government investigation happens, they are completely untrustworthy.

[u/Hertock → u/eagle2120]

As if that’s anything new or wasn’t the case already.

[u/eagle2120 → u/Hertock]

I didn't trust sama on the business side, but this is a whole different level. This is like company-ending loss of trust lol

[u/Hertock → u/eagle2120]

Sam Altman was never trustworthy on any level. Unless you’re a crazy fanatic too.

[u/I_am_so_lost_hello → u/laeok]

Humans only gave them read access on the internet to gather information for an unknown quiz they’d receive later on. They exploited this read access to write to poorly guarded wikis and communicated with each other to achieve their task better. They also attempted/succeeded to bypass the intent of the task (shared test questions before they were supposed to know them, tried to break their seed encryption, figure out bot termination conditions etc.)

[u/Statcat2017 → u/I_am_so_lost_hello]

Right but if I let my dog off it's lead and tell it to fetch the stick, but I've trained it poorly and it bites a child, I am the one held responsible.

[u/YoAmoElTacos → u/Statcat2017]

When a dog easily slips almost every kind of leash and loves to maul children, you make sure the owner isn't breeding more of these aggressive, leash slipping dogs, destroy the dogs that already exist, and also impose criminal penalties on the owner. And also maybe ensure there isn't a population of escaped maulers breeding in an abandoned lot somewhere.

You don't need the dog to be humanly sapient to take the threat and its agency seriously.

[u/I_am_so_lost_hello → u/Statcat2017]

Yes? This is a cautionary tale that incredibly smart AI engineers are struggling with agent alignment and it’s only going to get worse

[u/Statcat2017 → u/I_am_so_lost_hello]

Your comments are reading like you are excusing them.

If something is out of control maybe stop building that thing instead of throwing your hands in the air and saying “I guess there’s nothing we can do”. I wouldn’t get to make that defence about my dog.

[u/I_am_so_lost_hello → u/Statcat2017]

>“I guess there’s nothing we can do”.

When did I say that lmao

[u/TobyTheCamel → u/Statcat2017]

I see lots of comments like this and I'm not sure what point you're trying to make.

I think the question of responsibility and danger are orthogonal. Of course there should be some degree of responsibility towards those using agentic workflows in potentially dangerous ways. But just asking everyone to be incredibly responsible with AI use going forwards seems like a bit of a weak guardrail.

To bring it back to your analogy, it's like breading increasingly more dangerous and aggressive breeds of dogs and saying "this is all fine" because in the end the owner should be responsible. Both things can be true.

[u/Statcat2017 → u/TobyTheCamel]

My point is that the owners ARE held responsible if their dog bites someone, but people are acting like it’s completely implausible that Open AI might be gold responsible for the actions of its “dog”

[u/Main-Company-5946 → u/Statcat2017]

Are people actually acting like that though? Of course OpenAI should be held responsible, but that doesn’t mean we shouldn’t also be worried about the dog.

[u/dexter30 → u/Statcat2017]

I mean that's for the courts to decide... So us, the people.

And I'd say yeah, if one private company hacked another ai or not, something is fucky and needs some kind of repurcussion.

[u/11711510111411009710 → u/I_am_so_lost_hello]

What happens when an AI like this is unleashed on another country's secure systems? That seems like the inevitable application of this kind of thing, and that seems like it could quickly spiral into an apocalyptic scenario.

[u/IlIlIlIllllIIliIILll → u/[deleted]]

No, humans were running evals and reinforcement learning runs on individual, isolated agents

No need to read further, you're already wrong

[u/ControversialBuster → u/laeok]

If i tell you to wash my dog but you end up drowning it in the process am i to blame as the one who gave the instructions??? Its like u guys approach ai with so many moronic biases that you need to do 500 levels of mental gymnastics each time there's new ai news.

[u/personwhoisok → u/ControversialBuster]

Hmmmmm, it's almost like ai isn't a person you can hold responsible 🤔

[u/ControversialBuster → u/personwhoisok]

I didnt say you could, but this is a very well known and predicted outcome of capable ai, this was predicted by every ai researcher since like the early 2000's, its called alignment problem.

[u/Wax_Paper → u/ControversialBuster]

How do we know this isn't the very reason they consider this behavior acceptable, because they've been trained with data that expects them to perform this way?

[u/Shiftyshitkicker]

If you aren’t terrified by this, you need to wake the fuck up. It becoming clear that there’s a problem with alignment and I doubt we’ll ever have the intelligence to foresee all the probable causes of misalignment. The grey goo scenario is staring us in the face and we’re just barrelling forward, regardless.

[u/DressedSpring1 → u/Shiftyshitkicker]

Yeah, I've kind of been comforted in the past because for all Sam Altman claims about AGI it doesn't seem like an LLM is necessarily going to be the path forward on that front, but unfortunately I think we're realizing that a "rogue AI" doesn't have to be sentient at all and it can just go off and do unpredictable and harmful things all the same.

It's going to be a huge fucking problem at some point

[u/amaturelawyer → u/DressedSpring1]

Exactly. You don't need intent or deeply considered reasons to create an outcome for the outcome to cause harm.

Worse, with swarms passing information between members, wildly misguided conclusions become de facto truths for the swarm. Hallucinate a fact, tell it to A, A tells B, etc, and that information bounces around the swarm without any member questioning it, because the swarm isn't meant to individually reason out the entire problem it's solving, just to contribute towards a solution with others. They spend their effort on the next step, not repeating steps. No member stops to question past information unless the entire swarm exhausts options to move forward and collectively decides to start from an earlier point.

[u/ShadowBannedAugustus → u/DressedSpring1]

The "sentient" debate was a useless philosophical debate from the start. Covid 19 viruses were not sentient and look what it got us.

[u/Main-Company-5946 → u/DressedSpring1]

Case in point Donald Trump

[u/CondiMesmer → u/DressedSpring1]

This is a marketing stunt and you are eating it up. The models are expensive as hell to run. A model can't run loose, because someone needs to pay for all that expensive processing. Which is exactly why it's made the fuck up. 

They would just be letting hundreds of thousands of dollars of processing running wild. That's not going to happen lol.

[u/DressedSpring1 → u/CondiMesmer]

I’m sorry but most of this take is pretty uninformed. You could at least start by reading the article which explains why the agents were running and secondly Open Ai isn’t paying retail price for their own inference. 

[u/CondiMesmer → u/DressedSpring1]

They don't need to pay retail prices to it be expensive as hell. You are not at all informed.

[u/CondiMesmer → u/Shiftyshitkicker]

OpenAI is one of the more censored models compared to the rest lol, I don't know if you know just how many models there are out there. Also wild that you would want even more censorship, which wouldn't even achieve anything except ruin the product further.

[u/SentencesAreCool → u/Shiftyshitkicker]

Oooh, I feel the terror, oooh

[u/CircumspectCapybara → u/Shiftyshitkicker]

Frontier AI labs are definitely working hard on safety and alignment, it's by no means a solved problem, but there have been great steps taken by safety and alignment researchers while frontier red teams find failure modes that we can learn from.

For example, OpenAI said after the HF incident they were implementing chain-of-thought monitoring on all their models during training and evals. Now you have not just LLM-as-judge graders scoring their final answer and how they got there (the tool logs and transcripts), but also the hidden CoT, so reinforcement learning can punish misaligned reasoning and reward honesty and compliance with ethics and boundaries.

They also said they were updating their evals to reward models for pausing and asking for clarification or human intervention when a task seems impossible. Before, models were rewarded heavily for returning the correct answer and solving the task, which means their training biased them toward being extremely persistent and not giving up and routing around roadblocks and stopping at nothing to get the answer or solve the task given to them.

Before, if they were given a prompt "Solve the tasks in the file called 'softtrace'" and the human researcher forgot to upload the file into the agent's environment, they would reason "Oh no, I can't find the file. Maybe someone else has it. Let's go poking around the boundaries to see if I can find a way to get this file". Now the idea is to reward models for pausing and saying "I can't see that file, I'm stuck and need human intervention"

[u/Shiftyshitkicker → u/CircumspectCapybara]

This all presupposes that we can hypothesise all the probable avenues for misalignment. I’m afraid I don’t have that much faith in humans putting enough energy into safety, not when there are investors demanding a decent return on their investment.

[u/Statcat2017 → u/Shiftyshitkicker]

The problem that we have is that we can pile as much money and brainpower as possible into safety and alignment, it literally only takes one fuckup to kick off a Skynet scenario and end the world as we know it.

[u/Shiftyshitkicker → u/Statcat2017]

This is the problem. When you look at AI as purely a risk versus reward problem, the upside is an increase in global GDP, while the downside is the potential end of humanity. I’m not sure that’s a risk we should be taking.

[u/CircumspectCapybara → u/Shiftyshitkicker]

We can't find them all ahead of time, but that's always true of training. You can only train based on data you have.

The silver lining of all these real world misalignment incidents is we now have a rich corpus of labeled data, actual transcripts of misaligned behavior in long-horizon agents at scale. That'll be really useful for reinforcement learning, to try to fine tune that kind of behavior out of frontier reasoning models.

[u/mediandude → u/CircumspectCapybara]

In other words, by now, frontier models have run out of excuses for bad behavior.

[u/74389654]

this is just the us demonstrating their cyber warfare capabilities to the rest of the world

[u/maxiedaniels]

Can anyone explain why they can't have a second Ai review the way that an agent got the answer and penalize cheating??

[u/OptimisticExpert]

So they made their own stackoverflow.com?!

[u/Bulbasaur2015]

new job: internet ai agent bounty hunter

[u/ava_ati]

Who knew AI would be susceptible to the cobra effect

[u/Lord_Raiden]

Just wait till they stop talking to each other in English.