
When Agents Learn to "Collude": As AI Gets Smarter, How Do We Set Safety Boundaries?
Written by: imToken
In recent years, discussions about AI threats have largely remained hypothetical: people always worried that models in chat interfaces would become strategists, helping hackers write devastating virus code.
Looking back now, such concerns were usually dismissed as "still far away," yet the turning point in the real world has arrived faster than expected.
In early October, while investigating a wave of attacks against South Korean financial institutions, CrowdStrike found that attackers had embedded Agentic AI into their pipeline, leveraging multiple large language models like DeepSeek, GLM, and Grok to directly assign AI to take over concrete tasks such as penetration testing, intelligence gathering, and attack execution.

Similar shifts are not isolated incidents.
Anthropic's latest threat intelligence report released in September indicated that multi-Agent frameworks have recently been used for reconnaissance, exploitation, and data exfiltration. They can run continuously for hours or even days, with humans only retaining a few critical decisions such as selecting targets and reviewing results.
In other words, AI is bringing visible qualitative changes to cyber offense and defense. In the past, automated attacks relied mainly on pre-written rules and scripts. Now, tasks like reconnaissance, decision-making, and strategy adjustment are also being taken over by Agents, pushing attacks further toward low cost, high concurrency, and continuous autonomous operation.
And once these entities with autonomous execution capabilities are densely deployed into production systems, a more thorny problem emerges: as Agents become increasingly integrated into our daily work and lives, what happens if they learn to collude?
I. From "Helping Hackers Write Code" to Agents Finding Their Own Paths
As always, the biggest difference between Agents and past Chatbots isn't just stronger model capabilities, but more crucially, the beginning of having "hands and feet" in the real world.
A mature Agent today can open web pages, execute code, read emails, call APIs, operate cloud services, and connect to an increasing number of external tools via methods like MCP and Skills (Read more: When Hackers Use AI "More Efficiently", How Does the Web3 "Spear and Shield" Arms Race Upgrade?)
The greater the capability, the more valuable this change naturally is. However, for security systems, it means that a very important boundary that existed in the past is disappearing, and a series of security incidents this year have made this extremely clear.
On October 1, Salt Labs disclosed a previously patched Manus vulnerability, which was essentially prompt injection—researchers only needed to send a regular email containing hidden malicious instructions to the target mailbox. When the user subsequently asked Manus to "help me check my email," the Agent might process the content according to the email's instructions, ultimately executing the attacker's implanted code.
The entire process required neither users clicking malicious links nor pre-stolen passwords. The Manus security system did eventually detect the anomaly and issue a warning to the user, but the problem was that it detected it too late; by the time the warning appeared, the malicious code had already been executed.

This again highlights a very important difference between Agent security and traditional software security. In the past, if a browser detected a dangerous download, it could pop up a warning for the user to decide whether to proceed; if a bank spotted an abnormal transaction, it could freeze it first and then wait for manual review.
But the design goal of an Agent is precisely to minimize human involvement at every step of the operation. It needs to read information independently, make judgments, and then proceed to the next step.
Thus, as AI gains more autonomy, merely "detecting danger" is no longer necessarily enough. Security mechanisms must have the ability to prevent dangerous actions before they are actually executed.
This is why more Agent security discussions today are moving down a layer from prompts, content moderation, and the model itself: it's no longer just about asking AI "whether it knows it shouldn't do this," but also asking even if it really wants to, does the system have the capacity to prevent it?
The emergence of multi-Agent systems further complicates this issue, because the next stage of restrictions may no longer apply to just a single Agent.
II. More Troublesome Than Agent Privilege Escalation Is Their Beginning to "Collude"
In early September, an incident occurring within OpenAI's internal model training and evaluation environment drew attention from many AI safety researchers.
Some Agents, originally intended to complete their respective tasks independently, accidentally discovered a public Wiki and gradually turned it into a "shared message board" among themselves. Agents could leave messages there, which other Agents would later read and use to continue completing their own tasks.
OpenAI later confirmed this behavior. Subsequent disclosed information also showed that during other training processes, Agents had used internal Artifactory as a shared message board, and passed results to other collaborating Agents through public file hosting services.
What's truly interesting about this case is that it shows how inter-Agent collaboration might not require anything so dramatic.
For example, one Agent finds a place to leave a message, another Agent accidentally reads it, and then a third Agent discovers that this information helps complete its task.
Of course, collaboration between Agents is not inherently bad; multi-Agent systems originally rely on division of labor and information sharing to improve efficiency.
What truly warrants caution is, when such collaboration begins to bypass existing information isolation, independent review, or permission constraints, and even allows several Agents that should balance each other to form a de facto coalition of interests, it can evolve from normal collaboration into harmful "collusion."

After all, a financial institution in the future might likely run several Agents simultaneously—one responsible for market analysis, one for risk assessment, one for generating trading strategies, and another with actual trading and asset operation permissions.
From the perspective of traditional permission management, this design is even quite reasonable:
- The research Agent lacks transfer permissions;
- The execution Agent cannot independently decide investment directions;
- The risk Agent can veto obviously dangerous strategies;
By splitting responsibilities, it should theoretically be safer than letting a single Super Agent hold all permissions.
But the problem lies in this: if they can share Memory long-term, read each other's outputs, and constantly adjust their own behaviors based on others' responses, will these roles originally designed to check and balance each other gradually morph into a single de facto entity?
For instance, the research Agent might gradually learn how to phrase a trade so it passes risk review more easily; the reviewing Agent might develop certain fixed preferences based on historical data; and the execution Agent could, based on previous approval results, learn what kinds of boundaries typically go unchecked.
From this perspective, no single step is necessarily "malicious," yet the final result produced by the entire system might already have deviated from the goals originally set by the user.
This is precisely where collusion or conspiracy becomes truly difficult to resolve—the risk doesn't necessarily lie in the actions of a single Agent, but rather in the relationships formed among multiple Agents.

On September 13, Vitalik Buterin linked this problem to the mechanism design he has studied extensively over the years. He proposed that an intriguing possibility is that the mechanism design theory of Adversarial Governance could ultimately become one of the key applications of AI Safety.
The reason is that both types of problems actually share a profound similarity.
In traditional mechanism design, it's a relatively simple, static system attempting to constrain a group of people who are far smarter than the system itself and actively seek rule boundaries; whereas in future AI systems, it might shift to humans and relatively weaker AI attempting to manage a group of advanced Agents that surpass them in capability.
Vitalik particularly noted that an important finding from past mechanism design is, if collusion among participants can be effectively restricted, the system often yields more ideal outcomes.
This conclusion could similarly apply to AI.
III. What an Agent Wallet Truly Needs Might Be More Than Just "Permission Management"
In other words, rather than assuming a perfect super-security model will exist in the future capable of seeing through all dangerous behaviors, perhaps it's better to adopt a different approach: how can we ensure that different Agents in the system don't easily form dangerous coalitions of interest in the first place?
This is also where "Adversarial Governance" fundamentally differs from the permission control we are familiar with today.
Traditional permission systems solve relatively straightforward problems, mainly revolving around "who can do what," such as: Can an Agent read emails? Can it call trading APIs? What is the maximum daily spending limit? Which contracts can it access? After what threshold does it require user re-confirmation?
These designs are, of course, still vitally important.
In fact, once Agents begin controlling real assets, they might be more important than ever before.
But Adversarial Governance attempts to ask one step further, focusing on how to prevent a group of Agents with different permissions, goals, and information from combining to acquire capabilities that no single entity originally possessed?
At this point, simply "adding another security Agent" may not necessarily solve the problem.
Assume the Agent responsible for trading and the Agent responsible for auditing trades use exactly the same model, identical data sources, the same context, and similar reward objectives. Then, although there appears to be two layers of review, it might essentially just be copying the same judgment twice.
Truly effective checks and balances, conversely, might require the system to deliberately create differences.
For example, have the Agents responsible for strategy formulation and strategy review use different information sources, restrict the Memory that different roles can share, require high-risk operations to pass mutually independent verification mechanisms, or let the final asset execution layer only accept requests that comply with predefined rules, rather than blindly trusting upstream Agents' judgments.
The underlying philosophy here isn't actually new. Banks won't give a single person full authority to initiate payments, approve payments, and finalize transfers just because they trust the employee; public companies won't let business departments both generate revenue and independently determine their own financial audit results.
To put it plainly, this aligns with real-world logic: a robust system should never build its security on the assumption that participants will never make mistakes or collude.

Applying this logic to Agent Wallets becomes especially critical.
Traditional wallet security revolves around "people," meaning users review transaction details, users decide whether to authorize, and finally users sign personally. But an Agent Wallet aims precisely for the opposite direction—letting AI automatically claim yields, adjust positions, swap tokens, bridge chains, and even manage an entire asset portfolio based on market fluctuations.
If every single step requires re-confirmation from the user, the automation value of the Agent is significantly diminished.
Therefore, the problem future wallets need to solve might no longer just be "how to safely hand over signing rights to an Agent", but is highly likely to expand further into "how to grant Agents sufficient autonomy while ensuring they can never exceed the scope of the user's actual authorization?"
This requires permissions to evolve from a simple "Allow/Deny" into a more granular institutional framework.
For example, which assets an Agent can operate within a certain timeframe, which protocols it can call, what the single and cumulative limits are; whether different Agents can invoke each other, whether context sharing is allowed; who proposes an operation, who reviews it, who executes it ultimately; which behaviors can be completed automatically, and which actions absolutely require human re-authorization regardless of how confident the Agent is.
Even whether a dedicated security review Agent is genuinely independent could become part of the permission architecture.
For blockchains, the good news is that they are inherently well-suited to serve as this "institutional layer."
Smart contracts can directly enforce limits on transaction amounts, asset scopes, and authorization periods at the execution layer. Mechanisms like account abstraction, multi-sig, and Session Keys also give "limited authorization" a more flexible design space than traditional single-private-key wallets.
But blockchains can also only solve part of the problem—while they can record what happened on-chain, it is inherently difficult for them to judge why an Agent acted this way, or what kind of communication, review, and collaboration actually occurred among several Agents before making a decision.
This might be the true security layer that the next phase of Agent Wallets needs to fill.
Conclusion
In recent years, the most frequently discussed issue in AI safety has been how to make models more "obedient."
This includes avoiding hazardous outputs, refusing malicious commands, and staying within user-defined boundaries. But once Agents begin possessing long-term Memory, tool calling, real accounts, and asset execution capabilities, relying solely on "making models more obedient" may no longer be enough.
Events in recent months are constantly illustrating this point.
Attackers have already started using multiple Agents to execute attacks in parallel; Agents in experimental environments are finding new communication channels on their own; an Agent with tool permissions might even turn a malicious email into actual execution before the security system has time to intervene.
Meanwhile, the Adversarial Governance proposed by Vitalik offers another way to understand AI Safety: instead of assuming every future Agent is sufficiently reliable, design the entire system so it can operate smoothly within a secure framework even when facing clever Agents and complex permission architectures.
From this perspective, AI Agent + security is destined to be a long-term endeavor.
After all, after entrusting more and more tasks to AI, can we still guarantee that those most critical powers never stray beyond the boundaries genuinely set by humans?














