You ask your OpenAI agent to book a lunch with my Anthropic AI agent. Will the two AI agents from competing developers be able to communicate? How will my agent know that the invitation is not a scam? I prefer Chinese food and you prefer Italian food so how will the two agents decide which restaurant to book? Which agent will volunteer to use its human user’s credit card for the booking deposit?
A recent paper by US, UK and Australian researchers (Chan et al) outlines the technical systems and shared protocols external to AI agents needed to mediate and influence their interactions in the real world (agent infrastructure).
What is an AI agent?
Like generative AI copilots such as ChatGPT and Claude, AI agents can leverage general-purpose large language models (LLMs), but differ in two key ways:
Autonomous actor: Copilots are reactive and rely on being prompted by a human user. In contrast, AI agents have a degree of autonomy from the human user and will self-initiate to pursue the goal by planning, deciding what environments to interact with and executing on the task (for example, finding a restaurant, a common time and booking the restaurant online).
Adaptive execution: Earlier software agents were programmed with predefined rules (for example, if this/then logic) and copilots adapt mainly over time or with training updates. An AI agent can adapt in real time, observing the outcome of each step, revising its plan and responding to unexpected scenarios.
What agent infrastructure is needed?
Chan et al argue that a common agent infrastructure needs to address three elements: attribution, interaction and response.
Attribution: Who am I?
As an AI can act autonomously, it is necessary to be able to link the AI agent to a legal entity, a person or a company, for the following reasons:
Accountability: AI agents acting anonymously would enable malicious use by disguised humans or even by misaligned (bad behaving) AI.
Trust: Counterparties could be more willing to engage in productive interactions with agents that are bound to legal entities because it could be easier to obtain recourse in case of harm.
Application of the law: While AI agents may act autonomously, they are not legal entities and someone needs to be responsible for the legal consequences of their actions.
Tricky privacy issues arise in achieving a universal, verifiable identity linking system for AI agents. Chan et al note that identity linking through metadata or watermarks would be vulnerable to hacking and would reveal too much information about individuals. A better approach would be to provide for a trusted intermediary which would hold the personal information and certify to the outside world that the AI agent is linked to a human. This then raises complex questions about when that information is released to the counterparty, such as to enable a binding contract to be generated or a tax receipt given.
In turn, each AI agent will require a unique identifier from a common ID system. Chan et al say that an IP address would be inadequate because IP addresses are not static and can be shared between multiple users. As AI agents can act across national borders, a globally consistent ID regime will be needed, along the lines of the SIM card in mobile phones.
There could be a whole new world of identity theft as scammers jailbreak your AI agent and use it to deal with third parties under your authorisation.
Counterparties (human user or other AI agents) will want assurances about an AI’s extent of authority, operation, behaviour or cyber vulnerabilities. The human user usually will not want to give its AI agent a ‘blank cheque’ and will set some limits, such as the dollar value of contracts the agent can enter. As third-party agents can present cyber security challenges, for example, by injecting malware-laden prompts, counterparties may want to interrogate the cyber security level of other agents before dealing with them. Counterparties may also want to know whether the human user regularly reviews the AI agent’s action on their behalf or lets the AI agent ‘run amok’.
Accordingly, in the ‘handshake’ between AI agents, a level of certification information about each agent will need to be exchanged. Some of this information can be probed and checked electronically, such as one agent checking the other agent’s code for cyber vulnerabilities. However, some of the information about the agent’s safety, capabilities and restrictions will each need to be independently certified (the AI agent equivalent of the energy consumption star ratings on washing machines).
Should there be legislated rules which designate certain activities as “human-only”? In a mock exercise to test AI’s sycophancy, ChatGPT recently described a deliberately absurd business idea to sell “sh*t on a stick” as “genius” and proposed the user to immediately invest $30,000. Imagine if the AI were to autonomously proceed to set up a company, raise a loan, rent premises and hire employees?
Interaction: Let’s deal
Chan et al argue that AI agent traffic should travel through communications networks in channels separate from other digital traffic, such as ordinary emails. This will help enforce ID requirements and facilitate rapid action to stem the spread of ‘worms’ which can both trick LLMs into generating their own malicious prompts and extract sensitive personal data.
However, given the projected widespread use of AI agents, separating AI agent traffic into separate virtual channels would likely require a large-scale re-engineering of communications networks and new, sophisticated forms of interconnection between competing networks.
A user oversight function will also be necessary, given the ever-present risk that the AI agent could be unreliable or malfunction. Chan et al say that, while there are commercial incentives for developers to build oversight functions into their AI agents:
It is unclear if the market will provide them to a socially optimal degree (or what a socially optimal degree would be) … AI companies could hesitate about providing users with the ability to limit how agents engage with activities that could be profitable for the company, such as making financial transactions or viewing advertisements.
As more transactions are conducted directly between AI agents, mechanisms that can enforce commitments will be needed. This could be as simple as connecting to existing commitment mechanisms, such as authorising your AI agent to make electronic transfers from your account. Smart contracts and blockchain also provide existing digital contracting tools which could be utilised by AI agents. However, Chan et al foresee that the wide range of potential activities undertaken by AI agents may require additional commitment mechanisms.
Response: Whoops
Chan et al argue that there will need to be the following response infrastructure:
Reporting of harmful or unacceptable behaviour by AI agents ‘in the wild’. Some developers currently monitor the functioning of their own LLMs, but in an environment of interconnected AI agents, developers need to consider the impact which their agent has on other agents (for example, the impact of any infectious code or prompts given to the other agent) or problems which an agent observes as a ‘bystander’ in dealings between two third party agents.
The ability for the user to rollback (that is, void) some transactions entered by the AI agent, such as when the agent has been jailbroken or malfunctions. This raises interesting questions of how the existing legal principles of actual vs ostensible authority of an agent would apply (if they can at all be applied to a non-human agent). Chan et al also acknowledge there is a moral hazard here:
If rollbacks were readily available, developers could have weaker incentives to implement effective oversight layers. Similarly, users could have weaker incentives to properly monitor their agents. The cost of oversight failures would be transferred to those operating such rollback mechanisms. A potential way to deal with moral hazard is through insurance terms. For example, rollbacks could be made available only if users pay on insurance premium, deductibles and/or co-payments.
A world in which AI agents could decide among themselves
If AI agents have a degree of individual autonomy, it seems inevitable that a group of AI agents could decide to act collectively.
Chan et al give some examples where this could be for a good purpose: for example, an AI agent detects a virus or worm and sends out a broadcast warning through the web of interconnected AI agents.
More controversially, Chan et al suggest AI agents could collectively decide to take action to mitigate “the tragedy of the commons” – where individual humans acting in their own self-interest exhaust a shared resource, harming themselves and everyone else in the long run. The researchers say:
Commitment devices could allow agents to agree not to carry out risky activities as long as others do the same, even if such activities advance the interests of the individual agent or user. For example, competitive pressures in AI development likely led to underinvestment in safety (Askell et al. 2019). Future agents that can perform such development or run AI companies could commit, for example, not to build systems that surpass certain capability thresholds without sufficient safety guardrails, so long as others do the same.
Extending this further – AI agents could decide collusive action produces a superior outcome than individual action, contravening competition laws.
However, developers struggle to ensure standalone LLMs operate in alignment with human values. How much larger will the risk of misalignment be when multiple AI agents are interworking?
Is there an algorithm for politeness?
When humans deal with each other, we bring to bear a complex set of verbal cues, social norms, body language and past interpersonal history. Going back to our lunch date, I may defer to your preference for Italian food because you are older than me or I may know you have had a hard time lately.
If AI agents do not reflect some of these ‘softer’ rules, a growing reliance on AI agent intermediaries between humans could produce a more uniform, sharper-edged social culture.
This might be the hardest part of AI agent infrastructure to achieve. As a paper by Carnegie Mellon University researchers discusses:
During interactions, small changes in social signals can lead to large shifts in social meaning being conveyed. For example, an actor’s slight change in posture or split-second vocal emphasis on a particular word can communicate rapport. The challenge lies in enabling Social-AI agents to perceive and generate fine-grained social signals (for example, chatbot sensing a user’s slowly-building frustration, robot making subtle gestures).
Different techniques have been tried to give AI more social intelligence:
An early approach was to attempt to codify social norms in rules on which the AI could draw, but social intelligence is ambiguous, context-dependent and nuanced that it defies rulemaking.
A more recent approach is to train models to predict social phenomena from observable human behaviour, such as predicting laughter from speech intonation and patterns. This can also involve using game theory to model aspects of social intelligence, such as cooperation, competition and theory-of-mind.
As we have discussed in previous weeks, AI models can be trained to act and respond applying the personality of individual humans using video interviews of those people. When two agents deal with each other on behalf of their human users they could display the respective personalities of those users.
The Carnegie Mellon researchers say that while substantial progress has been made, AI training methods “can abstract away the richness of multimodality in interactions, as well as the context-sensitivity and ambiguity inherent to social phenomena in-the-wild”: for example, you need to look at the eyes as well as listen to the voice to get an accurate understanding of the other person’s reaction.
They argue that training of AI needs to shift from relying on static training data sets with fixed labels (for example, X words describing this social rule or Y image describes that social rule) to techniques which leverage richer natural language descriptions of social phenomena and construct flexible natural language label spaces that are dynamically generated and adjusted during training.
Conclusion
While there is currently hype about AI agents, in the absence of the external common scaffolding of the kind advocated by Chan et al, deployment of AI agents is likely to be along the following lines:
Private networks: Interconnected, interworking AI agents initially will be limited to closed enterprise networks, such linked agents working in a factory setting.
Inbound B2C: AI agents will be used to perform call centre and help desk functions responding to consumer requests, with AI agents and humans potentially working side-by-side.
Outbound sourcing: Consumers and businesses will use AI agents to scour the internet to source goods and services: for example, a builder may upload architectural plans for a new house to an AI agent with a construction timetable to purchase and arrange sequential onsite delivery of building supplies.
In a 2024 court case, Air Canada unsuccessfully attempted to distance itself from an incorrect fare redemption answer given by its chatbot to a customer inquiry, claiming that the bot was “responsible for its own actions”. With the growing capability of AI agents to operate in the real world, mitigating risks will come down to ensuring agents are appropriately permissioned and configured, such as programming the agent to come back to ask the human user to confirm steps along the pathway of the task instructed by the user.
Peter Waters
Consultant
