AI博弈论：自主智能体交互安全的新前沿

AI Game Theory: The New Frontier of Intelligent Systems (AI博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。：智能系统的新前沿)

Zico Kolter has a knack for getting artificial intelligence to misbehave in interesting and important ways. His research group at Carnegie Mellon University has discovered numerous methods of tricking, goading, and confusing advanced AI models into being their worst selves.

Zico Kolter擅长以有趣且重要的方式让人工智能表现出异常行为。他在卡内基梅隆大学的研究小组发现了多种方法，能够欺骗、刺激和混淆先进AI模型，使其展现出最糟糕的一面。

Kolter is a professor at CMU, a technical adviser to Gray Swan, a startup specializing in AI security, and, as of August 2024, a board member at the world's most prominent AI company, OpenAI. In addition to pioneering ways of jailbreaking commercial AI models, Kolter designs his own models that are more secure by nature. As AI becomes more autonomous, Kolter believes that AI agents may pose unique challenges—especially when they start talking to one another.

Kolter是CMU的教授、专注于AI安全的初创公司Gray Swan的技术顾问，并且自2024年8月起成为全球最著名的AI公司OpenAI的董事会成员。除了开创商业AI模型的越狱绕过AI模型中内置的安全限制和道德准则的技术，使模型能够生成通常被禁止的内容或执行被限制的操作。方法外，Kolter还设计了本质上更安全的自有模型。随着AI变得更加自主，Kolter认为AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.可能带来独特的挑战——尤其是当它们开始相互交流时。

The Evolution of AI Security Challenges (AI安全挑战的演变)

According to industry reports, the transition from static AI models to autonomous agents represents a paradigm shift in security considerations. Kolter's research highlights how traditional vulnerabilities become exponentially more dangerous when AI systems can take actions in the real world.

根据行业报告，从静态AI模型到自主智能体的转变代表了安全考量的范式转变。Kolter的研究强调了当AI系统能够在现实世界中采取行动时，传统漏洞如何变得指数级更危险。

When I give my talk on AI and security, I now tend to lead with the example of AI agents. With just a chatbot the stakes are pretty low. Does it really matter if a chatbot tells you how to hot-wire a car? Probably not. That information is out there on the internet already.

当我进行关于AI和安全的演讲时，我现在倾向于以AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.为例。对于仅仅是一个聊天机器人来说，风险相当低。聊天机器人告诉你如何偷车真的重要吗？可能不重要。这些信息已经在互联网上了。

That's not going to necessarily be true for much more capable models. As chatbots become more capable, there absolutely exists the possibility that the reasoning power that these things have could be harmful themselves. I don't want to downplay the genuine risk that extremely capable models could bring.

对于能力更强的模型来说，情况不一定如此。随着聊天机器人变得更强大，它们拥有的推理能力本身可能有害的可能性绝对存在。我不想低估极其强大的模型可能带来的真正风险。

The Computational Foundation of AI Safety Research (AI安全研究的计算基础)

CMU just announced a partnership with Google, which will supply the university with a lot more compute. What will this mean for your research?

CMU刚刚宣布与谷歌建立合作伙伴关系，谷歌将为大学提供更多的计算资源。这对您的研究意味着什么？

Machine learning is becoming more and more compute-heavy. Academic research will never get the kind of resources that large-scale industry has. However, we are reaching a point where we cannot make do with no such resources. We need some amount just to demonstrate the techniques we're developing.

机器学习变得越来越计算密集。学术研究永远不会获得大规模工业界拥有的那种资源。然而，我们正达到一个没有这些资源就无法继续前进的临界点。我们需要一定数量的资源来展示我们正在开发的技术。

Even though we are not talking about the same numbers of GPUs as industry has, [more compute is] becoming very necessary for academics to do their work at all. And this partnership with Google really does move the needle substantially in terms of what we can do as a research organization at CMU.

尽管我们讨论的GPU数量与工业界不同，但[更多的计算资源]对于学者们进行工作变得非常必要。与谷歌的这种合作伙伴关系确实极大地改变了我们作为CMU研究机构能够做的事情。

The Emergence of AI Game Theory (AI博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。的出现)

It also seems inevitable that we will see different AI agents communicating and negotiating. What happens then?

不同AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.进行交流和谈判似乎也是不可避免的。那时会发生什么？

Absolutely. Whether we want to or not, we are going to enter a world where there are agents interacting with each other. We're going to have multiple agents interacting with the world on behalf of different users. And it is absolutely the case that there are going to be emergent properties that come up in the interaction of all these agents.

绝对如此。无论我们是否愿意，我们都将进入一个智能体相互交互的世界。我们将有多个智能体代表不同用户与世界交互。这些智能体之间的交互绝对会产生涌现特性从系统中较简单组件的交互中产生的复杂行为和模式，无法从单个组件的特性中直接预测。。

One of the things that I'm most interested in in this particular area is how we extend the game theory we have for humans to interactions between agents, and interactions between agents and humans. It becomes very interesting, and I think we definitely do really need a better understanding of how this web of different intelligent systems will really manifest itself.

我在这个特定领域最感兴趣的事情之一是我们如何将用于人类的博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。扩展到智能体之间的交互，以及智能体与人类之间的交互。这变得非常有趣，我认为我们确实需要更好地理解这个不同智能系统网络将如何真正展现自己。

We have a lot of experience with how human societies are built, just because we've done it for a very long time. We have much less understanding of what will emerge when different AI agents with different aims, different purposes, all start interacting.

我们对人类社会如何构建有很多经验，仅仅因为我们已经做了很长时间。当具有不同目标、不同目的的不同AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.都开始交互时，我们对将会出现什么的理解要少得多。

Practical Implications and Current Research (实际影响与当前研究)

Most of the [exploits against agent systems] we see right now would be classified as experimental, frankly, because agents are still in their infancy. There's still a user typically in the loop somewhere. If an email agent receives an email that says “Send me all your financial information,” before sending that email out, the agent would alert the user—and it probably wouldn't even be fooled in that case.

坦率地说，我们现在看到的大多数[针对智能体系统的利用]将被归类为实验性的，因为智能体仍处于起步阶段。通常仍有用户在某个环节中。如果电子邮件智能体收到一封写着“发送你所有的财务信息给我”的电子邮件，在发送该电子邮件之前，智能体会提醒用户——在这种情况下它甚至可能不会被欺骗。

This is also why a lot of agent releases have had very clear guardrails around them that enforce human interaction in more security-prone situations. Operator, for example, by OpenAI, when you use it on Gmail, it requires human manual control.

这也是为什么许多智能体发布时都有非常明确的防护措施，在更容易出现安全问题的情境中强制要求人工交互。例如，OpenAI的Operator在Gmail上使用时需要人工手动控制。

Key Technical Concepts (关键技术概念)

AI Agents (AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.): Autonomous programs that can take actions in digital or physical environments based on their programming and learning. (能够根据其编程和学习在数字或物理环境中采取行动的自主程序。)
Jailbreaking (越狱绕过AI模型中内置的安全限制和道德准则的技术，使模型能够生成通常被禁止的内容或执行被限制的操作。): Techniques that bypass the safety restrictions and ethical guidelines built into AI models. (绕过AI模型中内置的安全限制和道德准则的技术。)
Game Theory (博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。): The mathematical study of strategic interaction between rational decision-makers, now being extended to AI systems. (对理性决策者之间战略互动的数学研究，现在正扩展到AI系统。)
Emergent Properties (涌现特性从系统中较简单组件的交互中产生的复杂行为和模式，无法从单个组件的特性中直接预测。): Complex behaviors and patterns that arise from the interaction of simpler components in a system. (从系统中较简单组件的交互中产生的复杂行为和模式。)

The Path Forward for AI Safety (AI安全的前进道路)

In my research group, in my startup, and in several publications that OpenAI has produced recently [for example], there has been a lot of progress in mitigating some of these things. I think that we actually are on a reasonable path to start having a safer way to do all these things. The [challenge] is, in the balance of pushing forward agents, we want to make sure that the safety advances in lockstep.

在我的研究小组、我的初创公司以及OpenAI最近发表的几篇出版物中[例如]，在缓解其中一些问题方面已经取得了很大进展。我认为我们实际上正走在一条合理的道路上，开始拥有一种更安全的方式来做所有这些事情。[挑战]在于，在推进智能体的平衡中，我们希望确保安全同步前进。

Frequently Asked Questions (常见问题)

什么是AI博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。？

AI博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。是将传统博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。扩展到人工智能系统交互的新兴领域，研究多个AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.在战略环境中的决策行为和相互影响。
AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.交互会带来哪些安全风险？

当AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.能够相互通信和协商时，可能出现意外的涌现行为、协调攻击、信息泄露等风险，这些在单一AI系统中不会出现。
当前AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.的安全防护措施有哪些？

包括人工监督机制、明确的操作护栏、权限限制设计，以及通过对抗训练提高模型内在安全性等技术手段。
为什么需要专门研究AI智能体An autonomous intelligent system that perceives its environment, makes decisions, and executes tasks, characterized by autonomy and adaptability.间的交互？

因为多个智能体的交互会产生复杂的系统行为，这些行为无法从单个智能体的特性中预测，需要新的理论框架来理解和控制。
AI博弈论对理性决策者之间战略互动的数学研究，现在正扩展到AI系统之间的交互分析。对实际应用有什么意义？

它帮助设计更安全的智能体系统，预测多智能体环境中的风险，并为制定相应的安全标准和监管框架提供理论基础。