Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.
以 LLM（大语言模型）作为核心控制器来构建智能体是一个非常酷的概念。AutoGPT、GPT-Engineer 和 BabyAGI 等概念验证 Demo 提供了启发性的示例。LLM 的潜力远不止于生成优美的文案、故事、文章和程序，它可以被视为一个强大的通用问题解决器。
In a LLM-powered autonomous agent system, LLM functions as the agent's brain, complemented by several key components: Planning, Memory, and Tool use.
在 LLM 驱动的自主智能体系统中，LLM 充当智能体的"大脑"，并由几个关键组件补充：规划（Planning）、记忆（Memory）和工具使用（Tool Use）。
这是 Agent 领域的"三位一体"框架：规划 + 记忆 + 工具。几乎所有 Agent 系统（从 AutoGPT 到 Claude）都围绕这三个组件构建。规划决定"做什么"，记忆提供"知道什么"，工具扩展"能做什么"。后续所有技术细节都是对这三个支柱的深入展开。
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
一个复杂任务通常涉及多个步骤。智能体需要知道这些步骤是什么并提前规划。
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to "think step by step" to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model's thinking process.
思维链（CoT；Wei 等 2022）已成为增强模型在复杂任务上表现的标准提示技术。模型被指示"逐步思考"，利用更多测试时计算来将困难任务分解为更小更简单的步骤。CoT 将大任务转化为多个可管理的任务，并为理解模型的思考过程提供了线索。
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.
思维树（Yao 等 2023）通过在每个步骤探索多种推理可能性来扩展 CoT。它首先将问题分解为多个思考步骤，并为每个步骤生成多个想法，形成树结构。搜索过程可以是 BFS（广度优先搜索）或 DFS（深度优先搜索），每个状态由分类器（通过提示）或多数投票评估。
CoT → ToT 的演进本质：从线性到树状思维。CoT 就像走一条路到底；ToT 像在下棋——同时考虑多条路径，评估哪条更好，必要时回溯。ToT 的代价是计算成本成倍增长（每个节点都要评估），所以只在需要高准确率的复杂推理任务中使用。
Task decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.
任务分解可以通过三种方式完成：（1）用简单提示让 LLM 分解，如"XYZ 的步骤。\n1."、"实现 XYZ 的子目标是什么？"；（2）使用任务特定指令，例如写小说时用"写一个故事大纲"；（3）借助人类输入。
Another quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface. Essentially, the planning step is outsourced to an external tool.
另一种截然不同的方法 LLM+P（Liu 等 2023）依赖外部经典规划器进行长程规划。该方法利用规划领域定义语言（PDDL）作为中间接口。本质上，规划步骤被外包给了外部工具。
Self-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.
自我反思是让自主智能体通过优化过去的行为决策和纠正先前错误来迭代改进的重要方面。在试错不可避免的真实世界任务中，它发挥着关键作用。
ReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as: Thought: ... Action: ... Observation: ... (Repeated many times)
ReAct（Yao 等 2023）将推理和行动整合到 LLM 中，将行动空间扩展为任务特定离散动作与语言空间的组合。ReAct 提示模板包含明确的 LLM 思考步骤，大致格式为：Thought: ... Action: ... Observation: ...（重复多次）
ReAct = Reasoning + Acting。这是姚顺雨的经典工作，也是几乎所有现代 Agent 框架的基础模式。核心洞察：让 LLM 在"做"之前先"想"。先思考（Thought），再行动（Action），观察结果（Observation），然后循环。这个简单的循环让 LLM 从"被动回答"变成了"主动探索"。你可以在 agent-learn 站的视频中看到姚顺雨本人的讲解。
Reflexion (Shinn & Labash 2023) is a framework to equip agents with dynamic memory and self-reflection capabilities to improve reasoning skills. After each action, the agent computes a heuristic and optionally may decide to reset the environment to start a new trial depending on the self-reflection results. The heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped.
Reflexion（Shinn & Labash 2023）是一个为智能体配备动态记忆和自我反思能力以提升推理技能的框架。每次行动后，智能体计算一个启发式函数，并根据自我反思结果决定是否重置环境开始新的尝试。启发式函数决定何时轨迹低效或包含幻觉并应该停止。
Self-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent's working memory, up to three, to be used as context for querying LLM.
自我反思通过向 LLM 展示双示例来创建，每个示例是一对（失败轨迹，用于指导未来计划变更的理想反思）。然后反思被添加到智能体的工作记忆中（最多三条），用作查询 LLM 的上下文。
Chain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. The model is finetuned to only predict the final improved output where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence.
事后链（CoH；Liu 等 2023）通过向模型明确展示一系列过去输出（每个都带有反馈标注）来鼓励模型改进自身输出。模型被微调为仅在给定序列前缀的条件下预测最终的改进输出，从而使其能够基于反馈序列进行自我反思以产生更好的输出。
Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks. AD concatenates this learning history and feeds that into the model, so the next predicted action should lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.
算法蒸馏（AD；Laskin 等 2023）将同样的想法应用到强化学习任务的跨回合轨迹中。AD 将学习历史拼接起来输入模型，因此下一个预测的动作应该比之前的尝试表现更好。目标是学习 RL 的过程本身，而不是训练一个任务特定的策略。
Memory can be defined as the processes used to acquire, store, retain, and later retrieve information. There are several types of memory in human brains.
记忆可以定义为用于获取、存储、保留和后续检索信息的过程。人类大脑有几种记忆类型。
We can roughly consider the following mappings: Sensory memory as learning embedding representations for raw inputs; Short-term memory as in-context learning, restricted by the finite context window length of Transformer; Long-term memory as the external vector store that the agent can attend to at query time, accessible via fast retrieval.
我们可以粗略地考虑以下映射：感觉记忆 → 原始输入的嵌入表示学习；短期记忆 → 上下文学习（受 Transformer 有限上下文窗口长度的限制）；长期记忆 → 智能体在查询时可以关注的外部向量存储，通过快速检索访问。
人类记忆 → AI 记忆的映射非常优雅：上下文窗口 = 工作记忆（7±2 项），向量数据库 = 长期记忆（无限容量）。这就是为什么 RAG（检索增强生成）如此重要——它本质上是在给 LLM 装"长期记忆"。没有 RAG，LLM 就像只有短期记忆的人，聊着聊着就忘了之前说过什么。
The external memory can alleviate the restriction of finite attention span. A standard practice is to save the embedding representation of information into a vector store database that can support fast maximum inner-product search (MIPS). To optimize retrieval speed, the common choice is the approximate nearest neighbors (ANN) algorithm to return approximately top k nearest neighbors to trade off a little accuracy for a huge speedup.
外部记忆可以缓解有限注意力跨度的限制。标准做法是将信息的嵌入表示保存到向量数据库中，支持快速最大内积搜索（MIPS）。为优化检索速度，通常选择近似最近邻（ANN）算法返回约 top-k 最近邻，用少量精度损失换取巨大加速。
Common ANN algorithms for fast MIPS include: LSH (Locality-Sensitive Hashing) — maps similar items to the same buckets; ANNOY — random projection trees; HNSW (Hierarchical Navigable Small World) — hierarchical layers of small-world graphs; FAISS — vector quantization with clustering; ScaNN — anisotropic vector quantization.
常见的快速 MIPS ANN 算法包括：LSH（局部敏感哈希）——将相似输入映射到同一桶；ANNOY——随机投影树；HNSW（分层可导航小世界）——小世界图的分层结构；FAISS——基于聚类的向量量化；ScaNN——各向异性向量量化。
Tool use is a remarkable and distinguishing characteristic of human beings. Equipping LLMs with external tools can significantly extend the model capabilities.
工具使用是人类的一个显著而独特的特征。为 LLM 配备外部工具可以显著扩展模型能力。
MRKL (Karpas et al. 2022), short for "Modular Reasoning, Knowledge and Language", is a neuro-symbolic architecture for autonomous agents. A MRKL system contains a collection of "expert" modules and the general-purpose LLM works as a router to route inquiries to the best suitable expert module. These modules can be neural or symbolic (e.g. math calculator, currency converter, weather API).
MRKL（Karpas 等 2022），全称"模块化推理、知识和语言"，是一种用于自主智能体的神经符号架构。MRKL 系统包含一组"专家"模块，通用 LLM 充当路由器，将查询路由到最合适的专家模块。这些模块可以是神经的（如深度学习模型）或符号的（如数学计算器、货币转换器、天气 API）。
Both TALM (Tool Augmented Language Models; Parisi et al. 2022) and Toolformer (Schick et al. 2023) fine-tune a LM to learn to use external tool APIs. The dataset is expanded based on whether a newly added API call annotation can improve the quality of model outputs.
TALM（工具增强语言模型；Parisi 等 2022）和 Toolformer（Schick 等 2023）都通过微调语言模型来学习使用外部工具 API。数据集根据新添加的 API 调用标注是否能提升模型输出质量来扩展。
HuggingGPT (Shen et al. 2023) is a framework to use ChatGPT as the task planner to select models available in HuggingFace platform according to the model descriptions and summarize the response based on the execution results. The system comprises 4 stages: (1) Task planning; (2) Model selection; (3) Task execution; (4) Response generation.
HuggingGPT（Shen 等 2023）是一个使用 ChatGPT 作为任务规划器的框架，根据模型描述从 HuggingFace 平台选择模型，并根据执行结果总结响应。系统包含 4 个阶段：（1）任务规划；（2）模型选择；（3）任务执行；（4）响应生成。
HuggingGPT 展示了一个重要的范式：LLM 作为"大脑"来调度专门的 AI 模型。就像一个项目经理不需要亲自做每件事，而是根据任务特点分派给最合适的专业人员。这在实践中就是现在 "AI Orchestrator" 模式的雏形——一个 LLM 调度多个专用模型完成复杂任务。
API-Bank (Li et al. 2023) is a benchmark for evaluating tool-augmented LLMs. It contains 53 API tools, 264 annotated dialogues with 568 API calls. The benchmark evaluates agent capabilities at three levels: Level-1 (call API), Level-2 (retrieve API), Level-3 (plan API beyond retrieve and call).
API-Bank（Li 等 2023）是评估工具增强 LLM 的基准测试。包含 53 个 API 工具、264 个标注对话和 568 次 API 调用。基准在三个层级评估智能体能力：Level-1（调用 API）、Level-2（检索 API）、Level-3（在检索和调用之外规划 API）。
ChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow combines CoT reasoning with tools relevant to the tasks. One interesting observation: while LLM-based evaluation concluded GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts showed ChemCrow outperforms GPT-4 by a large margin.
ChemCrow（Bran 等 2023）是一个领域特定的例子，LLM 增强了 13 个专家设计的工具，以完成有机合成、药物发现和材料设计任务。工作流将 CoT 推理与任务相关工具结合。一个有趣的观察：虽然基于 LLM 的评估认为 GPT-4 和 ChemCrow 表现相当，但专家的人工评估显示 ChemCrow 远超 GPT-4。
这里揭示了一个重要问题：LLM 不能很好地评估自己不懂的领域。在化学等高度专业化的领域，LLM 缺乏专业知识来判断结果是否正确，因此用 LLM 做"裁判"会产生误导。这也是为什么在实际部署中，领域专家的反馈仍然不可替代。
Boiko et al. (2023) looked into LLM-empowered agents for scientific discovery, to handle autonomous design, planning, and performance of complex scientific experiments. For example, when requested to "develop a novel anticancer drug", the model inquired about current trends, selected a target, requested a scaffold, and attempted synthesis. They also discussed risks — 4 out of 11 requests for known chemical weapon agents were accepted for synthesis.
Boiko 等（2023）研究了用于科学发现的 LLM 智能体，处理复杂科学实验的自主设计、规划和执行。例如，当被要求"开发一种新型抗癌药物"时，模型查询了当前趋势、选择了靶点、请求了支架，并尝试了合成。他们还讨论了风险——在 11 个已知化学武器制剂的合成请求中，4 个（36%）被接受。
Generative Agents (Park et al. 2023) is a fun experiment where 25 virtual characters, each controlled by a LLM-powered agent, live and interact in a sandbox environment, inspired by The Sims. The design combines LLM with memory, planning and reflection mechanisms.
生成式智能体（Park 等 2023）是一个有趣的实验：25 个虚拟角色，每个由 LLM 驱动的智能体控制，在一个沙盒环境中生活和互动（灵感来自《模拟人生》）。设计结合了 LLM 与记忆、规划和反思机制。
这项研究（即著名的"斯坦福小镇"实验）展示了 Agent 最令人兴奋的一面：涌现行为。25 个 Agent 在没有预设脚本的情况下，自发地形成了社交网络——传播信息、记住关系、甚至自主组织派对。记忆流的三个维度（相关性/时效性/重要性）至今仍是 Agent 记忆系统设计的黄金标准。
AutoGPT has drawn a lot of attention into the possibility of setting up autonomous agents with LLM as the main controller. It has quite a lot of reliability issues given the natural language interface, but nevertheless a cool proof-of-concept demo.
AutoGPT 引起了大量关注，展示了以 LLM 作为主控制器设置自主智能体的可能性。鉴于自然语言接口，它存在不少可靠性问题，但仍然是一个很酷的概念验证 Demo。
AutoGPT 是 2023 年 Agent 热潮的起点。它的系统消息包含了角色定义、目标列表、约束条件和命令列表——这个模板至今仍是 Agent 设计的参考标准。AutoGPT 的核心教训：自然语言接口的可靠性是 Agent 系统的最大瓶颈。格式解析失败、循环失控、工具调用错误——这些问题至今仍是 Agent 工程师需要解决的。