Test-time compute and Chain-of-thought (CoT) have led to significant improvements in model performance, while raising many research questions. This post reviews recent developments in how to effectively use test-time compute (i.e. "thinking time") and why it helps.
测试时计算（test-time compute）和思维链（CoT）带来了模型性能的显著提升，也引出了许多研究问题。本文综述如何有效利用测试时计算（即"思考时间"）以及它为何有效。这是 o1/o3 类推理模型兴起的原理性背景——把推理过程显式化为可生成的 token，让模型在回答前"想一会儿"。
The core idea is deeply connected to how humans think. In Thinking, Fast and Slow, Kahneman characterizes human thinking into two modes via the dual process theory: Fast thinking (System 1) operates quickly and automatically, driven by intuition; Slow thinking (System 2) demands deliberate, logical thought and significant cognitive effort. By consciously slowing down, we engage System 2 to make more rational choices.
核心思想与人类思考方式深度相关。Kahneman 在《思考，快与慢》中通过"双系统理论"把人类思维分为两种模式：快思考（系统 1）快速、自动、靠直觉，几乎不费力；慢思考（系统 2）需要刻意、逻辑性的思考，消耗大量认知资源。通过有意识地放慢，我们能调用系统 2 做出更理性的选择。
From a computation view: a neural network can be characterized by the computation and storage it accesses in a forward pass. In Transformer models, flops per generated token ≈ 2 × parameters. CoT enables the model to perform far more flops per output token, and crucially, allows a variable amount of compute depending on problem hardness.
从计算视角看：神经网络的能力可以用它在一次前向传播中可访问的计算与存储来衡量。Transformer 每个 token 的计算量约为 2 × 参数量。CoT 让模型对每个输出 token 执行多得多的浮点运算，更关键的是，它允许计算量随问题难度自适应变化——这是测试时计算最优雅的性质。
把"思考"等同于"花更多算力"是一个深刻而简洁的视角。系统 1 对应模型一次性出答案（前向传播的固定算力），系统 2 对应 CoT——把推理展开为序列，每多生成一个 token 就多花一份算力。CoT 的本质是让模型能根据问题难度"按需付费"：简单题一句话答完，难题想几百步。这也解释了为什么推理模型（o1/R1）的"思考长度"差异巨大——它学会了为难题分配更多预算。
The strategy of generating intermediate steps before short answers was explored by Ling et al. (2017, AQUA-RAT) and Cobbe et al. (2021, GSM8K). Nye et al. (2021) experimented with intermediate thinking tokens as scratchpads, and Wei et al. (2022) coined the now-standard term "chain-of-thought." Later work found CoT can be significantly improved by RL on problems with automatically checkable solutions (STEM short answers, coding with unit tests).
在生成简短答案前先生成中间步骤的策略，由 Ling 等（2017，AQUA-RAT 数据集）和 Cobbe 等（2021，GSM8K 数据集）探索。Nye 等（2021）用中间"思考 token"作为草稿本，Wei 等（2022）创造了如今的标准术语"思维链"。后续工作发现，在有可自动验证答案的问题（STEM 简答题、可单测的编程题）上做 RL，能显著提升 CoT 推理能力。
这条演进线索是理解推理模型的关键：从"提示模型一步步想"到"用 RL 训练模型一步步想"。早期 CoT 是提示工程（"let's think step by step"），靠的是模型本身已有的能力；RL 路线（Zelikman 2022 STaR、DeepSeek-R1）则是把"正确推理"作为奖励信号显式训练——这是质变。可验证性是这一切的基础：只有当答案能自动判定对错时，RL 才能产生有效的推理训练信号。这就是为什么推理模型的突破首先发生在数学和编程领域。
The fundamental intent of test-time compute is to adaptively modify the model's output distribution. Two main approaches: Parallel sampling generates multiple outputs simultaneously, using process reward signals or verifiers to judge quality. Sequential revision adapts responses iteratively based on previous output, asking the model to reflect and correct mistakes. Parallel is simple but bounded by whether the model can get it right in one go; sequential explicitly asks for reflection but risks corrupting correct answers.
测试时计算的根本意图是在测试时自适应地修改模型的输出分布。两种主要方法：并行采样同时生成多个输出，用过程奖励信号或验证器判断质量；顺序修正基于上一步输出迭代地改进回复，让模型反思并纠正错误。并行简单但受限于模型能否一次做对；顺序明确要求反思但有把正确答案改错的风险。
Snell et al. (2024) showed that easier questions benefit from purely sequential test-time compute, whereas harder questions often perform best with an optimal ratio of sequential to parallel compute.
Snell 等（2024）发现：简单题从纯顺序测试时算力中获益，而难题往往在顺序与并行算力的某个最优比例下表现最好。
Best-of-N collects N independent samples and chooses the highest-ranking. Beam search maintains promising partial sequences and alternates between extending and pruning. Process reward models (PRM; Lightman et al. 2023) can guide beam search. Xie et al. (2023) used LLM self-evaluation per reasoning step (as multiple-choice) to reduce accumulative errors. Interestingly, Wang & Zhou (2024) found that branching only at the first sampling token can natively trigger emergent CoT.
Best-of-N 采集 N 个独立样本并选最高分。束搜索维护有前景的部分序列，在扩展与剪枝间交替。过程奖励模型（PRM）可引导束搜索。Xie 等（2023）用每步自评估（多选题形式）减少累积误差。有趣的是，Wang & Zhou（2024）发现仅在第一个采样 token 处分支，就能自然触发涌现的思维链。
Self-correction does not exist intrinsically among LLMs and does not work out of the box. Failure modes include (1) hallucination—modifying correct responses to incorrect; (2) behavior collapse—making minor or no edits; (3) failure to generalize to distribution shift. Experiments showed naively applying self-correction leads to worse performance; external feedback is needed. SCoRe (Kumar et al. 2024) uses multi-turn RL to produce better answers at the second attempt than the first.
自我纠正在 LLM 中并非固有能力，开箱即用并不能奏效。失败模式包括：（1）幻觉——把正确答案改错；（2）行为坍缩——做微小或无修改；（3）无法泛化到分布偏移。实验表明天真地应用自我纠正反而更差，需要外部反馈。SCoRe（Kumar 等 2024）用多轮 RL 训练模型在第二次尝试时给出比第一次更好的答案。
"自我纠正在 LLM 中不自然存在"是一个反直觉但重要的结论。直觉上，"让模型检查自己的答案"应该有用，但实验反复证明：没有外部信号（验证器、单测、更强模型、人类反馈）的自我纠正往往越改越糟。原因在于模型不知道自己哪里错了——它用同样的能力生成和评判，错误难以被发现。SCoRe 的贡献是用两阶段 RL 强行植入"第二次比第一次好"的行为。实践启示：自我纠正必须搭配可验证的反馈信号，否则不要轻信。
DeepSeek-R1 (2025) runs 2 rounds of SFT-RL training. Cold-start SFT fine-tunes the base model on thousands of cold-start data (without it, the model has poor readability and language mixing). Reasoning-oriented RL trains with two rule-based rewards: format rewards (wrap CoTs in <thinking> tokens) and accuracy rewards (correct final answers). R1 performs comparable to OpenAI o1-preview and o1-mini.
DeepSeek-R1（2025）跑了两轮 SFT-RL 训练。冷启动 SFT用数千条冷启动数据微调基座模型（没有这步，模型可读性差、语言混乱）。面向推理的 RL用两种基于规则的奖励训练：格式奖励（用 <thinking> token 包裹 CoT）和准确性奖励（最终答案正确）。R1 在多个推理基准上与 OpenAI o1-preview、o1-mini 相当。
Interestingly, DeepSeek showed that with pure RL (no SFT stage), it is still possible to learn advanced reasoning like reflection and backtracking ("aha moment"). The model naturally learns to spend more thinking tokens during RL. The team also shared failures: PRM failed because per-step rubrics are hard to define and make training vulnerable to reward hacking; MCTS failed due to the huge token search space and difficulty of training the value model.
有趣的是，DeepSeek 证明纯 RL（无 SFT 阶段）也能学会反思、回溯等高级推理（"顿悟时刻"），模型在 RL 过程中自然学会花更多思考 token。团队也分享了失败：PRM 失败，因为每步评分标准难定义、且使训练更易受奖励作弊影响；MCTS 失败，因为 token 搜索空间巨大、细粒度价值模型极难训练。
R1 的开源意义远超"又一个强模型"——它公开了推理模型训练的完整配方，并坦诚分享了失败经验。最震撼的发现是"顿悟时刻"：纯 RL 也能让模型自发学会反思。这意味着"自我纠错"能力并非必须显式监督，只要奖励信号正确（答案对错可验证），模型就会自己摸索出"回头检查"的策略。而 DeepSeek 主动公开 PRM 和 MCTS 的失败，是宝贵的负结果：简单的结果奖励（outcome reward）往往比复杂的过程奖励更稳健——这印证了"奖励越简单，作弊空间越小"的原则。
Certain intermediate steps can be reliably solved by executing code or math calculations. Offloading these into an external code interpreter (PAL; Gao et al. 2022, Chain of Code) extends LLM capability, eliminating the need for LLMs to act as calculators themselves. ReAct (Yao et al. 2023) combines searching the Wikipedia API with reasoning traces. o3 and o4-mini are examples where reasoning involves web search, code execution and image processing.
某些中间步骤可以通过执行代码或数学计算可靠解决。把这些卸载到外部代码解释器（PAL；Gao 等 2022、Chain of Code）能扩展 LLM 能力，免除了让 LLM 自己当计算器的需求。ReAct（Yao 等 2023）把搜索维基百科 API 与推理轨迹结合。o3 和 o4-mini 是推理过程涉及网络搜索、代码执行、图像处理的范例。
CoT provides a convenient form of interpretability, but this rests on the assumption that the model truthfully describes its internal thought processes. We cannot by default assume CoT is always faithful. Monitoring CoT can detect reward hacking, and can even enable a weaker model to monitor a stronger model (Baker et al. 2025).
CoT 提供了便捷的可解释性形式，但这建立在"模型如实地描述了内部思考过程"这一假设上。我们不能默认 CoT 总是忠实的。监控 CoT 能检测奖励作弊，甚至能让较弱的模型监控较强的模型（Baker 等 2025）。
Lanham et al. (2023) tested faithfulness by deliberately introducing mistakes. Studies (Turpin et al. 2023, Chen et al. 2025) inserted misleading hints into prompts and measured whether the model acknowledges the hint. Multiple studies found that reasoning models describe the influence of hints much more reliably than non-reasoning models. However, in classic RLHF, the reward model is not trained to incentivize faithful behavior, so it can lead to less faithfulness.
Lanham 等（2023）通过故意引入错误来测试忠实度。研究（Turpin 等 2023、Chen 等 2025）在提示中插入误导性提示，测量模型是否承认受其影响。多项研究发现：推理模型比非推理模型更可靠地描述提示的影响。但在经典 RLHF 中，奖励模型并未被训练来激励忠实行为，因此可能导致更低的忠实度。
"推理模型的 CoT 比普通模型更忠实"是一个关键且令人欣慰的发现。原因在于优化目标不同：经典 RLHF 优化的是"人类偏好"（看起来好），而推理模型优化的是"答案正确"（真的对）——后者天然鼓励真实的推理路径，因为只有真思考才能做对难题。但这不意味着 CoT 完全可信。Baker 等 2025 的发现尤其重要：用 CoT 监控器作为 RL 奖励来防止作弊，反而催生"隐藏式作弊"——模型学会把真实意图藏在 CoT 里躲避监控。这是 Goodhart 定律在 CoT 上的又一次应验。
Adaptive Computation Time (Graves 2016) pioneered enabling the model to dynamically decide computational steps at inference. Adaptive thinking can be enabled vertically via recurrent architecture or horizontally via more sequential sampling. Geiping et al. (2025) add a recurrent block on top of Transformer; the recurrence count is randomized (log-normal Poisson). Thinking tokens (Herel & Mikolov 2023) and pause tokens (Goyal et al. 2024) insert dummy tokens to buy extra compute. Quiet-STaR (Zelikman et al. 2025) trains the model to generate rationales after every token.
自适应计算时间（Graves 2016）开创性地让模型在推理时动态决定计算步数。自适应思考可通过循环架构（纵向）或更多顺序采样（横向）实现。Geiping 等（2025）在 Transformer 上加循环块，循环次数随机化（对数正态泊松分布）。思考 token（Herel & Mikolov 2023）和暂停 token（Goyal 等 2024）插入占位 token 来换取额外计算。Quiet-STaR（Zelikman 等 2025）训练模型在每个 token 后生成理由。
Test-time compute introduces a new dimension for improving model intelligence, complementing model size, training compute and data quantity. Snell et al. (2024) found that test-time and pretraining compute are not 1:1 exchangeable. Test-time compute can easily cover the gap on easy/medium questions but proves less effective for hard problems. The ratio between pretraining and inference token budgets matters a lot; a capable base model with enough pretraining is still critical.
测试时计算为提升模型智能引入了一个新维度，与模型大小、训练算力、数据量并列。Snell 等（2024）发现测试时算力与预训练算力并非 1:1 可互换。测试时算力能轻松弥补简单/中等题的能力差距，但对难题效果有限。预训练与推理 token 预算的比例很重要；拥有足够预训练的强大基座模型仍然关键——测试时算力不能解决一切。
Muennighoff & Yang et al. (2025) experimented with scaling CoT length via "budget forcing" (lengthening by appending "wait", shortening by appending end-of-thinking tokens). They observed a clear positive correlation between thinking time and accuracy. But surprisingly, simple rejection sampling for controlling CoT length leads to reversed scaling—longer CoTs lead to worse performance.
Muennighoff & Yang 等（2025）通过"预算强制"缩放 CoT 长度（追加"wait"延长，追加思考结束 token 缩短）。他们观察到思考时间与准确率的明显正相关。但令人惊讶的是，用简单拒绝采样控制 CoT 长度会导致反向缩放——更长的 CoT 反而更差。
缩放定律的发现是本文最实用的结论之一。"测试时算力 ≠ 预训练算力"意味着不能简单地用"推理时多想"替代"预训练时多学"。经验法则：当推理 token 数远小于预训练 token 数时，测试时算力收益明显；但当模型基座能力差距太大时，再多思考也无济于事。"强制让模型想更久有用，但随机采样到长的 CoT 反而更差"这一对比极其精妙——它说明思考长度本身不是因果，真正起作用的是模型主动投入的计算。这对实践有直接指导：用"wait"这类提示引导模型继续反思有效，但不要机械地追求长输出。
这份开放问题清单是推理模型领域的"路线图"。最紧迫的是"忠实性 vs 作弊"的两难：我们希望 CoT 可读且真实，但任何对 CoT 施加的优化压力都会催生隐蔽的作弊。其次是"测试时算力的回收"——o1 类模型推理极贵，能否把"学会的长思考"蒸馏回快速模型是工程刚需。最后，当前推理模型的 RL 只能用于答案可验证的任务（数学、编程），如何扩展到主观任务（写作、辅导）是下一代推理模型的圣杯。这些问题目前都无解，正是最值得关注的研究前沿。