推理优化

Large transformer models are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale. This post looks into several approaches for making transformer inference more efficient.

大型 Transformer 模型强大但训练和使用都极其昂贵。推理在时间和内存上的极高成本，是把强大 Transformer 用于大规模真实任务的主要瓶颈。本文探讨让 Transformer 推理更高效的几种方法——这些技术（KV Cache、量化、蒸馏、剪枝、稀疏化、MoE）共同支撑了今天 LLM 的低成本部署。

推理为什么难？

Two main factors contribute to the inference challenge: Large memory footprint—both model parameters and intermediate states (e.g. KV cache) are needed in memory; for a batch size of 512 and context length of 2048, the KV cache totals 3TB, 3x the model size! And low parallelizability—autoregressive decoding is hard to parallelize.

两个主要因素造成推理挑战：巨大的内存占用——模型参数和中间状态（如 KV cache）都需驻留内存；例如批大小 512、上下文长度 2048 时，KV cache 总计 3TB，是模型大小的 3 倍！低并行度——自回归解码难以并行。

Goals for inference optimization: reduce memory footprint (fewer GPUs); reduce computation complexity (fewer FLOPs); reduce inference latency (run faster). Methods include parallelism, memory offloading, smart batching, network compression (pruning, quantization, distillation), and architecture-specific improvements.

推理优化的目标：减少内存占用（少用 GPU）、降低计算复杂度（少 FLOPs）、降低延迟（跑更快）。方法包括并行、内存卸载、智能批处理、网络压缩（剪枝、量化、蒸馏）以及针对架构的改进。

蒸馏

Knowledge Distillation builds a smaller, cheaper "student" model by transferring skills from a pre-trained expensive "teacher." The distillation loss minimizes the difference between two softmax outputs with a high temperature T, combined with a supervised objective. DistilBERT reduces BERT parameters by 40% while maintaining 97% performance and running 71% faster.

知识蒸馏（KD）通过把预训练的昂贵"教师"模型的技能转移到更小更便宜的"学生"模型来加速推理。蒸馏损失最小化高温 T 下两个 softmax 输出的差异，并结合监督目标。DistilBERT 把 BERT 参数减少 40%，保持 97% 性能，速度快 71%。

量化

Two common approaches: Post-Training Quantization (PTQ) converts weights to lower precision after training (cheap); Quantization-Aware Training (QAT) applies quantization during training (better performance but needs resources). A key challenge: simple 8-bit PTQ leads to significant performance drop mainly due to the high dynamic ranges of activation. As models grow to billions of parameters, outlier features of high magnitude emerge in all layers—some dimensions can be ~100× larger than most others.

两种常见方法：训练后量化（PTQ）训练后再把权重转成低精度（便宜）；量化感知训练（QAT）训练时就量化（性能更好但需资源）。关键挑战：简单的 8 位 PTQ 会导致显著性能下降，主要因为激活值动态范围大。随着模型增长到数十亿参数，所有层都出现高幅度异常值特征——某些维度可能比其他值大 100 倍。

训练后量化（PTQ）

Mixed-precision: LLM.int8() (Dettmers et al. 2022) keeps outlier activation features (20× larger) in FP16 while quantizing the rest to INT8. Fine-grained granularity: Q-BERT uses group-wise (per-head) quantization; ZeroQuant uses group-wise weights + token-wise activation with fused kernels. Second-order: GPTQ treats the weight matrix as row vectors and iteratively quantizes greedily using Hessian matrices—reducing OPT-175B weights to 3-4 bits.

混合精度：LLM.int8()（Dettmers 等 2022）把异常激活特征（大 20 倍）保留在 FP16，其余量化到 INT8。细粒度：Q-BERT 用分组（按头）量化；ZeroQuant 用分组权重 + token 级激活并融合算子。二阶信息：GPTQ 把权重矩阵视为行向量，用 Hessian 矩阵贪心地迭代量化——能把 OPT-175B 权重压到 3-4 位。

Outlier smoothing: SmoothQuant (Xiao & Lin 2022) smartly smooths outlier features from activations to weights via a mathematically equivalent transformation, then enables W8A8 quantization on both. A hyperparameter α controls how much difficulty migrates from activations to weights; α=0.5 is a sweet spot. Because both resulting matrices become easy to quantize, SmoothQuant has better hardware efficiency than mixed-precision.

异常值平滑：SmoothQuant（Xiao & Lin 2022）通过数学等价变换把激活的异常值特征智能地平滑到权重，然后对两者都做 W8A8 量化。超参数 α 控制从激活迁移多少难度到权重；α=0.5 是甜点。由于变换后两个矩阵都易量化，SmoothQuant 比混合精度有更好的硬件效率。

量化感知训练（QAT）

QAT fuses quantization into pre-training or fine-tuning. One approach treats the full-precision model as teacher and the lower-precision model as student, optimized with distillation loss—Wikipedia or even random tokens can give decent gains. Layer-by-layer Knowledge Distillation (LKD) quantizes the network layer by layer, using the unquantized version as teacher.

QAT 把量化融入预训练或微调。一种方法把全精度模型当教师、低精度模型当学生，用蒸馏损失优化——维基百科甚至随机 token 都能带来不错的收益。逐层知识蒸馏（LKD）逐层量化网络，用未量化版本当教师。

剪枝

Network pruning reduces model size by trimming unimportant weights while capacity remains. Unstructured pruning drops any weight but doesn't work well with modern hardware. Structured pruning maintains dense matrix form with zeros. Magnitude pruning is simplest yet effective—weights with smallest absolute values are trimmed. Iterative pruning alternates prune/retrain multiple times. The Lottery Ticket Hypothesis motivates this: a dense network contains a sparse "winning ticket" subnetwork that can achieve optimal performance in isolation.

网络剪枝通过修剪不重要的权重来减小模型大小，同时保持容量。非结构化剪枝可丢弃任意权重，但与现代硬件配合不佳。结构化剪枝保持含零的稠密矩阵形式。幅度剪枝最简单却有效——剪掉绝对值最小的权重。迭代剪枝多次交替剪枝/重训练。彩票假说为此提供动机：稠密网络中存在一个稀疏的"中奖彩票"子网络，能独立达到最优性能。

For retraining, weight rewinding (reinitialize to earlier training values) outperforms fine-tuning, and learning rate rewinding matches or outperforms weight rewinding. Zhu & Gupta found large sparse models achieve better performance than small but dense counterparts.

关于重训练，权重回退（重置为早期训练的值）优于微调，学习率回退与权重回退相当或更优。Zhu & Gupta 发现大的稀疏模型比小而稠密的模型性能更好。

稀疏化

N:M sparsity (N out of every M consecutive elements are zeros) works well with modern GPUs—Nvidia A100 supports 2:4 sparsity. Permuting columns can provide more options to maintain large-magnitude parameters. SR-STE (Sparse-refined STE) extends straight-through estimation to prevent large mask changes. Top-KAST preserves constant sparsity in both forward and backward passes without dense forward passes. The Scaling Transformer sparsifies both self-attention and FFN, achieving 37× speedup for single-example inference; combined with LSH attention and FFN recurrence it becomes Terraformer.

N:M 稀疏（每 M 个连续元素中 N 个为零）与现代 GPU 配合良好——Nvidia A100 支持 2:4 稀疏。列置换能提供更多选择以保留大幅值参数。SR-STE 扩展直通估计以防止掩码剧变。Top-KAST 在前向和反向都保持恒定稀疏度而无需稠密前向。Scaling Transformer 同时稀疏化自注意力和 FFN，单样本推理加速 37 倍；结合 LSH 注意力和 FFN 循环后成为 Terraformer。

混合专家模型（MoE）

With MoE architecture, only partial parameters are utilized at decoding time, saving inference cost. Expert capacity is adjusted by a capacity factor C. V-MoE adds MoE into Vision Transformer, matching SoTA with half the inference compute. Its Batch Priority Routing (BPR) assigns experts to high-priority-score tokens first, working much better than vanilla routing when capacity is low. Task MoE routes at the task level (not token level)—since routing is static for a fixed task, the server only needs to preload k experts, achieving 2.6× higher peak throughput.

MoE 架构在解码时只利用部分参数，节省推理成本。专家容量由容量因子 C 调整。V-MoE 把 MoE 加入 Vision Transformer，用一半推理算力匹配 SoTA。其批优先路由（BPR）先把专家分配给高优先级 token，在低容量时远胜普通路由。Task MoE 在任务级（而非 token 级）路由——由于固定任务的路由是静态的，服务器只需预加载 k 个专家，峰值吞吐量高 2.6 倍。

架构优化：高效 Transformer 全景

Since self-attention has quadratic time/memory complexity, all efficient transformer models apply some form of sparsity to the dense attention layer (Tay et al. 2020 survey). Categories:

由于自注意力是二次时间/内存复杂度，所有高效 Transformer 模型都对稠密注意力层施加某种形式的稀疏（Tay 等 2020 综述）。分类：

• 固定模式：分块（Blockwise）、局部（Image Transformer）、跨步（Sparse Transformer）。
• 组合模式：把跨步与局部结合；Axial Transformer 沿单一轴施加注意力。
• 可学习模式：Reformer 用 LSH 哈希聚类；Routing Transformer 用 k-means 聚类；Sinkhorn 学习排序。
• 局部+全局：ETC、Longformer、Big Bird 结合局部与全局上下文及随机注意力。
• 低秩：Linformer 把 N×N 降到 N×k；多查询注意力跨头共享 K/V。
• 随机特征：RFA 和 Performer 用核方法近似 softmax，实现线性复杂度。
• 循环：Transformer-XL 跨段复用隐状态；Compressive Transformer 压缩旧记忆。
• 自适应：Adaptive Attention Span 学习最优跨度；Depth-Adaptive/CALM 学习早退出。