ARTICLE · 1206357
AI训推框架周报 · 2026-10-11~10-18
AI训推框架周报 · 2026-10-11~10-18GLM5.3 Flash 默认 FP8 KV cache，E2E 吞吐 +9.8%，量化 KV 从可选转默认——长上下文零成本加速[1] 在线再量化 API 扩展至 MoE（NVFP4 modelopt/CT/quark），服务侧按需换量化下沉到主流稀疏模型；支持 compressed-tensors MXFP4 MoE（FP8 激活）[3][13] 上线 DeepSeek-V4.1-Flash 的 PCP 支持，新一代解耦式服务对关键模型适配[6] 对 DeepSeek-V4/V4.1 VL 改为流式加载权重，巨型 VL checkpoint 不再整档入内存，降启动显存与加载时延[12] Kimi K3 专用 decode 走 CuTeDSL 优化（GB300，1.5-2x 提速）；mamba 对齐 block-table gather 融合为单 Triton 内核（7→1，6x 提速）[14][15] 新增可插拔 KVCacheConfigBuilder，KV 缓存分配策略从引擎核心解耦，厂商/模型可注入规划[10] 流式解析 MiniMax M2 超长工具参数（16KB 调用由 5.9s 降至约 26ms），在线 agent 服务受益明显[16] 修复 Mamba2 chunk scan 在 sm90 的 NaN、Gemma 4 encoder CUDA graph 启动期 OOM、拒绝 float16/bfloat16 KV dtype 不匹配等生产问题[17][18] Rust 前端原生提供 /v1/tokenize、/v1/detokenize、/flush_cache 等接口，核心 HTTP 批量迁移，瘦身且长稳[9] DSv4.1 默认启用 decoder bounded replay；支持 DSV4 decode 在 ROCm unified KV 上的 radix cache（bf16/fp8）[19][20] MiMo 启用原生 MXFP4 路由与 CuTeDSL 融合；为 FlashInfer MegaMoE 引入 NVFP4 W4A16 与 SiTU 支持，降低巨型 MoE 显存与访存压力[21][22] DeepSeek-V4.1 用 DeepSelect 做 Hopper 全量 top-k decode；新增 sgl-router 原生 gRPC ChatComplete 及剩余推理 RPC[23] 修复 DeepSeek-V4 被 retract 请求在 prefill 阶段重算而非崩溃、CUDA Graph 捕获尾钩子 autograd retention 等问题[24] 新增 FLUX.3 图像生成、MiniCPM-V-4.7 模型支持；diffusion 融合 Wan VAE conv bias epilogue 并切块 RMSNorm+SiLU kernel[25] 默认启用 KV cache manager V2 与 Python transceiver（BREAKING），引擎级 KV 生命周期管理重构，升级部署需适配 breaking 变更[4] v1.4.0rc0 预发布：GLM-5.3-Flash/K-EXAONE-2 新模型、sparse KV cache、Mooncake store pool，并新增 First-N/Last-N 混合精度 KV caching（KVCM V2）[5][26] 支持 DeepSeek-V4.1-Flash、NVFP4 GLM-5.3-Flash checkpoints；MiniMax-M3 dense 层走 paged attention[27][28] DeepSeek-V4 TP decode all-reduce 脱离 NCCL，降低同步开销与抖动；统一经 TRTLLM_CACHE_DIR 做缓存管理[29] 为统一 KV 共享新增 Mooncake store 后端；回收空闲 eager CUTLASS MoE workspace 优化显存[30] 修复 SM120 DSA KV append 缺失 _cached_stride_factor 崩溃、speculative beam 的 d2h 竞态[31] 新增稀疏 KV offloading（固定 GPU hot pool + 跨层 prefetch），长上下文/agentic 负载释放 HBM 硬约束[32] AMD gfx950 落地 Gluon 动态 MXFP4 A4W4 MoE 与 QSA 稀疏注意力；gfx1250 上加速 DeepSeek-V4.1-Flash、Kimi K3 AttnRes[33][34] 新增 Hopper block-FP8 DeepGEMM 后端、small-M FP8 GEMM 与一次性 per-token FP8 量化器[35] 缩减 MegaMoE workspace 并为 K3 EP16 启用；升级 FlashInfer 至 0.7.2[36] 修复 SM90 GDN prefill 在 QKV split 时清零 padding、按请求抽取的 token 采样受 prefill 分块影响的正确性问题[37] per-tenant/per-model 并发隔离（bulkhead）防单租户耗尽共享资源；opt-in 精确匹配响应缓存降低重复推理开销[38][39] 新增 Reflection 的 Beam 服务支持、Muse Spark Open 多模态支持（图像/视频/音频）；跨工作器组加权流量拆分[40] tokenizer 支持 opt-in byte-level 解码加速；rollout 过滤前上报轨迹奖励提升可观测性[41] MTP 支持 Qwen3.8 的 DFlash2；新增 stop string 包含与 reasoning scope 控制[42] 统一 XGrammar 结构化输出与常规采样路径，结构化生成兼容性与性能一致[43] 发布 v0.7.2 正式版：MoE/MLA/GEMM/线性注意力覆盖扩展，DeepSeek-V4 NVFP4 稀疏 MLA（B200/B300）及 SM120 W4A8 后端[44] 上线 FROST 低延迟 MX MoE 后端（Blackwell）；SM120 Frost MoE Grouped GEMM 支持 Swap-AB / MXFP8 / NVFP4 等多格式[45][46] Blackwell MSA Top-K 流式选择提速 1.39-1.41×；优化稀疏 SM107 MegaMoE decode[47][8] 融合 NVFP4/FP8 GEMM + 运行时 BF16 LoRA 校正；MNNVL MoE all-to-all cake 后端融合量化/向量化 combine[48][49] 修复 Blackwell MSA FP8 decode 挂起（两阶段 mbarrier lead）、NVFP4 decode 中 tcgen05.mma 前 fence SMEM stores[50] 为 fp4 与 e8m0 增加原生类型支持，低精度算子拥有编译器级一等公民身份[51] 用 target-aware 算术向量化替换 NVIDIA SLP；恢复非对齐 warp-specialized barriers[52] KERNELS 暴露 top-k 与归约编译控制；MXFP8 转换时吸收 NVFP4 row scales[53] 发布 v3.6.0，对齐上游 Triton 3.6.0 并支持 CANN 9.2.0，昇腾矩阵乘与 layout 优化[54] 新增自动 constexpr 复用并默认开启 autotune，减少昇腾算子重复编译[55] ROCm 支持 DeepSeek NSA 前向与解码、mHC pre/post kernels；昇腾支持独立 AIV 执行[56][57] CUDA codegen 前拒绝不支持的 GEMM dtype 组合，提前拦截非法配置[58] 版本号升至 3.0.1（维护性）；CuTe/SM120 支持自动 num_splits[59] SM100 统一 hd256 反向（GQA、varlen、deterministic）；修复 paged KV-cache 的 CUDA Graph capture[60] CuTeDSL 增加 MLIR_RAW 并保留 clean IR 源码位置，DSL 编译产物可调试性增强[61] cute 与 pallas 后端提升为 beta；cutedsl 为 FlexAttention 增加 envelope plan 数据、匹配器与搜索面[62] cutedsl 为 Blackwell MXFP8 生成原生 lowering（full-K scaled dots）[63] Ascend v2 worker 支持 C8 MXFP8 KV cache 量化；集成 DeepSeek-V4 FP8 KV 的 mixed_quant_sparse_flash_mla 算子[64][65] 支持 GLM-5.3-Flash Mooncake 分层 KV 传输；ECMooncakeConnector 支持 Ascend Direct[66] 默认 Model Runner V2 覆盖全配置；GLM-5.3-Flash 启用 torch.compile（backbone+MTP）[67] 支持 DeepSeek V4.1 decoder 有界重放；修复 C8 K-scale 在 KV 传输中的损坏、Kimi A5 draft KV cache 与 FP8 MLA 输出[68][69] fused_deep_moe 的 max_experts 上限 512→1024；新增可移植 Gemma RMSNorm API[70] chain 推测采样 kernel 增加 block 校验模式[71] 支持 Ascend NPU 上的 Delta 权重同步（HCCL 传输），解锁国产 NPU RL 闭环[11] checkpoint 保存前同步重叠参数；vLLM 回放前校验 routing 复用，避免回放失配[72] Blackwell bf16 rollout 改用 SGLang 的 Triton MoE runner；修复 FSDP2 CPU offload 导致的参数损坏[73] 通过 verl-hardware-plugin 支持 TPU GRPO（TorchTitan+vLLM），TPU GRPO 示例提速约 14%[74] Disaggregated Inference End2End 落地，训练框架原生打通 PD 分离完整推理链路，训练→在线推理无缝衔接[75] 支持 fp8/fp4 权重流式 checkpoint 反量化，避免整档常驻显存；min-memory DSA attention kernels[76][77] Shortcut MoE 与宽残差支持推理 CUDA Graph；CP 接入 MIMO[78] --save-retain-interval 不再误删更新的 checkpoint；使 MoE sigmoid 推理路由在 bit-exact 输出下优化[79] Automodel 策略支持 rollout routing replay（R3）；启用 vLLM PIECEWISE CUDA graphs 到两个 eager release recipes[80][81] TRT-LLM prefill/decode 分离用于 GRPO rollout；external-vLLM 支持持久化 judge model 池[82] data-plane 向量化 captured-row staging 张量；修复 vLLM 集成中 Super Omni 末层视觉 LayerNorm 丢失[83] 发布 v1.16：收紧 peft≥0.17、torch≥2.6 依赖底线，为统一工具链做清理[84] 新增 Tree attention；异步 trainer 在 num_train_epochs 停止时保存 final step[85] 修复 FSDP2 vLLM colocate 权重同步对 tied-embedding 模型的共享 embedding 对齐[86] 支持 GLM-5.3 Flash 训练；升级 SGLang 至 0.5.21 并对齐 CUDA 13 运行时，保证训练-推理一致性[87] 支持 128k Qwen 上下文的 fully-async coding-agent RL[88] 支持 GLM-5 的 per-expert checkpoint 与 EP；增加 deepseek-v4 NPU kernels[89] 停止 NPU GDN Triton kernel 按 batch 长度重编译，消除动态长编译开销[90] [1] https://github.com/vllm-project/vllm/commit/2f18d13cf083c4db8a6d140d8475fbfcffc084fd [2] https://github.com/vllm-project/vllm/commit/bc3ddb198a0d5eb864bf484c4719864b671cc93b [3] https://github.com/vllm-project/vllm/commit/c9bfa574ce690e45d1da5d506d17ed1586dcf9dd [4] https://github.com/NVIDIA/TensorRT-LLM/commit/afc1fa27447464405e971b39ff3c7df63cc85e71 [5] https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.4.0rc0 [6] https://github.com/vllm-project/vllm/commit/fa75d3456d60a5ffe923818c4b495295675b38e1 [7] https://github.com/THUDM/slime/commit/5bec7a10859f29ae1bf2c9ddd5a67aa70ead0fac [8] https://github.com/flashinfer-ai/flashinfer/commit/0a0cecb50b1250f0b609a9c51f1b25816ba073b8 [9] https://github.com/sgl-project/sglang/commit/0d13efb2b8df90487ab7fdb7867a298b6c0c30ec [10] https://github.com/vllm-project/vllm/commit/b5445874080ce85c575f2bcb847852235f1af7bc [11] https://github.com/verl-project/verl/pull/8272 [12] https://github.com/vllm-project/vllm/commit/026f6ab2689397e64c1f53a5ac57d37c4dc46b67 [13] https://github.com/vllm-project/vllm/commit/984c242ede0beaf1c3ba9eb4c01b98bed8973eac [14] https://github.com/vllm-project/vllm/commit/642c18d64c4cb6db3ded8fbe1786718cd4005fcb [15] https://github.com/vllm-project/vllm/commit/c825e1d235896694ca50160286440859ed46d7d3 [16] https://github.com/vllm-project/vllm/commit/b088eba94431b711ffaa2f356e33361cf38b8fc5 [17] https://github.com/vllm-project/vllm/commit/783380af397ceeae91a40f4f759439eb8c1ab690 [18] https://github.com/vllm-project/vllm/commit/48f17b77bba9430ffedc3297186a2c9d932d94c7 [19] https://github.com/sgl-project/sglang/commit/360382454170073d54c202745e9884ee51f72fa3 [20] https://github.com/sgl-project/sglang/commit/f05da43d44067e4e4f95302b6d3bd40f7deb3b1d [21] https://github.com/sgl-project/sglang/commit/c54ba66cd40f32a6a663b537d390570ce2e9fc4e [22] https://github.com/sgl-project/sglang/commit/7098ec387c9f9d9f85a0c2c16bc1233f40f1d407 [23] https://github.com/sgl-project/sglang/commit/b50fdeb0ef9be35916b88efb5a5df2cb8e5ac31d [24] https://github.com/sgl-project/sglang/commit/43e4e84731119e7ebf5a614826c873cae64e8e51 [25] https://github.com/sgl-project/sglang/commit/09b7383624c11a60e50af75ab40f65d4d4a99939 [26] https://github.com/NVIDIA/TensorRT-LLM/pull/20452 [27] https://github.com/NVIDIA/TensorRT-LLM/commit/16df7eaf251ab981021f355a0f7608b9248e945c [28] https://github.com/NVIDIA/TensorRT-LLM/commit/4271011718a350bbf40f296127879d3713502936 [29] https://github.com/NVIDIA/TensorRT-LLM/commit/bf0e298f11ec7fff43b86fc559cf6f64181323c8 [30] https://github.com/NVIDIA/TensorRT-LLM/pull/20473 [31] https://github.com/NVIDIA/TensorRT-LLM/commit/9fd891783959eed960a6c58144241db38f9b1580 [32] https://github.com/lightseekorg/tokenspeed/commit/b9c265eb247ba72d868d0e78d1041502e86c7632 [33] https://github.com/lightseekorg/tokenspeed/commit/90909ffb1cfe578eb85b43462fcc0aa0803a133b [34] https://github.com/lightseekorg/tokenspeed/commit/34ee21a4b2311ee1d36ed3cdf6efebdecfbc36b1 [35] https://github.com/lightseekorg/tokenspeed/commit/73fb9f8b73fe77b978b061bb06189e15b2ff1611 [36] https://github.com/lightseekorg/tokenspeed/commit/c8225898a97ba31076a4b0b11fd6b37eb6ef9794 [37] https://github.com/lightseekorg/tokenspeed/commit/dc4e6e365260dc84f68b6a10060844d32af424ae [38] https://github.com/smg-project/smg/pull/3613 [39] https://github.com/smg-project/smg/pull/3607 [40] https://github.com/smg-project/smg/pull/3622 [41] https://github.com/smg-project/smg/commit/94fd83e79f191844ab863efbc097c3e36d0dd9c3 [42] https://github.com/ModelTC/lightllm/commit/d6a34eb0c30bab887ae745ba897e4e0ca6d7fa2b [43] https://github.com/ModelTC/LightLLM/commit/3d9415f44470202b9bd24e9b5b6f32ded726413a [44] https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.7.2 [45] https://github.com/flashinfer-ai/flashinfer/pull/6626 [46] https://github.com/flashinfer-ai/flashinfer/commit/9073fc413404126f8ff75e2a6cea8d2ff3ba4a88 [47] https://github.com/flashinfer-ai/flashinfer/commit/ba4defb942f35ecc3320f81803188a59f89e4f92 [48] https://github.com/flashinfer-ai/flashinfer/commit/649eb1060905ff26e0b9446420435c632c1aa735 [49] https://github.com/flashinfer-ai/flashinfer/commit/52b0de0b815419fbc890a0d049a243f01b4e231a [50] https://github.com/flashinfer-ai/flashinfer/commit/d0546b42398d8137b85271e953704926adc09b23 [51] https://github.com/triton-lang/triton/commit/4283dc7bb52e6977b07ef2465e5411e5aaa7b036 [52] https://github.com/triton-lang/triton/commit/78e80b9e9f7bf720945f828947e6e196284f7e6a [53] https://github.com/triton-lang/triton/commit/faafaf91de0bf424cf38133a9bdfbc4d8315a130 [54] https://github.com/triton-lang/triton-ascend/releases/tag/v3.6.0 [55] https://github.com/triton-lang/triton-ascend/pull/2612 [56] https://github.com/tile-ai/tilelang/commit/fd28d886f3d4ce0c9a6778e93147564bc043d9a3 [57] https://github.com/tile-ai/tilelang/commit/390ecf34918ce1e4d7d9967df254f507b10557d8 [58] https://github.com/tile-ai/tilelang/commit/7a8ded9f1d91b0693fa82467656311947cfd8922 [59] https://github.com/dao-AILab/flash-attention/commit/bb25ba3ce4b1c8df5d4ef6a94807f0d3910599ca [60] https://github.com/Dao-AILab/flash-attention/commit/2b00d76f222c842b245fd98847223c898bc476db [61] https://github.com/NVIDIA/cutlass/commit/56ca28adb32ffadd097566545b8f7d83b652e998 [62] https://github.com/pytorch/helion/commit/fdaa61031a0205978a5521afe210f328ad9bf739 [63] https://github.com/pytorch/helion/commit/39bc6e873790b61741318a1af60abf0e9b26ae5b [64] https://github.com/vllm-project/vllm-ascend/commit/fe686bc79a81bcc2cbcff5b1717b07ed67a66404 [65] https://github.com/vllm-project/vllm-ascend/commit/7a50474d372e5aa04231cfeaab6a089daf179826 [66] https://github.com/vllm-project/vllm-ascend/commit/c4622400eb45bc877f8eb050a84ed47b18b38ac4 [67] https://github.com/vllm-project/vllm-ascend/commit/93f8d0fcacded2b3e1ce8824b54423694f7baac6 [68] https://github.com/vllm-project/vllm-ascend/commit/a58da927ab5e7875acbf748f10ebb0d01e02cfb8 [69] https://github.com/vllm-project/vllm-ascend/commit/e6c0f4b7d2e2be253ecfc0d0d34f4864ff7fe107 [70] https://github.com/sgl-project/sgl-kernel-npu/commit/d9813afe15ae90ec61b38f924c53a62b30e9f3fc [71] https://github.com/sgl-project/sgl-kernel-npu/pull/864 [72] https://github.com/verl-project/verl/pull/8264 [73] https://github.com/verl-project/verl/commit/e5a627cf55c94d85406556854c2c1a22db4ff5e7 [74] https://github.com/verl-project/verl/commit/f95be0226a65d3f335901ee6703f974fe82f840d [75] https://github.com/NVIDIA/Megatron-LM/pull/6675 [76] https://github.com/NVIDIA/Megatron-LM/commit/6e7e6ed435e41af04fc41e7007ef862049153fdc [77] https://github.com/NVIDIA/Megatron-LM/commit/1e3c0c62fbb79c26d083f506f7da46f5f7add49e [78] https://github.com/NVIDIA/Megatron-LM/commit/e7c933671e9ab3404ef707a5fa94109212f87522 [79] https://github.com/NVIDIA/Megatron-LM/commit/438ef0025d805bdae486ec1bfaef2002ec3cf4d5 [80] https://github.com/NVIDIA/NeMo-RL/pull/4458 [81] https://github.com/NVIDIA/NeMo-RL/pull/4568 [82] https://github.com/NVIDIA/NeMo-RL/commit/339fad1f7463ad5bf3d1830a4a13e28dd3dd2c26 [83] https://github.com/NVIDIA/NeMo-RL/commit/537e056d121960dd359ef8c771a1638119aafeb6 [84] https://github.com/huggingface/trl/pull/7756 [85] https://github.com/huggingface/trl/commit/ea4d31033e216377bdf66576d426e48187d3eb06 [86] https://github.com/huggingface/trl/commit/52c9ebd3aaf0363e0c0b5b27ad3cd4b5d820b5cc [87] https://github.com/THUDM/slime/commit/4ac368bbb196bdb7637c8f36e2be91b56277b335 [88] https://github.com/THUDM/slime/commit/2f4280e3b0e6934b954db93153bd79d9c9312830 [89] https://github.com/ByteDance-Seed/VeOmni/commit/0832477680a5a9f8f4209211d6110cdb199dd086 [90] https://github.com/ByteDance-Seed/VeOmni/commit/9251d15990beb454b15ffe0a41a2e558de49b271
导读：本周的低位宽量化正式从"可选开关"变成"框架默认配置"——GLM5.3 Flash 默认 FP8 KV、在线再量化下探到 MoE，老卡与国产/AMD 平台同吃红利；解耦式推理（PD 分离）与稀疏推理（SSM/DSA）端到端打通，训练框架 Megatron 甚至原生上线了 Prefill/Decode 分离的完整推理链路；旗舰稀疏模型适配持续冲刺，同时 NVIDIA 的 Rubin（SM107）与 AMD 的 gfx950/gfx1250 在到货前就完成了多层软件铺底；SGLang Rust 前端、vLLM 可插拔 KV 规划代表着引擎从"跑通新模型"转向"服务面长稳"；训练-推理的增量/流式权重交接与 RL 回放一致性也一起收口。
重点更新
量化 KV Cache 从"可选"转"默认" ：GLM5.3 Flash 默认 FP8 KV 端到端吞吐 +9.8%，FP8 QSA 下探到 Ampere 老卡，在线再量化延伸至 MoE，Triton 原生支持 fp4/e8m0。低位宽量化的显存红利进入"开箱即用"规模化阶段[1][2][3]
解耦式推理 + 稀疏推理端到端打通：TensorRT-LLM 默认启用 KV cache manager V2（BREAKING）并带来 sparse KV + Mooncake store，Megatron 原生上线 Disaggregated Inference End2End。PD 分离 + 稀疏注意力 + KV offload 成为支撑超长上下文与 agentic 负载的标准底座[4][5][6]
旗舰稀疏模型适配冲刺 + 下一代硬件布线：GLM-5.3-Flash / DSv4.1-Flash / Kimi K3 多引擎同步完成适配，同时 Rubin（SM107）与 AMD gfx950/gfx1250 在到货前已多线布软。NVIDIA/AMD 双代际代际切换断档风险持续收窄[7][8]
推理引擎工程化收口：SGLang Rust 前端原生下沉 tokenize/detokenize 等核心接口，vLLM 把 KV 缓存规划改为可插拔、统一 JIT 编译池。引擎重心转向服务面长稳与多机运维一致性[9][10]
训练-推理一体化以增量/流式交接收口：verl 在昇腾 NPU 上打通 Delta 权重同步（HCCL），vLLM 对 VL 权重改流式加载，Megatron 支持 fp8/fp4 流式反量化，NeMo-RL 上车 rollout routing replay。权重交接不断链、回放不失配[11][12]
