4月20日“软件新技术讲坛”学术报告

4月20日“软件新技术讲坛”学术报告 - 傅泳淦高级研究员

Efficient Language Modeling with Hybrid Architectures

时间：2026年4月20日（星期一）14:30

地点：人工智能学院C209室

傅泳淦

NVIDIA Research 高级研究员

摘要

Transformers with attention mechanisms have become the dominant choice for language models (LMs) due to their strong performance and long-term recall enabled by key-value (KV) caches. However, their quadratic computational cost and high memory demands pose significant efficiency challenges. In contrast, state space models (SSMs) such as Mamba offer constant-time complexity and are well suited for hardware efficiency, but they struggle with memory-intensive recall tasks.

In this talk, we present our research on building hybrid architectures that combine the strengths of attention mechanisms and SSMs to achieve both accurate and efficient language modeling. We first introduce Hymba (ICLR’25 Spotlight), a hybrid-head LM architecture that integrates attention heads and SSM heads within the same layer, enabling parallel and complementary processing of the same inputs. This hybrid-head design allows each layer to simultaneously leverage the high-resolution recall of attention and the efficient context summarization of SSMs, increasing the model’s expressiveness in handling diverse information flows.

To further improve real-device efficiency, we introduce Nemotron-Flash (NeurIPS’25), one of the strongest 3B-scale small language models to date. We identify two key architectural factors—depth–width ratios and operator choices—which are critical for small-batch latency and large-batch throughput, respectively, and develop targeted architectural improvements to optimize both. Beyond architectural design, we also enhance LM training with a weight normalization technique that enables more effective weight updates and improves convergence. We hope that the actionable insights and guidelines presented in this work will inform future research on low-latency, high-throughput language models.

报告人简介

Yonggan Fu is a Senior Research Scientist at NVIDIA Research. He obtained his PhD from Georgia Institute of Technology, advised by Dr. Yingyan (Celine) Lin. Before that, he obtained his Bachelor's degree with a dual major in Applied Physics and Computer Science from the School of The Gifted Young at the University of Science and Technology of China in 2019. He is a recipient of the IBM PhD Fellowship and was selected as Machine Learning and Systems Rising Stars 2023. His research focuses on developing efficient foundation models that bring frontier AI to everyday platforms. His research work has been featured as spotlight papers at ICLR (2025 & 2021 & 2021 & 2020), an oral paper at ECCV (2024), and an IEEE Micro Top Pick (2023).

欢迎参加