NanoFlow: Towards Optimal Large Language Model Serving Throughput
Published in Symposium on Operating Systems Design and Implementation (OSDI), 2025
NanoFlow improves LLM serving throughput by exploiting intra-device parallelism through operation co-scheduling.
Recommended citation: Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. "NanoFlow: Towards Optimal Large Language Model Serving Throughput." Symposium on Operating Systems Design and Implementation (OSDI), 2025.
Download Paper