With the continuous growth of large model scale and service demand, reducing the high cost of autoregressive decoding has become a major pain point in the industry. To address this, the Tencent technical team officially open-sourced the unified training framework AngelSpec, aiming to comprehensively improve the inference throughput of large models in real-world scenarios through multi-scheme collaboration and workload heterogeneity adaptation.

Considering the differences in text characteristics under different application scenarios, AngelSpec adopts differentiated training strategies. For high-entropy, open-ended multi-turn dialogue tasks, the system employs a lightweight and stable multi-token prediction mechanism (MTP), combined with training-time testing (TTT) and end-to-end total variation loss, enabling it to effectively adapt to the state distribution during autoregressive expansion; while for code generation and mathematical reasoning tasks with stronger structural constraints, it enhances performance through a dedicated block diffusion model, fully leveraging the potential of long-span predictable sequences.

image.png

In terms of architecture design, the framework introduces an innovative block diffusion framework called DFly. It optimizes target feature utilization and intra-block dependency modeling while maintaining high-throughput parallel generation by combining a hybrid target conditional encoding backbone with a previous conditional autoregressive head. Additionally, the system incorporates a dynamic verification mechanism that adaptsively adjusts the verification depth at runtime based on current online load, hardware conditions, and prefix confidence, achieving a balance between computational resources and throughput efficiency.

image.png

In system testing of the Hy3 model series, the solution equipped with DFly demonstrated significant performance advantages. In all tests with concurrency ranging from 4 to 64, this solution achieved the highest average throughput, showing significant improvements over traditional autoregressive decoding, providing solid open-source tools support for the engineering deployment and efficient inference of large models.

Project: https://github.com/Tencent/AngelSpec

Paper: https://arxiv.org/pdf/2607.25852