Huawei has open-sourced OpenPangu-2.0-Flash, a Mixture-of-Experts (MoE) large language model with 92 billion total parameters and 6 billion active parameters, supporting a 512K token context length. The release includes model weights, inference code, and training operations. This open-sourcing provides the community with a high-performance, long-context MoE model from a major tech company, potentially lowering the barrier for researchers and developers to experiment with large-scale MoE architectures. It also signals Huawei's growing involvement in the open-source AI ecosystem. The model uses an MoE architecture where only 6B of the 92B parameters are activated per token, enabling efficient inference. It achieves a 512K token context window, and a larger Pro variant (505B total, 18B active) is slated for release in July.
Background
Mixture of Experts (MoE) is a neural network architecture that divides work among specialized sub-networks called experts, activating only a subset per input to improve efficiency. A longer context length (e.g., 512K tokens) allows the model to process large documents or conversations in one pass, which is valuable for tasks like long-form question answering and code analysis.