#mixture-of-experts

Hugging Face authors have released a handbook for training and serving Mixture-of-Experts models with Megatron-LM, aimed at engineers who have trained a dense model and want to move to MoE. Chapter 2 is a runbook covering hardware testing, building the image, training a dense baseline, and then training a first MoE on one node to compare against that baseline. Chapter 3 takes the MoE layer apart—router, routing, load balancing, experts, block layout, training and systems—giving the Megatron-LM code, what to watch, a recommendation, and what frontier MoEs chose for each part; Chapter 4 follows the model into serving and RL, and chapters 3 and 4 require no GPU. The release includes the handbook, a gallery of 55 figures, the scripts, and every recommended flag collected in one Megatron-LM block; the authors say the lessons draw on roughly 800k GH200 GPU-hours on the Alps cluster at CSCS, label results from their own runs, cite papers, configs or code for frontier-model claims, and mark toy simulations as such.

Boson AI announced Higgs Realtime Duplex, a 45B mixture-of-experts multimodal model designed to process audio, video and text in 160 ms chunks while listening, watching and speaking concurrently. Unlike turn-based voice models, it keeps thinking, calling tools and delegating harder problems to a larger backend model as background spans in the same stream, and can begin speaking before staged tool results are complete. The model is not open weights: access to the Duplex API is waitlisted, while the earlier turn-based Higgs Realtime is available now. Boson&\x27;s reported comparison shows the model competitive with, but on five of seven benchmarks 1–5 points behind, OpenAI&\x27;s best settings, with turn-taking its weakest area.