Researchers introduced ALEM, a JAX-based benchmark for open-ended multi-agent coordination, and evaluated 13 LLMs, finding they average only 6% normalized return, while zero-shot Gemini 3.1 Pro matches a trained MARL agent on the hardest setting. This benchmark fills a gap in evaluating LLMs' ability to coordinate in long-horizon, open-ended environments, highlighting that coordination is a distinct bottleneck beyond individual task competence, which has implications for deploying LLMs in multi-agent systems. The ALEM benchmark features nine procedurally generated levels with controllable coordination demands, and the study includes ablations showing communication has the largest effect on performance. The paper, code, and interactive traces are publicly available.
Background
Multi-agent coordination involves multiple agents working together to achieve shared goals in a shared environment. Prior benchmarks often focus on single-agent tasks or short, structured interactions. ALEM builds on Craftax-like dynamics, requiring agents to explore, communicate, trade, craft, and fight in open-ended worlds.