#AI Research

Researchers introduced ALEM, a JAX-based benchmark for open-ended multi-agent coordination, and evaluated 13 LLMs, finding they average only 6% normalized return, while zero-shot Gemini 3.1 Pro matches a trained MARL agent on the hardest setting. This benchmark fills a gap in evaluating LLMs&\x27; ability to coordinate in long-horizon, open-ended environments, highlighting that coordination is a distinct bottleneck beyond individual task competence, which has implications for deploying LLMs in multi-agent systems. The ALEM benchmark features nine procedurally generated levels with controllable coordination demands, and the study includes ablations showing communication has the largest effect on performance. The paper, code, and interactive traces are publicly available.