#hallucination

Researchers at the Idiap Research Institute, in joint work with Logitech, have released DialFork, a benchmark for what they call "procedural hallucination": conversational agents that produce fluent, factual turns and correct tool calls but stop following the prescribed dialogue procedure. DialFork was built from simulated dialogues across 24 tasks and 12 domains; the authors replay each correct dialogue and inject an instruction at a chosen turn that makes the agent drift, producing more than 36k dialogues with known drift start points, checked by 30 human annotators and an LLM judge calibrated on their judgments. It defines three tasks—locating the drift turn in a dialogue, measuring procedural distance between a correct dialogue and its hallucinated versions, and comparing sets of dialogues as distributions—with first results reported for embedding-based distances, streaming detectors, and zero-shot LLMs. Data, code, paper, all paper predictions, and human judgments are open, and each task has a track on a public Hugging Face leaderboard that accepts a CSV of predictions; the authors describe DialFork as the first benchmark for this axis "to our knowledge."