MathFormer, a 4-million parameter seq2seq model, achieves 98.6% accuracy on symbolic math expansion tasks without any prior mathematical knowledge, suggesting it learns structural token transformations rather than true reasoning. This finding challenges the common assumption that large language models (LLMs) 'reason' mathematically, implying their performance may stem from sophisticated pattern completion. Understanding this distinction is crucial for developing models with genuine reasoning capabilities. The model uses a GPT-style transformer architecture and is trained solely on token-level sequence mapping from factorized to expanded polynomial expressions, without any encoding of mathematical operators or variable semantics.
Background
Symbolic math tasks like polynomial expansion require manipulating expressions according to algebraic rules. Sequence-to-sequence models are designed to transform input sequences into output sequences of potentially different lengths using an encoder-decoder architecture. This experiment specifically tests whether a small model can learn such transformations without explicit rule knowledge, shedding light on the debate whether LLMs reason or rely on pattern recognition.