11. What It Cannot Catch

The Honest Boundary of the Instrument

Benford’s law serves as a statistical signal rather than an absolute truth that can serve as the sole foundation for an accusation of fraud. William Goodman noted in 2016 that many published claims of non-conformance in real records are merely reports of deviations within the range of ordinary variation. These instances function as false alarms rather than evidence of misconduct because the underlying data still falls within expected statistical bounds for natural distributions. When researchers rely on the law to detect anomalies, they often mistake natural statistical noise for the deliberate manipulation of numerical records by a human agent. The instrument is sensitive to the structure of datasets, yet it lacks the capacity to distinguish between a creative accountant and a simple statistical outlier. Understanding this limit is essential for any practitioner who wishes to use the distribution of leading digits to flag potential discrepancies in financial or scientific records.

The Problem of Small Samples and Estimation

Andrea Cerasa and Miguel A. Fernández conducted a rigorous Monte Carlo simulation study in 2022 to examine how these tests function at very small sample sizes. Their research demonstrated that existing conformity tests behave unreliably when the number of observations is limited, often leading to misleading conclusions about the data integrity. In response to these findings, Cerasa and Fernández proposed a combined three-test procedure to restore statistical power and improve the reliability of the analysis process. Legitimate rounding and estimation artefacts can easily skew the distribution, creating patterns that appear suspicious even when the numbers were generated through entirely honest, standard processes. Fraud that is carefully constructed to preserve the natural digit distribution remains invisible to the test, allowing a sophisticated actor to bypass the check entirely. A test that is strong enough to convince an observer is often strong enough to convict the innocent, especially when the sample size is dangerously small.

The Status of the Findings

There is no serious dispute found in the academic literature regarding the cautionary nature of these recent findings by Goodman, Cerasa, and Fernández. Both papers act as corrective measures within the Benford literature, emphasizing that these tests are meant to clarify the evidence rather than reject the law itself. The consensus remains that the law is a valid regularity, provided the analyst recognizes the boundary where the mathematical model ceases to be a reliable guide.

Example: A Benford test is like a metal detector on a beach: it alerts you to something beneath the sand, but it cannot tell a gold coin from a discarded pull-tab.

Verdict. Benford conformity is a red flag, not proof of fraud, and should never be treated as a self-executing verdict of criminal intent.

Overturned if: A future meta-analysis proves that standard conformity tests maintain high sensitivity and low false-positive rates when applied to datasets containing fewer than fifty items.

Sources. 2016 · William Goodman, "The Promises and Pitfalls of Benford's Law" — Significance, Wiley · 2022 · Andrea Cerasa, Miguel A. Fernández, "Testing for Benford's Law in Very Small Samples" — PLOS One