September 11, 2000–An independent assessment of Mathematica entitled “The Accuracy of Mathematica 4 as a Statistical Package” was published in the September 2000 issue of Computational Statistics. In it, the precision of Mathematica‘s calculations was found to be “markedly superior” to that of other major statistical packages such as SAS, S-Plus, SPSS, and Excel. The paper was written by Dr. B. D. McCullough, a senior economist at the Federal Communications Commission, who has published several articles on the reliability of statistical and econometric software.

Applying the same methodology to Mathematica as that used on other packages, McCullough assessed reliability in three areas: linear and nonlinear estimation, random number generation, and statistical distributions. McCullough found that “Mathematica achieves unparalleled accuracy and reliability on the National Institute of Standards and Technology (NIST) Standard Reference Datasets (StRD) and on the ELV benchmark for statistical distributions.”

The typical statistical package relies on fixed, machine-precision calculation that involves approximations and roundoffs, thereby introducing error. However, “by virtue of its variable-precision arithmetic and symbolic power, Mathematica‘s performance on these reliability tests far exceeds any finite-precision statistical package,” notes McCullough. But since much data is recorded to only three or four significant figures, what makes Mathematica‘s accuracy so important?

McCullough states that the purpose of “benchmarking,” as the assessment tests are called, is “not to count digits, but to assess the quality of the implemented algorithm” used to perform a calculation. The questions of interest to users are “Where does it break down?” and “Will the program warn me?” For example, Microsoft Excel, arguably the package most commonly used for statistical calculation, has been found to perform inadequately in almost all areas of the StRD.

StRD Results for Nonlinear Problem Ratkowsky 43

Coefficient	NIST	Excel
b1	699.64151270	676.0986499
b2	5.2771253025	39.7190456
b3	0.75962938329	4.559009025
b4	1.2792483859	13.02379155

McCullough and Wilson, Computational Statistics and Data Analysis 31 (1999)

Even at default settings, Mathematica does not break down on any of the tests. Mathematica not only will inform the user if it cannot complete the procedure as defined but also will tell the user what settings need to be altered in order to complete the computation. By increasing the required precision of the calculation, Mathematica is able to produce a perfect score on all tests in all four areas of the StRD. Although other programs are able to produce “correct” solutions (accurate to three to four significant digits) on the tests, no other program can come close to matching Mathematica‘s overall performance.

Overall: 58 Problems (taking best options)

Comparison of McCullough Results to Previous Package Assessments (2000)

McCullough highlights Wolfram Research’s excellent technical support and gigaNumerics, a project that uses more efficient data structures and algorithms for speed gains in processing very large numbers, as additional reasons for his confidence in Mathematica. Given the trend toward increasingly large computer memory and processing speed–and increasingly large datasets–McCullough says that the “potential [of cumulative rounding error] for completely corrupting results obtained using traditional methods and algorithms, is most worrisome. Hope may well lie in Mathematica‘s variable-precision arithmetic and Wolfram [Research]’s gigaNumerics project.”

It is the combination of all of these readily accessible features that makes Mathematica so accurate and straightforward to use, even for the newcomer. McCullough himself says he “shall be using Mathematica regularly for statistical purposes” as a supplement to his regular package. Wolfram Research’s commitment to “forward-looking” development, and the pending release of new statistics add-on packages for Mathematica, will only serve to make it a more natural application.

For more information on Mathematica in statistics, visit the Mathematica Statistics Solutions on our web site.