Validation
Every statistic in OrcStat comes from a tested engine, never from the AI. Here is how the engine is checked, and the reference results it has to match on every change.
How it's checked
The NIST Statistical Reference Datasets publish certified results to 15 significant digits. OrcStat's engine must reproduce them within the tolerances below.
Every cell of a set of reference output tables, from descriptives to regression, is compared with fixed expected values. A change that moves a number fails the build.
The golden tables are also compared across macOS, Windows and Linux, with and without the native maths kernels, so a library difference can't change a result.
The engine alone has over 8,000 tests, run on every push and pull request alongside the frontend tests.
NIST StRD
The NIST Statistical Reference Datasets are the standard check for statistical software accuracy. Each case below runs the dataset through OrcStat's own syntax and compares the output with NIST's certified values. The tolerance is relative: 1e-10 means the result agrees to about ten significant digits.
| Dataset | NIST difficulty | OrcStat runs | Values checked | Tolerance |
|---|---|---|---|---|
| Longley | Higher | REGRESSION, 6 predictors (16 cases) | Coefficients, standard errors, R², residual SD, ANOVA | 1e-10 |
| Norris | Lower | REGRESSION, 1 predictor (36 cases) | Coefficients, standard errors, R², residual SD, ANOVA | 1e-10 |
| Pontius | Lower | REGRESSION, quadratic (40 cases) | Coefficients, standard errors, R², residual SD, ANOVA | 1e-10 |
| NumAcc4 | Higher | DESCRIPTIVES (1,001 values) | Mean, standard deviation | 1e-15 mean, 1e-8 SD |
| PiDigits | Lower | DESCRIPTIVES (5,000 values) | Mean, standard deviation | 1e-15 mean, 1e-13 SD |
| Lottery | Lower | DESCRIPTIVES (218 values) | Mean, standard deviation | 1e-15 mean, 1e-13 SD |
| Lew | Lower | DESCRIPTIVES (200 values) | Mean, standard deviation | 1e-15 mean, 1e-13 SD |
These run in continuous integration on every push and pull request.
The AI and the numbers
OrcStat's AI chooses and explains analyses, but it doesn't produce statistics. The engine computes every value in every table, and before you read anything the AI wrote, each number in its text is checked against the engine's output.