Macro Forecast Evaluation & Regime Diagnostics

Research Toolkit · Macro Forecast Evaluation

Compare forecast models on consistent samples and understand their errors.

Status: Implemented and tested evaluation · Historical research example · Pseudo-out-of-sample design · Look-ahead documented · Not a live forecast · Toolkit overview · Data contract

Does a model really forecast better than the baseline on the same dates, and is the gap larger than chance?

A reusable evaluation workflow puts every model on the same forecast origins, target and horizon, compares errors on matched windows with a test of equal predictive accuracy, and splits errors by the signal state, showing how far a conclusion depends on the evaluation window and on the output version. Use it before a new predictor or model is allowed to change a forecast that feeds an allocation or risk view.

Inputs Per-origin forecast records (origin, horizon, model, outcome, forecast) for one shared target, and a baseline fixed in advance. Tested on synthetic cases and the two UK runs; a loader for new record files is a planned extension.

Analysis Errors of every model on the same origins, the cumulative loss differential (when a gap built up), and a Diebold–Mariano test of equal predictive accuracy, across windows and versions.

Outputs Error tables with forecast counts and dates, charts with their numbers, the test of each model against the baseline, and the same comparison across three output versions.

Historical example & limitations

Archive
UK services inflation, one application of the workflow (archived research example, February 2026 research snapshot): models M0–M6 (M6 in the latest run only) against a fixed-specification baseline, M2, at 3- and 6-month horizons.
Data
ONS monthly data through Dec 2025 (release of 18 February 2026, downloaded on 22 and 24 February 2026), latest vintage only; evaluation computed at build time from the archived records.
Sample
Common windows of 66 to 117 forecast origins, with overlapping (serially correlated) multi-step errors; one signal episode (September 2013 to February 2018).
Versions
Results change with the output version and the window: the sign of M4’s comparison with M2 flips between versions, and no difference on this page is significant at the 5% level under the main test.
Timing
Expanding-window pseudo-out-of-sample design, but neither a real-time vintage test nor a strict pseudo-out-of-sample test: the smoothed persistence state, the full-sample z-scores and, in the archived runs, the last h − 1 training outcomes were not available at the forecast origin; data are the latest vintage.
Testing
80 p-values with no multiple-testing adjustment, so a few could fall below 0.05 by chance alone; a model ranking from one window or one version is not robust here.
Scope
It evaluates past forecasts; it is not a forecast and not an operational forecasting service, and no result here shows that one model forecasts better than another in general. Full evidence and limitations

What it helps answer

Questions a researcher or portfolio manager can put to any forecasting comparison with this workflow:

  • Did the new predictor reduce errors on the same origins, or only on a different sample?
  • Is the difference larger than sampling error, once the overlap of multi-step forecasts is allowed for?
  • Does the conclusion survive a change of sample start, output version or training rule?
  • Were the losses accumulated steadily or in one episode?
  • Which inputs would not have been known at the forecast origin?

Inputs

Input In the UK example Requirement for a new application
Target and horizon Change in the services CPI 12-month rate (ONS series D7NN) over 3 and 6 months, in percentage points One definition shared by every model
Forecast origins Origin month t, with data for month t treated as known (in practice CPI for t is published in month t + 1 and labour-market data later). M1–M4 and M6 from 2006 (latest run) or 2010 (research-summary run); M0 from 2001 or 2005; M5 from 2016 or 2020; up to the last month with an observed outcome (September 2025 at 3 months, June 2025 at 6 months) The same origins for every model in a comparison
Predictors Services inflation, private-sector regular pay growth, a labour-market slack index, and three representations of persistence A timestamp saying when each value was first available
Model forecasts Per-origin records: origin, horizon, model, outcome, forecast One record per model, origin and horizon
Baseline M2, a Phillips-style model with a fixed specification (coefficients re-estimated at every origin) Fixed before the comparison is run
Data vintage The latest ONS vintage at download (February 2026); no historical vintages Vintage or release date per value, where available

Analyses

  1. Ingest Read the per-origin forecast records of each archived run and the persistence series. Implemented & tested.
  2. Validate Check that every model has the expected forecasts, that outcomes agree across models at each origin, that errors equal outcome minus forecast, and that months are consecutive. Implemented & tested (unit tests feed synthetic invalid records to each check).
  3. Configure Choose the output version, horizon, model and comparison window with the controls below. Implemented (the controls select among precomputed results; every combination is tested in the JSON).
  4. Compute RMSE, MAE, direction hit rate and n; RMSE relative to M2; the loss differential, its running sum, and a Diebold–Mariano test with the original equal-weight long-run variance and the Harvey–Leybourne–Newbold correction, with a Newey–West (Bartlett) variance as a sensitivity. Implemented & tested (unit tests on synthetic cases and a pinned reference value; parity with the archived model-comparison tables to 10⁻⁹).
  5. Compare Own windows against the common window, three output versions side by side, and errors split by the signal state. Implemented & tested.
  6. Interpret and export Charts with numeric tables, a static table of the default view, and the full computed data as JSON. Implemented & tested.

Interactive demonstration

The controls switch among results computed at build time from the archived records: every combination is a distinct calculation, not a relabelling. The default view is the latest archived run, 3 months, M4 against M2, common window. The third output version re-estimates the latest run with training restricted to outcomes observed by the origin; it removes only the overlapping training targets, and the smoothed state, full-sample z-scores and latest-vintage data still carry look-ahead.

Step 1: Measure persistence

Figure 1: How persistent is UK services inflation? Two estimates that answer different questions, with services and goods CPI below.

How to read it. Persistence is the autoregressive coefficient of 3-month annualised services inflation. The rolling estimate (dashed) uses only the previous 120 months at each date, so it uses no later months (apart from publication lags and the latest data vintage). The smoothed state-space estimate (solid, with its 68% band) uses the whole sample, including later months: it is a retrospective description, not information available at a forecast origin. The grey span marks months when the smoothed estimate stayed below the threshold for three consecutive months (the durable low-persistence signal); there is one such episode, September 2013 to February 2018. Estimates come from the latest archived run (sample from 2001); the controls do not change this figure. Source: ONS MM23 release of 18 February 2026 (Open Government Licence), series D7NN and D7NM; persistence estimates from the author’s archived research outputs, 24 February 2026.

Step 2: Design a fair comparison

Each model is re-estimated at every origin by least squares on an expanding window (at least 60 months), one regression per horizon (direct multi-step forecasts). The persistence models need extra history: in the latest run M5’s rolling estimate starts in December 2010 (in the research-summary run, later), so M5 has the fewest origins. Scoring each model on its own window therefore compares different periods; the common window removes that difference but is set by the shortest model.

Figure 2: Which origins does each model cover? Own windows (bars) and the common window (shaded), for the chosen version and horizon.

How to read it. Each bar runs from a model’s first to its last forecast origin with an observed outcome; n is the number of origins. The highlighted bar is the model chosen above; M2, the baseline, is dark grey. The shaded band is the common window used when “Common window” is selected. Source: computed at build time from the per-origin records of the author’s archived research outputs (February 2026 release of the study).

Step 3: Inspect errors

Table 1: Forecast errors by model. RMSE, MAE, direction hit rate and n for the chosen version, horizon and window.

How to read it. Errors are in percentage points of the 12-month rate. RMSE vs M2 = 100 × (model RMSE ÷ M2 RMSE − 1): negative means smaller errors than M2, positive means larger. Direction hit rate is the share of origins where the forecast change has the same sign as the outcome (the no-change model M0 forecasts zero and scores low by construction). On own windows, the main ratio uses the origins each model shares with M2; the archived-table column divides by M2’s RMSE on M2’s own origins, so it mixes periods (for M5 it even changes sign). Source: computed at build time from the author’s archived research outputs (February 2026 release of the study); matches the archived model-comparison tables to 10⁻⁹.

Step 4: Test robustness

Figure 3: Is the difference larger than sampling error, and when did it build up? Cumulative loss differential, the Diebold–Mariano test, and the same comparison across versions and windows.

How to read it. The loss differential at each origin is d = (model error)² − (M2 error)². The line is its running sum: it rises when the model’s error is larger than M2’s. The Diebold–Mariano statistic divides the mean of d by a long-run standard error with h − 1 lags, because forecasts h months ahead overlap (their errors are serially correlated up to lag h − 1); as in the original test, the first h − 1 autocovariances get equal weights (a rectangular kernel). The Harvey–Leybourne–Newbold correction adjusts it for short samples, and the p-value is two-sided from a t distribution. The p-value depends on the variance estimator: with Newey–West (Bartlett) weights, also shown, the smallest p-value on the page (M6, 6 months, observed-outcomes version, common window) falls from 0.07 to 0.02. For nested pairs (M0 and M1 inside M2, M2 inside M6) the DM statistic has a non-standard limiting distribution under recursive estimation, so the t reference distribution does not apply; a Clark–West adjustment would be the appropriate test, so those p-values are indicative only. Under the null of equal population accuracy, the larger model of a nested pair is expected to show slightly larger errors in a finite sample, because it estimates extra coefficients. Failing to reject equal accuracy is not evidence that two models are equally accurate: with a sample of this size and overlapping errors, the test has low power. The signal-state split uses the signal that enters M4 (previous month); the signal is derived from the smoothed estimate, a retrospective classification not known at the forecast origin, and there is only one signal episode, so the split is close to a split by period. Source: computed at build time from the author’s archived research outputs (February 2026 release of the study).

What this shows. The answer depends on the version and on the evaluation window. On the common window, M4’s RMSE is 0.68% (3 months) and 3.09% (6 months) lower than M2’s in the research-summary run (sample from 2005, origins from January 2020), but 2.99% and 3.40% higher in the latest run (sample from 2001, origins from January 2016). Much of that difference comes from the evaluation window: scored on the research-summary run’s window (from January 2020), the latest run gives +0.51% at 3 months and −4.04% at 6 months. The remaining change (sample start 2001 rather than 2005, added earnings and vacancies inputs) cannot be separated in the archive. The re-estimation that trains only on observed outcomes keeps the sign of the latest run (+2.82% and +1.64%, against +2.99% and +3.40%; its common windows start h − 1 months later, in March and June 2016); it does not remove the other look-ahead listed under Limitations. In the latest run, the cumulative loss differential builds up from 2018 to 2020, after the only signal episode ended in February 2018, and partly reverses in 2021–2022; a few years, not a steady pattern, drive the gap. No p-value from the main test (equal-weight variance with the HLN correction) is below 0.05. The smallest, 0.07, is for M6, which is worse than M2 in the observed-outcomes version at 6 months (+6.81%); with the Bartlett variance it is 0.02, and among 80 tests with no multiple-testing adjustment one such result is to be expected. A model ranking from one window or one version is not robust here.

Outputs

Without JavaScript, the default view is available as a static table, generated at build time from the same data:

Static table (no JavaScript needed): the default view. Latest archived run (24 Feb 2026; sample from 2001); 3-month horizon; common window (forecast origins 2016-01 to 2025-09, n = 117). Relative RMSE = 100 × (model RMSE ÷ M2 RMSE − 1); negative means smaller errors than M2. DM = HLN-corrected statistic with its two-sided p-value (long-run variance with equal weights on the first h − 1 autocovariances); M0, M1 and M6 are nested with M2, so their p-values are indicative only.
Model RMSE vs M2 n RMSE MAE Direction hit rate DM (HLN) p-value
M0 no change +6.95% 117 0.535 0.396 0.13 1.25 0.21
M1 AR(1) +8.93% 117 0.544 0.396 0.51 1.53 0.13
M2 baseline — 117 0.500 0.370 0.56 — —
M3 slack → smoothed persistence +2.78% 117 0.514 0.372 0.62 0.62 0.53
M4 slack → durable signal +2.99% 117 0.515 0.397 0.62 0.70 0.48
M5 slack → rolling persistence −4.05% 117 0.480 0.347 0.62 −0.67 0.50
M6 M4v2 +2.82% 117 0.514 0.395 0.59 1.28 0.20
Version sensitivity: M4’s RMSE relative to M2 (%), by output version, horizon and comparison window; in brackets the matched origins n and the DM p-value. Own-window columns: RMSE ratio, n and p-value all on the origins where M4 and M2 overlap. The last row scores the latest run’s own archived forecasts on the research-summary run’s common window, which isolates the effect of the evaluation window.
Version 3m, own window 3m, common window 6m, own window 6m, common window
Latest run, 24 Feb 2026 (sample from 2001) −1.86%
(232; p = 0.55)
+2.99%
(117; p = 0.48)
−2.01%
(229; p = 0.75)
+3.40%
(114; p = 0.71)
Research-summary run, 23 Feb 2026 (sample from 2005) −3.15%
(184; p = 0.36)
−0.68%
(69; p = 0.86)
−2.64%
(181; p = 0.69)
−3.09%
(66; p = 0.70)
Latest inputs, trained only on observed outcomes −2.67%
(230; p = 0.45)
+2.82%
(115; p = 0.56)
−4.91%
(224; p = 0.51)
+1.64%
(109; p = 0.89)
Latest run on the research-summary run’s common window (from 2020-01) — +0.51%
(69; p = 0.91)
— −4.04%
(66; p = 0.61)
  • Definitions. Target y(t, h) = services CPI 12-month rate at t + h minus at t (percentage points). RMSE and MAE in percentage points. Relative RMSE = 100 × (model ÷ M2 − 1). DM: d = (model error)² − (M2 error)²; long-run variance with equal weights on autocovariances up to lag h − 1 (Bartlett weights if that estimate is not positive; a Bartlett p-value is also reported); HLN correction; t distribution with n − 1 degrees of freedom; two-sided.
  • Data dates. ONS data through December 2025 (CPI through January 2026 in Figure 1); MM23 release of 18 February 2026; downloads of 22 and 24 February 2026. Output runs of 23 and 24 February 2026. These dates belong to the archived study and are separate from the date this page was built.
  • Machine-readable output. forecast_eval.json: every number on this page, with SHA-256 hashes of the archived source files, the parity checks and the code version.

How to use and adapt

Other variables, predictors, models or countries need matched, properly licensed data, re-estimation at every origin and a fresh pseudo-out-of-sample evaluation on matched windows. The archived exercise is not an always-current inflation forecast and is not connected to live data feeds.

Step 5: Adapt to other variables, predictors or countries

The evaluation functions (errors, relative RMSE, Diebold–Mariano, cumulative loss) take a table of per-origin records (origin, horizon, model, outcome, forecast) and are tested on synthetic data and on the two UK runs; a loader for new record files is a Planned extension. Producing those records for a new question takes upstream work that this page does not do:

  1. Define the target and its timing. Choose the series and horizons, and record when each value is published; use historical vintages where they exist.
  2. Build predictors that were available at the origin. Lag each predictor by its publication delay, and use filtered or rolling estimates rather than smoothed ones.
  3. Re-estimate every model at every origin, training only on outcomes already observed at that origin, and write the per-origin records.
  4. Evaluate on matched origins with the same baseline, horizon and target, then run the version and window checks above.

What is tested: the evaluation computations (unit tests, parity with the archived tables), the input checks (synthetic invalid records) and the expanding-window forecasting function, which reproduces the latest UK run’s stored forecasts to 10⁻¹⁰. What is not: data download, the state-space estimation, and any other variable, predictor or country (Planned extension). The UK results are not results for new inputs; a new predictor needs its own data-timing checks, re-estimation and a matched pseudo-out-of-sample window.

Historical example, evidence and limitations

UK services inflation, archived research example (February 2026). A short research study of whether changes in the persistence of UK services inflation help to forecast it, written in February 2026 as a research snapshot.

Model Predictors (values for the origin month unless marked)
M0 No change Forecast change set to zero
M1 AR(1)-style Services inflation
M2 Fixed-specification Phillips-style baseline Services inflation, wage growth, labour-market slack
M3 Slack replaced by smoothed persistence Services inflation, wage growth, smoothed persistence (previous month)
M4 Slack replaced by durable signal Services inflation, wage growth, durable low-persistence signal from the smoothed estimate (previous month)
M5 Slack replaced by rolling persistence Services inflation, wage growth, rolling 120-month persistence (previous month)
M6 Regime-gated Phillips (M4v2; latest run only) M2’s predictors, the signal (previous month), and the signal times inflation and wage growth

M3–M5 drop the slack index, so their comparison with M2 swaps one predictor for another; M6 is the only model that adds the signal to M2’s predictors.

  • Sources. Office for National Statistics: consumer price inflation (MM23; services D7NN, goods D7NM), labour market statistics, average weekly earnings (private-sector regular pay), vacancies; Open Government Licence v3.0. Bank of England: nominal gilt yield curves, used in the original study’s rates cross-check only; they do not enter the forecasts evaluated here.
  • Output versions. The run of 23 February 2026 (sample from 2005, models M0–M5) is the version behind the author’s February 2026 research summary.1 The run of 24 February 2026 (sample from 2001, models M0–M6) is the latest. The common windows differ (from January 2020 and from January 2016), because M5 starts later in the shorter sample. Scoring the latest run on the earlier window isolates that part: on the research-summary run’s window the latest run gives +0.51% (3 months, n = 69, p = 0.91) and −4.04% (6 months, n = 66, p = 0.61) for M4 against M2. The later run also moved the sample start and added the earnings and vacancies files to its inputs; the archive cannot separate those two changes. The third version is computed here and is not archived: the latest run’s inputs, with training restricted to outcomes observed by the origin.
  • Reproduction scope. Evaluation computations: recomputed from the per-origin records of both runs and tested (Implemented & tested). Forecasts of the latest run: re-estimated from its archived processed inputs, matching the stored forecasts to 10⁻¹⁰. Data download, the state-space smoother and the rolling regressions were not rerun (Archived research example). The research-summary run’s processed inputs were overwritten by the later run, so its forecasts are evaluated as archived and cannot be split by signal state. The parity results and SHA-256 hashes of the archived source files are recorded in the meta block of forecast_eval.json.
  • Limitations. The design is pseudo-out-of-sample (recursive re-estimation, forecasts scored only after the origin), but not strictly so: a strict test uses only data available at the origin, and these forecasts also use later information (items 1 to 4; only the use of the latest data vintage in item 3 is the usual pseudo-out-of-sample simplification). (1) M3, M4 and M6 use the smoothed persistence estimate, which draws on later data; lagging it one month does not make it available at the origin. The signal threshold is also set from the 2001–2019 distribution of the smoothed estimate, so for earlier origins it uses later information. Only M5’s rolling estimate is backward-looking. (2) The archived training rule includes the last h − 1 outcomes, which are observed only after the origin; the observed-outcomes version removes only these, and the smoothed state, full-sample z-scores and latest-vintage data still carry look-ahead. (3) Values are dated by reference month, not release date, from the latest vintage: CPI for a month is published in the following month, labour-market data later and subject to revision. (4) The slack index combines z-scores computed over the full sample. (5) Common windows are short (66 to 117 origins), errors overlap, and no adjustment is made for testing several models, horizons and versions: the controls give access to 80 p-values, and at the 5% level a few would fall below 0.05 by chance alone. (6) One signal episode only. (7) M0 and M1 are nested in M2, and M2 in M6, so the DM p-values of M0, M1 and M6 against M2 are indicative only.

Footnotes

  1. Dated note on the research-summary figures. The author’s February 2026 research summary reported that, on the common window, M4’s RMSE was 0.68% lower than M2’s at 3 months and 3.09% lower at 6 months. Both figures are reproduced here from the 23 February 2026 run’s per-origin records (69 and 66 origins, from January 2020). The Diebold–Mariano test with the HLN correction gives p = 0.86 and 0.70: neither difference is statistically significant. In the latest run (24 February 2026), the same comparison gives an RMSE 2.99% and 3.40% higher than M2’s (p = 0.48 and 0.71).↩︎