Forward test: record forecasts before their outcomes exist.

`snapshot` writes each tracked model's forecast quantiles at the seven
backtest horizons from the latest price to data/forecasts/<origin>.csv. It
refuses stale data (older than two days) and duplicate dates, so snapshots
can't be reconstructed after the fact; committing them dates them.
`forward` scores every recorded forecast whose target date has passed,
reusing the backtest's scoring (now factored out as evaluate.score).

Tracked: random_walk, drift_rw, cycle, powerlaw, plus powerlaw_ou,
powerlaw_ou_param and cycle_on_powerlaw, which development data couldn't
settle. `just weekly` runs update, snapshot and forward.

First snapshot: 2026-09-23 (BTC $84.4K). The first outcomes are due
2026-10-23.
This commit is contained in:
sam
2026-09-24 03:08:30 -07:00
parent 082bcfbbbc
commit 67d016fe25
7 changed files with 316 additions and 19 deletions
+19
View File
@@ -123,6 +123,23 @@ at 2-3 years (mean PIT 0.53-0.56) but with intervals too wide (its 80% interval
held every 2- and 3-year outcome). The holdout is now spent for these models;
a new model needs new data to be tested honestly.
### Forward test (from 2026-09-23)
New data is the only honest test left, so forecasts are now recorded before
their outcomes exist. `just snapshot` writes every tracked model's forecast
from the latest price to `data/forecasts/<date>.csv`. It refuses if the price
data is more than two days old or the date already has a snapshot, and the
files are committed so version control dates them. `just forward` scores
every recorded forecast whose target date has passed, the same way the
backtest does.
Tracked: the four registered models plus `powerlaw_ou`, `powerlaw_ou_param`
and `cycle_on_powerlaw`, which development data couldn't settle. Snapshots
store the forecasts themselves, so if a model's definition changes it gets a
new name. The routine is `just weekly` (update, snapshot, score), then commit
the new snapshot. The first 1-month outcomes arrive 2026-10-23; 1-year results
need until late 2027, and 4-year ones until 2030.
## Usage
With Nix:
@@ -133,6 +150,7 @@ just update # fetch new daily prices from Coinbase
just backtest # score models on development data -> output/backtest/
just forecast # forecast from the latest price -> output/forecast/
just ab # run A/B experiments -> output/ab/
just weekly # new prices, a forward-test snapshot, and its scores so far
just test
just holdout # score on held-out outcomes; sparingly
```
@@ -149,6 +167,7 @@ btcmodel/
forecast.py Forecast (quantiles of log price), CRPS, PIT
evaluate.py walk-forward backtest and summary
experiments.py A/B tests: hypothesis, control, variants, verdict rule
forward.py forward test: record forecasts, score them as outcomes arrive
models/ drift, volatility and shape components; register models in __init__.py
plots.py fan chart, skill and calibration charts
```