Forward test: record forecasts before their outcomes exist.
`snapshot` writes each tracked model's forecast quantiles at the seven backtest horizons from the latest price to data/forecasts/<origin>.csv. It refuses stale data (older than two days) and duplicate dates, so snapshots can't be reconstructed after the fact; committing them dates them. `forward` scores every recorded forecast whose target date has passed, reusing the backtest's scoring (now factored out as evaluate.score). Tracked: random_walk, drift_rw, cycle, powerlaw, plus powerlaw_ou, powerlaw_ou_param and cycle_on_powerlaw, which development data couldn't settle. `just weekly` runs update, snapshot and forward. First snapshot: 2026-09-23 (BTC $84.4K). The first outcomes are due 2026-10-23.
This commit is contained in:
@@ -123,6 +123,23 @@ at 2-3 years (mean PIT 0.53-0.56) but with intervals too wide (its 80% interval
|
||||
held every 2- and 3-year outcome). The holdout is now spent for these models;
|
||||
a new model needs new data to be tested honestly.
|
||||
|
||||
### Forward test (from 2026-09-23)
|
||||
|
||||
New data is the only honest test left, so forecasts are now recorded before
|
||||
their outcomes exist. `just snapshot` writes every tracked model's forecast
|
||||
from the latest price to `data/forecasts/<date>.csv`. It refuses if the price
|
||||
data is more than two days old or the date already has a snapshot, and the
|
||||
files are committed so version control dates them. `just forward` scores
|
||||
every recorded forecast whose target date has passed, the same way the
|
||||
backtest does.
|
||||
|
||||
Tracked: the four registered models plus `powerlaw_ou`, `powerlaw_ou_param`
|
||||
and `cycle_on_powerlaw`, which development data couldn't settle. Snapshots
|
||||
store the forecasts themselves, so if a model's definition changes it gets a
|
||||
new name. The routine is `just weekly` (update, snapshot, score), then commit
|
||||
the new snapshot. The first 1-month outcomes arrive 2026-10-23; 1-year results
|
||||
need until late 2027, and 4-year ones until 2030.
|
||||
|
||||
## Usage
|
||||
|
||||
With Nix:
|
||||
@@ -133,6 +150,7 @@ just update # fetch new daily prices from Coinbase
|
||||
just backtest # score models on development data -> output/backtest/
|
||||
just forecast # forecast from the latest price -> output/forecast/
|
||||
just ab # run A/B experiments -> output/ab/
|
||||
just weekly # new prices, a forward-test snapshot, and its scores so far
|
||||
just test
|
||||
just holdout # score on held-out outcomes; sparingly
|
||||
```
|
||||
@@ -149,6 +167,7 @@ btcmodel/
|
||||
forecast.py Forecast (quantiles of log price), CRPS, PIT
|
||||
evaluate.py walk-forward backtest and summary
|
||||
experiments.py A/B tests: hypothesis, control, variants, verdict rule
|
||||
forward.py forward test: record forecasts, score them as outcomes arrive
|
||||
models/ drift, volatility and shape components; register models in __init__.py
|
||||
plots.py fan chart, skill and calibration charts
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user