diff --git a/NOTES.md b/NOTES.md index 7cbfd6e..93842aa 100644 --- a/NOTES.md +++ b/NOTES.md @@ -74,7 +74,7 @@ When trained on 2016-2024 (two full cycles): 4. Calibrated confidence intervals through era-aware scaling 5. Attempted model refinements for transition periods -### Recent Refinement Attempts (2024) +### Part 1: Failed Paths #### Attempt 1: Era Subdivision - Split transition period into multiple sub-periods @@ -98,6 +98,52 @@ When trained on 2016-2024 (two full cycles): - Early Period: Similar coverage but worse accuracy - Outcome: Returned to original implementation due to comparable performance with less complexity +### Part 2: Backtest Refinements + +#### Backtest Analysis Insights + +After running comprehensive backtests across multiple periods from 2013-2024, we identified: +- Best performance during "normal" market conditions with MAPE ~30-35% +- Significant degradation during stress periods (COVID crash, 2021 peak) with MAPE reaching 57% +- Inconsistent confidence interval coverage, particularly for 68% CI (ranging from 20% to 91%) +- Largest errors ($45-52K) concentrated around major market events + +#### Methodology Changes + +1. Refined Backtest Framework +- Implemented more granular 6-month step testing periods +- Standardized minimum training period (1 years) and validation window (8 years) +- Separated results into "normal" and "stress" periods for clearer performance assessment + +2. Performance Evaluation Approach +- Recognized that optimizing for stress periods could compromise normal period performance +- Decided to focus optimization efforts on normal market conditions while accepting wider margins of error during stress periods +- Enhanced results reporting to separate normal and stress period metrics + +#### Key Learnings + +1. Model focus should align with intended use: + - As a long-term cycle-aware model, primary optimization should target normal market conditions + - Stress period performance is informative but secondary + - Wider confidence intervals during transitions may be more appropriate than trying to predict extreme moves + +2. Backtest methodology matters: + - Previous ad-hoc testing may have over-emphasized stress period performance + - Systematic testing with consistent windows provides clearer picture of baseline performance + - Special period testing remains valuable for understanding limitations + +#### Future Considerations + +1. Conservative improvements to explore: + - Fine-tune uncertainty scaling for more consistent CI coverage during normal periods + - Consider dynamic uncertainty scaling based on market conditions + - Potential for regime-aware parameter adjustments + +2. Areas needing further investigation: + - Optimal balance between normal and stress period performance + - Impact of training window length on model stability + - Methods for early detection of transition into stress periods + ## Current Implementation The model uses four main functions: 1. `analyze_trends()`: Calculates cycle-position-specific log returns