Add notes on new backtesting framework.

NOTES.md

@@ -74,7 +74,7 @@ When trained on 2016-2024 (two full cycles):
 4. Calibrated confidence intervals through era-aware scaling
 5. Attempted model refinements for transition periods
 
-### Recent Refinement Attempts (2024)
+### Part 1: Failed Paths
 
 #### Attempt 1: Era Subdivision
 - Split transition period into multiple sub-periods
@@ -98,6 +98,52 @@ When trained on 2016-2024 (two full cycles):
   - Early Period: Similar coverage but worse accuracy
 - Outcome: Returned to original implementation due to comparable performance with less complexity
 
+### Part 2: Backtest Refinements
+
+#### Backtest Analysis Insights
+
+After running comprehensive backtests across multiple periods from 2013-2024, we identified:
+- Best performance during "normal" market conditions with MAPE ~30-35%
+- Significant degradation during stress periods (COVID crash, 2021 peak) with MAPE reaching 57%
+- Inconsistent confidence interval coverage, particularly for 68% CI (ranging from 20% to 91%)
+- Largest errors ($45-52K) concentrated around major market events
+
+#### Methodology Changes
+
+1. Refined Backtest Framework
+- Implemented more granular 6-month step testing periods
+- Standardized minimum training period (1 years) and validation window (8 years)
+- Separated results into "normal" and "stress" periods for clearer performance assessment
+
+2. Performance Evaluation Approach
+- Recognized that optimizing for stress periods could compromise normal period performance
+- Decided to focus optimization efforts on normal market conditions while accepting wider margins of error during stress periods
+- Enhanced results reporting to separate normal and stress period metrics
+
+#### Key Learnings
+
+1. Model focus should align with intended use:
+   - As a long-term cycle-aware model, primary optimization should target normal market conditions
+   - Stress period performance is informative but secondary
+   - Wider confidence intervals during transitions may be more appropriate than trying to predict extreme moves
+
+2. Backtest methodology matters:
+   - Previous ad-hoc testing may have over-emphasized stress period performance
+   - Systematic testing with consistent windows provides clearer picture of baseline performance
+   - Special period testing remains valuable for understanding limitations
+
+#### Future Considerations
+
+1. Conservative improvements to explore:
+   - Fine-tune uncertainty scaling for more consistent CI coverage during normal periods
+   - Consider dynamic uncertainty scaling based on market conditions
+   - Potential for regime-aware parameter adjustments
+
+2. Areas needing further investigation:
+   - Optimal balance between normal and stress period performance
+   - Impact of training window length on model stability
+   - Methods for early detection of transition into stress periods
+
 ## Current Implementation
 The model uses four main functions:
 1. `analyze_trends()`: Calculates cycle-position-specific log returns