bitcoin-model/NOTES.md

Bitcoin Price Model Description

Overview

This Bitcoin price prediction model uses a combination of log returns, cycle awareness, and Monte Carlo simulation to generate price projections with confidence intervals. The model was developed through several iterations, with each refinement aimed at improving accuracy and reliability.

Core Components

1. Trend Analysis

• Uses log returns of Bitcoin prices to better handle exponential growth patterns
• Groups returns by position within the halving cycle (0-1460 days)
• Applies simple moving average smoothing to reduce noise while preserving underlying patterns
• Position within cycle is calculated linearly between known halving dates

2. Market Era Adjustments

Uses distinct eras with specific characteristics:

• Early (2013-2017): Higher volatility, conservative trends, moderate trend following
• Transition (2017-2020): Slightly elevated uncertainty during futures market introduction
• Mature (2020+): Reduced volatility, balanced trends, lighter trend following

Each era has specific scaling factors for volatility, trend expectations, and trend following behavior

3. Volatility Estimation

Uses a multi-timeframe approach:

• Short window (30 days)
• Medium window (90 days)
• Long window (180 days)
• Combines these using weighted exponential moving averages
• Applies era-specific scaling factors

4. Price Projection

• Uses Monte Carlo simulation with 1000 paths
• Incorporates era-aware adjustments to volatility and trends
• Generates both point estimates and confidence intervals
• Projects forward using cycle-aware return expectations

Key Features

Log Returns

Instead of working directly with prices or simple returns, the model uses log returns which:

• Better handle Bitcoin's exponential price growth
• Provide more stable statistical properties
• Allow for simpler cumulative return calculations

Cycle Awareness

• Recognizes Bitcoin's ~4 year (1460 day) halving cycle
• Maps historical returns to cycle positions
• Allows the model to capture recurring patterns around halving events

Enhanced Monte Carlo

• Base simulation using normal distribution
• Includes era-specific adjustments for volatility and trends
• Generates both median projections and confidence intervals
• Well-calibrated uncertainty estimates for post-2013 data

Performance Metrics

When trained on 2016-2024 (two full cycles):

• MAPE: 11.7%
• RMSE: ~$9,011
• Max Error: $18,786
• 95% CI Coverage: 99.5%
• 68% CI Coverage: 69.0%

Model Limitations

• Not suitable for pre-2013 market data
• Assumes future cycles will resemble post-2013 patterns
• Uses simple linear interpolation between halving dates
• May not capture extreme market events well
• Shows increased error in transition periods (2015-2018)

Development History

1. Started with direct cycle analysis of returns
2. Added log-based analysis for better handling of exponential growth
3. Refined volatility calculation using multiple timeframes
4. Calibrated confidence intervals through era-aware scaling
5. Attempted model refinements for transition periods

Part 1: Failed Paths

Attempt 1: Era Subdivision

• Split transition period into multiple sub-periods
• Added skew adjustments for different market phases
• Results: Made the model more complex without clear benefits
• Outcome: Abandoned in favor of simpler approach

Attempt 2: Dynamic Regime Detection

• Implemented real-time market regime detection
• Used volatility ratios and trend strength metrics
• Results: Made the model too reactive to short-term changes
• Outcome: Less stable than original era-based approach

Attempt 3: Weighted Rolling Regression

• Used weighted regression for trend estimation
• Combined cycle position and time-based weights
• Results: Similar performance to original model but more complex
• Key Findings:
  • Recent Period: Slightly worse (MAPE 12.4% vs 11.7%)
  • Mid Period: Marginally better coverage but similar overall
  • Early Period: Similar coverage but worse accuracy
• Outcome: Returned to original implementation due to comparable performance with less complexity

Part 2: Backtest Refinements

Backtest Analysis Insights

After running comprehensive backtests across multiple periods from 2013-2024, we identified:

• Best performance during "normal" market conditions with MAPE ~30-35%
• Significant degradation during stress periods (COVID crash, 2021 peak) with MAPE reaching 57%
• Inconsistent confidence interval coverage, particularly for 68% CI (ranging from 20% to 91%)
• Largest errors ($45-52K) concentrated around major market events

Methodology Changes

1. Refined Backtest Framework

• Implemented more granular 6-month step testing periods
• Standardized minimum training period (2 years) and validation window (8 years)
• Separated results into "normal" and "stress" periods for clearer performance assessment

2. Performance Evaluation Approach

• Recognized that optimizing for stress periods could compromise normal period performance
• Decided to focus optimization efforts on normal market conditions while accepting wider margins of error during stress periods
• Enhanced results reporting to separate normal and stress period metrics

Key Learnings

1. Model focus should align with intended use:

   • As a long-term cycle-aware model, primary optimization should target normal market conditions
   • Stress period performance is informative but secondary
   • Wider confidence intervals during transitions may be more appropriate than trying to predict extreme moves

2. Backtest methodology matters:

   • Previous ad-hoc testing may have over-emphasized stress period performance
   • Systematic testing with consistent windows provides clearer picture of baseline performance
   • Special period testing remains valuable for understanding limitations

Future Considerations

1. Conservative improvements to explore:

   • Fine-tune uncertainty scaling for more consistent CI coverage during normal periods
   • Consider dynamic uncertainty scaling based on market conditions
   • Potential for regime-aware parameter adjustments

2. Areas needing further investigation:

   • Optimal balance between normal and stress period performance
   • Impact of training window length on model stability
   • Methods for early detection of transition into stress periods

Uncertainty Estimation Improvements

Attempted several approaches to improve uncertainty estimation in the model:

1. Initial market-aware uncertainty scaling

   • Added dynamic scaling based on market conditions
   • Incorporated trend strength, drawdowns, and volatility regimes
   • Improved CI coverage consistency between stress/normal periods
   • Results showed more balanced coverage (88% for 95% CI, ~60-65% for 68% CI)

2. Volatility projection system

   • Attempted to model how volatility evolves through cycles
   • Added cycle-aware volatility forecasting
   • Included volatility clustering effects
   • Implementation challenges with numerical stability
   • Led to issues with extreme value generation

3. Lessons Learned

   • Direct volatility modeling proved more complex than anticipated
   • Simple bounds and scaling may be more robust than sophisticated projections
   • Need to balance sophistication with stability
   • Market awareness improved results but requires careful calibration
   • Systematic backtesting revealed weaknesses not apparent in cherry-picked tests

Uncertainty Estimation Improvements

Initial changes focused on improving the model's uncertainty estimates:

• Enhanced CI calculation with more consistent coverage
• Better balanced coverage between stress and normal periods
• Results showed more reliable performance (88% for 95% CI, ~60-65% for 68% CI)
• Maintained good performance across different market conditions

Attempted Market Regime Analysis (Reverted)

Subsequently attempted to add market regime awareness:

• Added volatility regime detection and adaptation
• Incorporated trend strength and market condition analysis
• While theoretically promising, this change was ultimately reverted because:
  • Added complexity without clear performance benefits
  • Introduced additional failure modes in backtests
  • Benefits didn't justify the increased complexity
  • Original uncertainty improvements were already handling different market conditions well

Key Learning

Sometimes simpler is better: The initial uncertainty improvements provided robust performance without needing explicit regime detection. This experience reinforces that additional complexity should only be retained when it provides clear, measurable benefits.

Future Considerations

1. Consider returning to simpler uncertainty estimation with targeted improvements
2. Focus on numerical stability while maintaining market awareness
3. Need better validation of extreme scenario handling
4. Consider alternative approaches to long-term uncertainty growth

Key insight: While more sophisticated volatility modeling seemed promising, the added complexity introduced more failure modes. Future work should focus on making the existing system more robust rather than adding new layers of complexity.

Current Implementation

The model uses four main functions:

1. analyze_trends(): Calculates cycle-position-specific log returns
2. calculate_volatility(): Computes era-adjusted volatility estimates
3. project_prices(): Generates price projections using Monte Carlo simulation
4. get_projection_adjustments(): Handles uncertainty scaling over time

Strengths

• Well-calibrated uncertainty estimates for post-2013 data
• Handles exponential price growth naturally
• Balances complexity with interpretability
• Adapts to different market eras
• Maintains consistent performance across multiple validation periods

Key Learnings from Recent Attempts

1. Added complexity doesn't necessarily improve performance:

   • Multiple sub-periods led to over-fitting
   • Dynamic regime detection made the model too reactive
   • Weighted regression provided similar results with more complexity

2. Transition period challenges:

   • 2015-2018 remains consistently challenging across approaches
   • Sharp regime changes are difficult to model without compromising overall stability
   • Simple era boundaries may be as effective as more complex transitions

3. Model stability considerations:

   • Simpler approaches tend to be more robust
   • Era-based adjustments provide good balance of adaptability and stability
   • Over-optimization for specific periods can harm general performance

Future Improvement Possibilities

1. Conservative Approaches:

   • Fine-tune existing era boundaries
   • Adjust scaling factors within current framework
   • Optimize window sizes for different periods

2. Alternative Approaches to Consider:

   • Hybrid models combining multiple timeframes
   • Conditional volatility models
   • Cycle strength indicators

3. Areas Needing Further Research:

   • Better handling of regime transitions
   • More robust volatility estimation during market structure changes
   • Improved cycle position effects near halving events

Usage Notes

The model works best when:

• Using post-2013 data
• Trained on at least one full cycle of data
• Used for medium-term projections (months to years)
• Interpreted probabilistically rather than as point forecasts

The confidence intervals should be understood as ranges of likely outcomes based on historical patterns, not hard bounds on future prices. When using the model, particular attention should be paid to the training period selection, as this can significantly impact projection quality.

Special consideration should be given to projections spanning major market structure changes or halving events, as these periods have shown higher uncertainty in backtests.