diff --git a/NOTES.md b/NOTES.md index 5fff35d..e9b90ff 100644 --- a/NOTES.md +++ b/NOTES.md @@ -1,257 +1,361 @@ -# Bitcoin Price Model Description +# Bitcoin Price Model Documentation -## Overview -This Bitcoin price prediction model uses a combination of log returns, cycle awareness, and Monte Carlo simulation to generate price projections with confidence intervals. The model was developed through several iterations, with each refinement aimed at improving accuracy and reliability. +## Model Overview +A probabilistic price projection model combining log returns analysis, cycle awareness, and Monte Carlo simulation. The model generates projected price ranges with confidence intervals, balancing short-term market dynamics with long-term cyclical patterns. -## Core Components +## Core Design Principles -### 1. Trend Analysis -- Uses log returns of Bitcoin prices to better handle exponential growth patterns -- Groups returns by position within the halving cycle (0-1460 days) -- Applies simple moving average smoothing to reduce noise while preserving underlying patterns -- Position within cycle is calculated linearly between known halving dates +### 1. Return Analysis +- Uses log returns for better handling of exponential growth +- Combines multiple timeframes for volatility estimation +- Implements adaptive window sizing based on market conditions +- Handles volatility clustering through regime-aware adjustments -### 2. Market Era Adjustments -Uses distinct eras with specific characteristics: -- Early (2013-2017): Higher volatility, conservative trends, moderate trend following -- Transition (2017-2020): Slightly elevated uncertainty during futures market introduction -- Mature (2020+): Reduced volatility, balanced trends, lighter trend following - -Each era has specific scaling factors for volatility, trend expectations, and trend following behavior - -### 3. Volatility Estimation -Uses a multi-timeframe approach: -- Short window (30 days) -- Medium window (90 days) -- Long window (180 days) -- Combines these using weighted exponential moving averages -- Applies era-specific scaling factors - -### 4. Price Projection -- Uses Monte Carlo simulation with 1000 paths -- Incorporates era-aware adjustments to volatility and trends -- Generates both point estimates and confidence intervals -- Projects forward using cycle-aware return expectations - -## Key Features - -### Log Returns -Instead of working directly with prices or simple returns, the model uses log returns which: -- Better handle Bitcoin's exponential price growth -- Provide more stable statistical properties -- Allow for simpler cumulative return calculations - -### Cycle Awareness +### 2. Cycle Integration - Recognizes Bitcoin's ~4 year (1460 day) halving cycle -- Maps historical returns to cycle positions -- Allows the model to capture recurring patterns around halving events +- Maps historical returns to cycle positions (0-1 scale) +- Adjusts expectations based on position in cycle +- Handles transitions between cycles with uncertainty scaling -### Enhanced Monte Carlo -- Base simulation using normal distribution -- Includes era-specific adjustments for volatility and trends -- Generates both median projections and confidence intervals -- Well-calibrated uncertainty estimates for post-2013 data +### 3. Market Era Recognition +Three distinct eras with specific characteristics: +- Early (2013-2017): Higher base volatility, conservative trends +- Transition (2017-2020): Futures market introduction period +- Mature (2020+): Institutional participation, reduced base volatility -## Performance Metrics -When trained on 2016-2024 (two full cycles): -- MAPE: 11.7% -- RMSE: ~$9,011 -- Max Error: $18,786 -- 95% CI Coverage: 99.5% -- 68% CI Coverage: 69.0% +### 4. Uncertainty Estimation +- Generates both point estimates and confidence intervals +- Adapts uncertainty based on market conditions +- Uses asymmetric volatility response +- Implements dynamic confidence interval calibration -## Model Limitations -- Not suitable for pre-2013 market data -- Assumes future cycles will resemble post-2013 patterns -- Uses simple linear interpolation between halving dates -- May not capture extreme market events well -- Shows increased error in transition periods (2015-2018) +## Architecture -## Development History -1. Started with direct cycle analysis of returns -2. Added log-based analysis for better handling of exponential growth -3. Refined volatility calculation using multiple timeframes -4. Calibrated confidence intervals through era-aware scaling -5. Attempted model refinements for transition periods +### Key Components +1. **Trend Analysis (`analyze_trends`)** + - Calculates cycle-position-specific returns + - Applies position-aware smoothing + - Handles cycle boundaries -### Part 1: Failed Paths +2. **Volatility Estimation (`calculate_volatility`)** + - Adaptive window sizing + - Multi-timeframe integration + - Era-specific scaling + - Regime detection and response -#### Attempt 1: Era Subdivision -- Split transition period into multiple sub-periods -- Added skew adjustments for different market phases -- Results: Made the model more complex without clear benefits -- Outcome: Abandoned in favor of simpler approach +3. **Price Projection (`project_prices`)** + - Monte Carlo simulation engine + - Dynamic uncertainty scaling + - Confidence interval calculation + - Trend integration -#### Attempt 2: Dynamic Regime Detection -- Implemented real-time market regime detection -- Used volatility ratios and trend strength metrics -- Results: Made the model too reactive to short-term changes -- Outcome: Less stable than original era-based approach +4. **Projection Adjustment (`get_projection_adjustments`)** + - Time-varying uncertainty scaling + - Market condition response + - Cycle position awareness + - Minimum uncertainty bounds -#### Attempt 3: Weighted Rolling Regression -- Used weighted regression for trend estimation -- Combined cycle position and time-based weights -- Results: Similar performance to original model but more complex -- Key Findings: - - Recent Period: Slightly worse (MAPE 12.4% vs 11.7%) - - Mid Period: Marginally better coverage but similar overall - - Early Period: Similar coverage but worse accuracy -- Outcome: Returned to original implementation due to comparable performance with less complexity +# Model Performance & Validation -### Part 2: Backtest Refinements +## Performance Characteristics -#### Backtest Analysis Insights +### Normal Market Conditions +- MAPE: 30-40% typical +- 95% CI Coverage: ~95% +- 68% CI Coverage: ~73% +- Best performance in mature market periods (2020+) +- Most reliable for 3-6 month horizons -After running comprehensive backtests across multiple periods from 2013-2024, we identified: -- Best performance during "normal" market conditions with MAPE ~30-35% -- Significant degradation during stress periods (COVID crash, 2021 peak) with MAPE reaching 57% -- Inconsistent confidence interval coverage, particularly for 68% CI (ranging from 20% to 91%) -- Largest errors ($45-52K) concentrated around major market events +### Stress Periods +- MAPE: 30-60% +- 95% CI Coverage: ~95% +- 68% CI Coverage: ~76% +- Wider but well-calibrated confidence intervals +- Maintains reliability through increased uncertainty -#### Methodology Changes +### Key Strengths +1. Consistent confidence interval coverage +2. Rapid adaptation to volatility changes +3. Robust handling of cycle transitions +4. Well-calibrated uncertainty estimates -1. Refined Backtest Framework -- Implemented more granular 6-month step testing periods -- Standardized minimum training period (2 years) and validation window (8 years) -- Separated results into "normal" and "stress" periods for clearer performance assessment +### Known Limitations +1. Higher error during market structure changes +2. Increased uncertainty in early cycle periods +3. Limited incorporation of external factors +4. May underestimate extreme events -2. Performance Evaluation Approach -- Recognized that optimizing for stress periods could compromise normal period performance -- Decided to focus optimization efforts on normal market conditions while accepting wider margins of error during stress periods -- Enhanced results reporting to separate normal and stress period metrics +## Validation Framework -#### Key Learnings +### Backtest Configuration +- Minimum training period: 8 years +- Validation period: 2 years +- Rolling window approach +- Separate evaluation of normal/stress periods -1. Model focus should align with intended use: - - As a long-term cycle-aware model, primary optimization should target normal market conditions - - Stress period performance is informative but secondary - - Wider confidence intervals during transitions may be more appropriate than trying to predict extreme moves +### Key Test Periods +1. **Cycle Transitions** + - Pre/post halving periods + - Historical halvings (2016, 2020, 2024) + - Cycle peak/trough transitions -2. Backtest methodology matters: - - Previous ad-hoc testing may have over-emphasized stress period performance - - Systematic testing with consistent windows provides clearer picture of baseline performance - - Special period testing remains valuable for understanding limitations +2. **Market Structure Changes** + - Futures introduction (2017) + - Institution adoption (2020-2021) + - Major market events (e.g., COVID crash) -#### Future Considerations +3. **Recent History** + - 2021 bull market + - 2022 drawdown + - 2024 recovery -1. Conservative improvements to explore: - - Fine-tune uncertainty scaling for more consistent CI coverage during normal periods - - Consider dynamic uncertainty scaling based on market conditions - - Potential for regime-aware parameter adjustments +### Validation Metrics +1. **Accuracy Measures** + - MAPE (Mean Absolute Percentage Error) + - RMSE (Root Mean Square Error) + - Maximum deviation -2. Areas needing further investigation: - - Optimal balance between normal and stress period performance - - Impact of training window length on model stability - - Methods for early detection of transition into stress periods +2. **Calibration Measures** + - Confidence interval coverage + - Uncertainty estimation accuracy + - Regime transition handling -### Uncertainty Estimation Improvements +3. **Stability Measures** + - Parameter sensitivity + - Training period dependence + - Regime change response -Attempted several approaches to improve uncertainty estimation in the model: +# Technical Implementation -1. Initial market-aware uncertainty scaling - - Added dynamic scaling based on market conditions - - Incorporated trend strength, drawdowns, and volatility regimes - - Improved CI coverage consistency between stress/normal periods - - Results showed more balanced coverage (88% for 95% CI, ~60-65% for 68% CI) +## Core Functions -2. Volatility projection system - - Attempted to model how volatility evolves through cycles - - Added cycle-aware volatility forecasting - - Included volatility clustering effects - - Implementation challenges with numerical stability - - Led to issues with extreme value generation +### Volatility Calculation +```python +def calculate_volatility(df, short_window=30, medium_window=90, long_window=180): + """ + Adaptive volatility calculation combining multiple timeframes. -3. Lessons Learned - - Direct volatility modeling proved more complex than anticipated - - Simple bounds and scaling may be more robust than sophisticated projections - - Need to balance sophistication with stability - - Market awareness improved results but requires careful calibration - - Systematic backtesting revealed weaknesses not apparent in cherry-picked tests + Features: + - Dynamic window sizing based on market conditions + - Era-specific scaling factors + - Regime-aware adjustments + - Robust error handling and fallbacks + """ +``` -#### Uncertainty Estimation Improvements +Key parameters: +- `short_window`: Fast response (default 30 days) +- `medium_window`: Primary estimate (default 90 days) +- `long_window`: Stability baseline (default 180 days) -Initial changes focused on improving the model's uncertainty estimates: -- Enhanced CI calculation with more consistent coverage -- Better balanced coverage between stress and normal periods -- Results showed more reliable performance (88% for 95% CI, ~60-65% for 68% CI) -- Maintained good performance across different market conditions +Adaptive features: +- Windows shrink in high volatility periods +- Expand during low volatility +- Minimum size constraints for stability +- Weighted combination based on regime -#### Attempted Market Regime Analysis (Reverted) +### Cycle Position +```python +def get_cycle_position(date, halving_dates): + """ + Calculate position in halving cycle (0 to 1). + 0 = halving event + 1 = just before next halving + """ +``` -Subsequently attempted to add market regime awareness: -- Added volatility regime detection and adaptation -- Incorporated trend strength and market condition analysis -- While theoretically promising, this change was ultimately reverted because: - - Added complexity without clear performance benefits - - Introduced additional failure modes in backtests - - Benefits didn't justify the increased complexity - - Original uncertainty improvements were already handling different market conditions well +Position calculation: +- Linear interpolation between halvings +- Special handling for pre-first-halving +- Extension mechanism for future cycles +- Built-in boundary condition handling -#### Key Learning -Sometimes simpler is better: The initial uncertainty improvements provided robust performance without needing explicit regime detection. This experience reinforces that additional complexity should only be retained when it provides clear, measurable benefits. +### Price Projection +```python +def project_prices(df, days_forward=365, simulations=1000, + confidence_levels=[0.95, 0.68]): + """ + Generate price projections with confidence intervals. -#### Future Considerations + Core simulation parameters: + - Number of paths: 1000 + - Confidence levels: 95% and 68% + - Dynamic uncertainty scaling + """ +``` -1. Consider returning to simpler uncertainty estimation with targeted improvements -2. Focus on numerical stability while maintaining market awareness -3. Need better validation of extreme scenario handling -4. Consider alternative approaches to long-term uncertainty growth +## Data Requirements -Key insight: While more sophisticated volatility modeling seemed promising, the added complexity introduced more failure modes. Future work should focus on making the existing system more robust rather than adding new layers of complexity. +### Input Data +Minimum fields: +- Date +- Close price +- Trading volume (optional) +- High/Low (optional) -## Current Implementation -The model uses four main functions: -1. `analyze_trends()`: Calculates cycle-position-specific log returns -2. `calculate_volatility()`: Computes era-adjusted volatility estimates -3. `project_prices()`: Generates price projections using Monte Carlo simulation -4. `get_projection_adjustments()`: Handles uncertainty scaling over time +Format requirements: +- Daily data preferred +- Sorted chronologically +- No missing dates +- Prices > 0 -## Strengths -- Well-calibrated uncertainty estimates for post-2013 data -- Handles exponential price growth naturally -- Balances complexity with interpretability -- Adapts to different market eras -- Maintains consistent performance across multiple validation periods +### Training Data +Minimum requirements: +- 2 years for basic operation +- 8 years recommended +- Must include at least one cycle transition +- Should span multiple market regimes -## Key Learnings from Recent Attempts -1. Added complexity doesn't necessarily improve performance: - - Multiple sub-periods led to over-fitting - - Dynamic regime detection made the model too reactive - - Weighted regression provided similar results with more complexity +## Error Handling -2. Transition period challenges: - - 2015-2018 remains consistently challenging across approaches - - Sharp regime changes are difficult to model without compromising overall stability - - Simple era boundaries may be as effective as more complex transitions +### Data Validation +- Missing value detection and interpolation +- Outlier identification +- Zero/negative price handling +- Volume anomaly detection -3. Model stability considerations: - - Simpler approaches tend to be more robust - - Era-based adjustments provide good balance of adaptability and stability - - Over-optimization for specific periods can harm general performance +### Runtime Guards +- Minimum data length checks +- Window size validation +- Numerical stability checks +- Regime transition handling -## Future Improvement Possibilities -1. Conservative Approaches: - - Fine-tune existing era boundaries - - Adjust scaling factors within current framework - - Optimize window sizes for different periods +### Fallback Mechanisms +1. Simple volatility calculation +2. Default uncertainty estimates +3. Conservative parameter sets +4. Standard cycle assumption -2. Alternative Approaches to Consider: - - Hybrid models combining multiple timeframes - - Conditional volatility models - - Cycle strength indicators +## Memory and Performance -3. Areas Needing Further Research: - - Better handling of regime transitions - - More robust volatility estimation during market structure changes - - Improved cycle position effects near halving events +### Optimization Features +- Efficient numpy operations +- Vectorized calculations where possible +- Smart data windowing +- Caching of intermediate results -## Usage Notes -The model works best when: -- Using post-2013 data -- Trained on at least one full cycle of data -- Used for medium-term projections (months to years) -- Interpreted probabilistically rather than as point forecasts +### Resource Usage +Typical requirements for 10-year dataset: +- Memory: ~100MB +- CPU: ~2-5 seconds per projection +- Storage: Negligible -The confidence intervals should be understood as ranges of likely outcomes based on historical patterns, not hard bounds on future prices. When using the model, particular attention should be paid to the training period selection, as this can significantly impact projection quality. +### Parallelization +- Multiprocessing support for backtests +- Independent path simulation +- Multiple period analysis +- Backtest parallelization -Special consideration should be given to projections spanning major market structure changes or halving events, as these periods have shown higher uncertainty in backtests. +# Development History & Evolution + +## Major Versions + +### Version 1.0 (Initial Implementation) +- Basic log return analysis +- Fixed volatility windows +- Simple cycle position calculation +- Base Monte Carlo simulation + +### Version 2.0 (Market Structure) +- Added era-based adjustments +- Improved cycle handling +- Multiple timeframe volatility +- Enhanced Monte Carlo engine + +### Version 3.0 (Current) +- Adaptive volatility windows +- Dynamic uncertainty scaling +- Improved regime detection +- Enhanced confidence interval calibration + +## Key Improvements + +### Volatility Estimation +1. **Fixed → Adaptive Windows** + - Initial: Fixed 30/90/180 day windows + - Current: Dynamic sizing based on regime + - Result: Better regime transition handling + +2. **Uncertainty Calibration** + - Initial: Fixed scaling factors + - Current: Market-aware dynamic scaling + - Result: More reliable confidence intervals + +3. **Era Recognition** + - Initial: Single model for all periods + - Current: Era-specific adjustments + - Result: Better handling of market evolution + +### Simulation Engine +1. **Path Generation** + - Initial: Basic random walks + - Current: Regime-aware path simulation + - Result: More realistic price trajectories + +2. **Confidence Intervals** + - Initial: Fixed width + - Current: Dynamic, asymmetric intervals + - Result: Better calibrated uncertainty + +## Failed Experiments + +### 1. Complex Regime Detection +- Attempted multiple indicator fusion +- Added excessive complexity +- Reduced model stability +- Reverted to simpler approach + +### 2. Machine Learning Integration +- Tested neural network components +- Reduced interpretability +- Inconsistent improvements +- Kept traditional statistical approach + +### 3. External Factor Integration +- Tried incorporating macro indicators +- Added noise to projections +- Complicated parameter estimation +- Maintained focus on price dynamics + +## Recent Improvements (2024) + +### Adaptive Volatility Windows +- Implementation: Dynamic window sizing +- Purpose: Better regime handling +- Results: + - Improved 95% CI coverage to ~95% + - Better stress period handling + - More reliable uncertainty estimates + +### Performance Metrics +Normal Periods: +- MAPE: 39.9% +- RMSE: $12,007 +- 95% CI Coverage: 95.9% +- 68% CI Coverage: 72.5% + +Stress Periods: +- MAPE: 32.8% +- RMSE: $12,794 +- 95% CI Coverage: 95.2% +- 68% CI Coverage: 76.1% + +## Future Directions + +### Short Term +1. Fine-tune adaptive parameters +2. Improve transition period handling +3. Enhanced backtest framework +4. Additional regime indicators + +### Medium Term +1. Cycle strength indicators +2. Volume analysis integration +3. Improved documentation +4. Performance optimization + +### Long Term +1. Real-time adaptation framework +2. Advanced regime detection +3. Market microstructure integration +4. External API integration