sam/bitcoin-model
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

New, streamlined NOTES.

Browse Source
This commit is contained in:
sam
2024-11-16 21:37:24 -08:00
parent 5679d99ece
commit 06f5620cb3
1 changed files with 312 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files
NOTES.md
+312 -208
View File
@@ -1,257 +1,361 @@
# Bitcoin Price Model Description # Bitcoin Price Model Documentation
## Overview ## Model Overview
This Bitcoin price prediction model uses a combination of log returns, cycle awareness, and Monte Carlo simulation to generate price projections with confidence intervals. The model was developed through several iterations, with each refinement aimed at improving accuracy and reliability. A probabilistic price projection model combining log returns analysis, cycle awareness, and Monte Carlo simulation. The model generates projected price ranges with confidence intervals, balancing short-term market dynamics with long-term cyclical patterns.
## Core Components ## Core Design Principles
### 1. Trend Analysis ### 1. Return Analysis
- Uses log returns of Bitcoin prices to better handle exponential growth patterns - Uses log returns for better handling of exponential growth
- Groups returns by position within the halving cycle (0-1460 days) - Combines multiple timeframes for volatility estimation
- Applies simple moving average smoothing to reduce noise while preserving underlying patterns - Implements adaptive window sizing based on market conditions
- Position within cycle is calculated linearly between known halving dates - Handles volatility clustering through regime-aware adjustments
### 2. Market Era Adjustments ### 2. Cycle Integration
Uses distinct eras with specific characteristics:
- Early (2013-2017): Higher volatility, conservative trends, moderate trend following
- Transition (2017-2020): Slightly elevated uncertainty during futures market introduction
- Mature (2020+): Reduced volatility, balanced trends, lighter trend following
Each era has specific scaling factors for volatility, trend expectations, and trend following behavior
### 3. Volatility Estimation
Uses a multi-timeframe approach:
- Short window (30 days)
- Medium window (90 days)
- Long window (180 days)
- Combines these using weighted exponential moving averages
- Applies era-specific scaling factors
### 4. Price Projection
- Uses Monte Carlo simulation with 1000 paths
- Incorporates era-aware adjustments to volatility and trends
- Generates both point estimates and confidence intervals
- Projects forward using cycle-aware return expectations
## Key Features
### Log Returns
Instead of working directly with prices or simple returns, the model uses log returns which:
- Better handle Bitcoin's exponential price growth
- Provide more stable statistical properties
- Allow for simpler cumulative return calculations
### Cycle Awareness
- Recognizes Bitcoin's ~4 year (1460 day) halving cycle - Recognizes Bitcoin's ~4 year (1460 day) halving cycle
- Maps historical returns to cycle positions - Maps historical returns to cycle positions (0-1 scale)
- Allows the model to capture recurring patterns around halving events - Adjusts expectations based on position in cycle
- Handles transitions between cycles with uncertainty scaling
### Enhanced Monte Carlo ### 3. Market Era Recognition
- Base simulation using normal distribution Three distinct eras with specific characteristics:
- Includes era-specific adjustments for volatility and trends - Early (2013-2017): Higher base volatility, conservative trends
- Generates both median projections and confidence intervals - Transition (2017-2020): Futures market introduction period
- Well-calibrated uncertainty estimates for post-2013 data - Mature (2020+): Institutional participation, reduced base volatility
## Performance Metrics ### 4. Uncertainty Estimation
When trained on 2016-2024 (two full cycles): - Generates both point estimates and confidence intervals
- MAPE: 11.7% - Adapts uncertainty based on market conditions
- RMSE: ~$9,011 - Uses asymmetric volatility response
- Max Error: $18,786 - Implements dynamic confidence interval calibration
- 95% CI Coverage: 99.5%
- 68% CI Coverage: 69.0%
## Model Limitations ## Architecture
- Not suitable for pre-2013 market data
- Assumes future cycles will resemble post-2013 patterns
- Uses simple linear interpolation between halving dates
- May not capture extreme market events well
- Shows increased error in transition periods (2015-2018)
## Development History ### Key Components
1. Started with direct cycle analysis of returns 1. **Trend Analysis (`analyze_trends`)**
2. Added log-based analysis for better handling of exponential growth - Calculates cycle-position-specific returns
3. Refined volatility calculation using multiple timeframes - Applies position-aware smoothing
4. Calibrated confidence intervals through era-aware scaling - Handles cycle boundaries
5. Attempted model refinements for transition periods
### Part 1: Failed Paths 2. **Volatility Estimation (`calculate_volatility`)**
- Adaptive window sizing
- Multi-timeframe integration
- Era-specific scaling
- Regime detection and response
#### Attempt 1: Era Subdivision 3. **Price Projection (`project_prices`)**
- Split transition period into multiple sub-periods - Monte Carlo simulation engine
- Added skew adjustments for different market phases - Dynamic uncertainty scaling
- Results: Made the model more complex without clear benefits - Confidence interval calculation
- Outcome: Abandoned in favor of simpler approach - Trend integration
#### Attempt 2: Dynamic Regime Detection 4. **Projection Adjustment (`get_projection_adjustments`)**
- Implemented real-time market regime detection - Time-varying uncertainty scaling
- Used volatility ratios and trend strength metrics - Market condition response
- Results: Made the model too reactive to short-term changes - Cycle position awareness
- Outcome: Less stable than original era-based approach - Minimum uncertainty bounds
#### Attempt 3: Weighted Rolling Regression # Model Performance & Validation
- Used weighted regression for trend estimation
- Combined cycle position and time-based weights
- Results: Similar performance to original model but more complex
- Key Findings:
- Recent Period: Slightly worse (MAPE 12.4% vs 11.7%)
- Mid Period: Marginally better coverage but similar overall
- Early Period: Similar coverage but worse accuracy
- Outcome: Returned to original implementation due to comparable performance with less complexity
### Part 2: Backtest Refinements ## Performance Characteristics
#### Backtest Analysis Insights ### Normal Market Conditions
- MAPE: 30-40% typical
- 95% CI Coverage: ~95%
- 68% CI Coverage: ~73%
- Best performance in mature market periods (2020+)
- Most reliable for 3-6 month horizons
After running comprehensive backtests across multiple periods from 2013-2024, we identified: ### Stress Periods
- Best performance during "normal" market conditions with MAPE ~30-35% - MAPE: 30-60%
- Significant degradation during stress periods (COVID crash, 2021 peak) with MAPE reaching 57% - 95% CI Coverage: ~95%
- Inconsistent confidence interval coverage, particularly for 68% CI (ranging from 20% to 91%) - 68% CI Coverage: ~76%
- Largest errors ($45-52K) concentrated around major market events - Wider but well-calibrated confidence intervals
- Maintains reliability through increased uncertainty
#### Methodology Changes ### Key Strengths
1. Consistent confidence interval coverage
2. Rapid adaptation to volatility changes
3. Robust handling of cycle transitions
4. Well-calibrated uncertainty estimates
1. Refined Backtest Framework ### Known Limitations
- Implemented more granular 6-month step testing periods 1. Higher error during market structure changes
- Standardized minimum training period (2 years) and validation window (8 years) 2. Increased uncertainty in early cycle periods
- Separated results into "normal" and "stress" periods for clearer performance assessment 3. Limited incorporation of external factors
4. May underestimate extreme events
2. Performance Evaluation Approach ## Validation Framework
- Recognized that optimizing for stress periods could compromise normal period performance
- Decided to focus optimization efforts on normal market conditions while accepting wider margins of error during stress periods
- Enhanced results reporting to separate normal and stress period metrics
#### Key Learnings ### Backtest Configuration
- Minimum training period: 8 years
- Validation period: 2 years
- Rolling window approach
- Separate evaluation of normal/stress periods
1. Model focus should align with intended use: ### Key Test Periods
- As a long-term cycle-aware model, primary optimization should target normal market conditions 1. **Cycle Transitions**
- Stress period performance is informative but secondary - Pre/post halving periods
- Wider confidence intervals during transitions may be more appropriate than trying to predict extreme moves - Historical halvings (2016, 2020, 2024)
- Cycle peak/trough transitions
2. Backtest methodology matters: 2. **Market Structure Changes**
- Previous ad-hoc testing may have over-emphasized stress period performance - Futures introduction (2017)
- Systematic testing with consistent windows provides clearer picture of baseline performance - Institution adoption (2020-2021)
- Special period testing remains valuable for understanding limitations - Major market events (e.g., COVID crash)
#### Future Considerations 3. **Recent History**
- 2021 bull market
- 2022 drawdown
- 2024 recovery
1. Conservative improvements to explore: ### Validation Metrics
- Fine-tune uncertainty scaling for more consistent CI coverage during normal periods 1. **Accuracy Measures**
- Consider dynamic uncertainty scaling based on market conditions - MAPE (Mean Absolute Percentage Error)
- Potential for regime-aware parameter adjustments - RMSE (Root Mean Square Error)
- Maximum deviation
2. Areas needing further investigation: 2. **Calibration Measures**
- Optimal balance between normal and stress period performance - Confidence interval coverage
- Impact of training window length on model stability - Uncertainty estimation accuracy
- Methods for early detection of transition into stress periods - Regime transition handling
### Uncertainty Estimation Improvements 3. **Stability Measures**
- Parameter sensitivity
- Training period dependence
- Regime change response
Attempted several approaches to improve uncertainty estimation in the model: # Technical Implementation
1. Initial market-aware uncertainty scaling ## Core Functions
- Added dynamic scaling based on market conditions
- Incorporated trend strength, drawdowns, and volatility regimes
- Improved CI coverage consistency between stress/normal periods
- Results showed more balanced coverage (88% for 95% CI, ~60-65% for 68% CI)
2. Volatility projection system ### Volatility Calculation
- Attempted to model how volatility evolves through cycles ```python
- Added cycle-aware volatility forecasting def calculate_volatility(df, short_window=30, medium_window=90, long_window=180):
- Included volatility clustering effects """
- Implementation challenges with numerical stability Adaptive volatility calculation combining multiple timeframes.
- Led to issues with extreme value generation
3. Lessons Learned Features:
- Direct volatility modeling proved more complex than anticipated - Dynamic window sizing based on market conditions
- Simple bounds and scaling may be more robust than sophisticated projections - Era-specific scaling factors
- Need to balance sophistication with stability - Regime-aware adjustments
- Market awareness improved results but requires careful calibration - Robust error handling and fallbacks
- Systematic backtesting revealed weaknesses not apparent in cherry-picked tests """
```
#### Uncertainty Estimation Improvements Key parameters:
- `short_window`: Fast response (default 30 days)
- `medium_window`: Primary estimate (default 90 days)
- `long_window`: Stability baseline (default 180 days)
Initial changes focused on improving the model's uncertainty estimates: Adaptive features:
- Enhanced CI calculation with more consistent coverage - Windows shrink in high volatility periods
- Better balanced coverage between stress and normal periods - Expand during low volatility
- Results showed more reliable performance (88% for 95% CI, ~60-65% for 68% CI) - Minimum size constraints for stability
- Maintained good performance across different market conditions - Weighted combination based on regime
#### Attempted Market Regime Analysis (Reverted) ### Cycle Position
```python
def get_cycle_position(date, halving_dates):
"""
Calculate position in halving cycle (0 to 1).
0 = halving event
1 = just before next halving
"""
```
Subsequently attempted to add market regime awareness: Position calculation:
- Added volatility regime detection and adaptation - Linear interpolation between halvings
- Incorporated trend strength and market condition analysis - Special handling for pre-first-halving
- While theoretically promising, this change was ultimately reverted because: - Extension mechanism for future cycles
- Added complexity without clear performance benefits - Built-in boundary condition handling
- Introduced additional failure modes in backtests
- Benefits didn't justify the increased complexity
- Original uncertainty improvements were already handling different market conditions well
#### Key Learning ### Price Projection
Sometimes simpler is better: The initial uncertainty improvements provided robust performance without needing explicit regime detection. This experience reinforces that additional complexity should only be retained when it provides clear, measurable benefits. ```python
def project_prices(df, days_forward=365, simulations=1000,
confidence_levels=[0.95, 0.68]):
"""
Generate price projections with confidence intervals.
#### Future Considerations Core simulation parameters:
- Number of paths: 1000
- Confidence levels: 95% and 68%
- Dynamic uncertainty scaling
"""
```
1. Consider returning to simpler uncertainty estimation with targeted improvements ## Data Requirements
2. Focus on numerical stability while maintaining market awareness
3. Need better validation of extreme scenario handling
4. Consider alternative approaches to long-term uncertainty growth
Key insight: While more sophisticated volatility modeling seemed promising, the added complexity introduced more failure modes. Future work should focus on making the existing system more robust rather than adding new layers of complexity. ### Input Data
Minimum fields:
- Date
- Close price
- Trading volume (optional)
- High/Low (optional)
## Current Implementation Format requirements:
The model uses four main functions: - Daily data preferred
1. `analyze_trends()`: Calculates cycle-position-specific log returns - Sorted chronologically
2. `calculate_volatility()`: Computes era-adjusted volatility estimates - No missing dates
3. `project_prices()`: Generates price projections using Monte Carlo simulation - Prices > 0
4. `get_projection_adjustments()`: Handles uncertainty scaling over time
## Strengths ### Training Data
- Well-calibrated uncertainty estimates for post-2013 data Minimum requirements:
- Handles exponential price growth naturally - 2 years for basic operation
- Balances complexity with interpretability - 8 years recommended
- Adapts to different market eras - Must include at least one cycle transition
- Maintains consistent performance across multiple validation periods - Should span multiple market regimes
## Key Learnings from Recent Attempts ## Error Handling
1. Added complexity doesn't necessarily improve performance:
- Multiple sub-periods led to over-fitting
- Dynamic regime detection made the model too reactive
- Weighted regression provided similar results with more complexity
2. Transition period challenges: ### Data Validation
- 2015-2018 remains consistently challenging across approaches - Missing value detection and interpolation
- Sharp regime changes are difficult to model without compromising overall stability - Outlier identification
- Simple era boundaries may be as effective as more complex transitions - Zero/negative price handling
- Volume anomaly detection
3. Model stability considerations: ### Runtime Guards
- Simpler approaches tend to be more robust - Minimum data length checks
- Era-based adjustments provide good balance of adaptability and stability - Window size validation
- Over-optimization for specific periods can harm general performance - Numerical stability checks
- Regime transition handling
## Future Improvement Possibilities ### Fallback Mechanisms
1. Conservative Approaches: 1. Simple volatility calculation
- Fine-tune existing era boundaries 2. Default uncertainty estimates
- Adjust scaling factors within current framework 3. Conservative parameter sets
- Optimize window sizes for different periods 4. Standard cycle assumption
2. Alternative Approaches to Consider: ## Memory and Performance
- Hybrid models combining multiple timeframes
- Conditional volatility models
- Cycle strength indicators
3. Areas Needing Further Research: ### Optimization Features
- Better handling of regime transitions - Efficient numpy operations
- More robust volatility estimation during market structure changes - Vectorized calculations where possible
- Improved cycle position effects near halving events - Smart data windowing
- Caching of intermediate results
## Usage Notes ### Resource Usage
The model works best when: Typical requirements for 10-year dataset:
- Using post-2013 data - Memory: ~100MB
- Trained on at least one full cycle of data - CPU: ~2-5 seconds per projection
- Used for medium-term projections (months to years) - Storage: Negligible
- Interpreted probabilistically rather than as point forecasts
The confidence intervals should be understood as ranges of likely outcomes based on historical patterns, not hard bounds on future prices. When using the model, particular attention should be paid to the training period selection, as this can significantly impact projection quality. ### Parallelization
- Multiprocessing support for backtests
- Independent path simulation
- Multiple period analysis
- Backtest parallelization
Special consideration should be given to projections spanning major market structure changes or halving events, as these periods have shown higher uncertainty in backtests. # Development History & Evolution
## Major Versions
### Version 1.0 (Initial Implementation)
- Basic log return analysis
- Fixed volatility windows
- Simple cycle position calculation
- Base Monte Carlo simulation
### Version 2.0 (Market Structure)
- Added era-based adjustments
- Improved cycle handling
- Multiple timeframe volatility
- Enhanced Monte Carlo engine
### Version 3.0 (Current)
- Adaptive volatility windows
- Dynamic uncertainty scaling
- Improved regime detection
- Enhanced confidence interval calibration
## Key Improvements
### Volatility Estimation
1. **Fixed → Adaptive Windows**
- Initial: Fixed 30/90/180 day windows
- Current: Dynamic sizing based on regime
- Result: Better regime transition handling
2. **Uncertainty Calibration**
- Initial: Fixed scaling factors
- Current: Market-aware dynamic scaling
- Result: More reliable confidence intervals
3. **Era Recognition**
- Initial: Single model for all periods
- Current: Era-specific adjustments
- Result: Better handling of market evolution
### Simulation Engine
1. **Path Generation**
- Initial: Basic random walks
- Current: Regime-aware path simulation
- Result: More realistic price trajectories
2. **Confidence Intervals**
- Initial: Fixed width
- Current: Dynamic, asymmetric intervals
- Result: Better calibrated uncertainty
## Failed Experiments
### 1. Complex Regime Detection
- Attempted multiple indicator fusion
- Added excessive complexity
- Reduced model stability
- Reverted to simpler approach
### 2. Machine Learning Integration
- Tested neural network components
- Reduced interpretability
- Inconsistent improvements
- Kept traditional statistical approach
### 3. External Factor Integration
- Tried incorporating macro indicators
- Added noise to projections
- Complicated parameter estimation
- Maintained focus on price dynamics
## Recent Improvements (2024)
### Adaptive Volatility Windows
- Implementation: Dynamic window sizing
- Purpose: Better regime handling
- Results:
- Improved 95% CI coverage to ~95%
- Better stress period handling
- More reliable uncertainty estimates
### Performance Metrics
Normal Periods:
- MAPE: 39.9%
- RMSE: $12,007
- 95% CI Coverage: 95.9%
- 68% CI Coverage: 72.5%
Stress Periods:
- MAPE: 32.8%
- RMSE: $12,794
- 95% CI Coverage: 95.2%
- 68% CI Coverage: 76.1%
## Future Directions
### Short Term
1. Fine-tune adaptive parameters
2. Improve transition period handling
3. Enhanced backtest framework
4. Additional regime indicators
### Medium Term
1. Cycle strength indicators
2. Volume analysis integration
3. Improved documentation
4. Performance optimization
### Long Term
1. Real-time adaptation framework
2. Advanced regime detection
3. Market microstructure integration
4. External API integration