Mega-squash

Implement some basic backtesting.

Switch to simpler log-based projection.

Add improved vol calculation.

Implement trend smoothing.

Add Claude's notes on current model.

Run projections and backtests in parallel.

Add market maturity projection adjustments.

Improve volume handling in market maturity calculations.

New backtest suite proposed by Claude.

Fix create_plots() output.

Merge branch 'new-backtests' into market-maturity-backtests

Add era-aware market maturity adjustments.

Add updated notes from Claude.

Add justfile to simplify organizing results for comparison.

Run more projections from various start dates.

Tuning session; removed market maturity.

The market maturity score only complicated the model with no clear
benefit. Still working on getting the various backtests tuned.

Add Claude's notes from recent session.

More helpful additions to workflow.

New systematic backtest framework.

Add notes on new backtesting framework.

Use ruff linter.

Tweak the backtests.

* Start from 2011 instead of 2013.
* Validate over two years instead of one.

Improve uncertainty estimation.

Add Claude's notes from the last revision.

Add adaptive volatility window.

New, streamlined NOTES.

Fix projection plot bugs.

Update prices.csv.

Actually use long-term vol in adaptive calculation.

Use more conservative 1e-6 to prevent division by zero.

it's an error not to provide halving dates

warn when val period shorter than projection period

Update prices.csv

Manage output files in less-janky fashion.

Use market fundamentals intsead of empirical era adjustments.

Improve CI coverage.

Use S2F metrics for trend analysis.

update prices

Merge branch 'next' into mkt-fndm

Update prices.

Merge branch 'next' into mkt-fndm

Add CDPR plot.

Merge branch 'next' into mkt-fndm

Update prices.

Slight optimization to cdpr plot gen.

Update prices.

Merge branch 'next' into mkt-fndm

Add price to CDPR plot.

Update prices.

Add .private to .gitignore.

Update prices.

.gitignore

@@ -1 +1,5 @@
-*.png
+bitcoin_*.png
+bitcoin_*.txt
+/output
+scratch.py
+/.private

NOTES.md

@@ -0,0 +1,361 @@
+# Bitcoin Price Model Documentation
+
+## Model Overview
+A probabilistic price projection model combining log returns analysis, cycle awareness, and Monte Carlo simulation. The model generates projected price ranges with confidence intervals, balancing short-term market dynamics with long-term cyclical patterns.
+
+## Core Design Principles
+
+### 1. Return Analysis
+- Uses log returns for better handling of exponential growth
+- Combines multiple timeframes for volatility estimation
+- Implements adaptive window sizing based on market conditions
+- Handles volatility clustering through regime-aware adjustments
+
+### 2. Cycle Integration
+- Recognizes Bitcoin's ~4 year (1460 day) halving cycle
+- Maps historical returns to cycle positions (0-1 scale)
+- Adjusts expectations based on position in cycle
+- Handles transitions between cycles with uncertainty scaling
+
+### 3. Market Era Recognition
+Three distinct eras with specific characteristics:
+- Early (2013-2017): Higher base volatility, conservative trends
+- Transition (2017-2020): Futures market introduction period
+- Mature (2020+): Institutional participation, reduced base volatility
+
+### 4. Uncertainty Estimation
+- Generates both point estimates and confidence intervals
+- Adapts uncertainty based on market conditions
+- Uses asymmetric volatility response
+- Implements dynamic confidence interval calibration
+
+## Architecture
+
+### Key Components
+1. **Trend Analysis (`analyze_trends`)**
+   - Calculates cycle-position-specific returns
+   - Applies position-aware smoothing
+   - Handles cycle boundaries
+
+2. **Volatility Estimation (`calculate_volatility`)**
+   - Adaptive window sizing
+   - Multi-timeframe integration
+   - Era-specific scaling
+   - Regime detection and response
+
+3. **Price Projection (`project_prices`)**
+   - Monte Carlo simulation engine
+   - Dynamic uncertainty scaling
+   - Confidence interval calculation
+   - Trend integration
+
+4. **Projection Adjustment (`get_projection_adjustments`)**
+   - Time-varying uncertainty scaling
+   - Market condition response
+   - Cycle position awareness
+   - Minimum uncertainty bounds
+
+# Model Performance & Validation
+
+## Performance Characteristics
+
+### Normal Market Conditions
+- MAPE: 30-40% typical
+- 95% CI Coverage: ~95%
+- 68% CI Coverage: ~73%
+- Best performance in mature market periods (2020+)
+- Most reliable for 3-6 month horizons
+
+### Stress Periods
+- MAPE: 30-60%
+- 95% CI Coverage: ~95%
+- 68% CI Coverage: ~76%
+- Wider but well-calibrated confidence intervals
+- Maintains reliability through increased uncertainty
+
+### Key Strengths
+1. Consistent confidence interval coverage
+2. Rapid adaptation to volatility changes
+3. Robust handling of cycle transitions
+4. Well-calibrated uncertainty estimates
+
+### Known Limitations
+1. Higher error during market structure changes
+2. Increased uncertainty in early cycle periods
+3. Limited incorporation of external factors
+4. May underestimate extreme events
+
+## Validation Framework
+
+### Backtest Configuration
+- Minimum training period: 8 years
+- Validation period: 2 years
+- Rolling window approach
+- Separate evaluation of normal/stress periods
+
+### Key Test Periods
+1. **Cycle Transitions**
+   - Pre/post halving periods
+   - Historical halvings (2016, 2020, 2024)
+   - Cycle peak/trough transitions
+
+2. **Market Structure Changes**
+   - Futures introduction (2017)
+   - Institution adoption (2020-2021)
+   - Major market events (e.g., COVID crash)
+
+3. **Recent History**
+   - 2021 bull market
+   - 2022 drawdown
+   - 2024 recovery
+
+### Validation Metrics
+1. **Accuracy Measures**
+   - MAPE (Mean Absolute Percentage Error)
+   - RMSE (Root Mean Square Error)
+   - Maximum deviation
+
+2. **Calibration Measures**
+   - Confidence interval coverage
+   - Uncertainty estimation accuracy
+   - Regime transition handling
+
+3. **Stability Measures**
+   - Parameter sensitivity
+   - Training period dependence
+   - Regime change response
+
+# Technical Implementation
+
+## Core Functions
+
+### Volatility Calculation
+```python
+def calculate_volatility(df, short_window=30, medium_window=90, long_window=180):
+    """
+    Adaptive volatility calculation combining multiple timeframes.
+
+    Features:
+    - Dynamic window sizing based on market conditions
+    - Era-specific scaling factors
+    - Regime-aware adjustments
+    - Robust error handling and fallbacks
+    """
+```
+
+Key parameters:
+- `short_window`: Fast response (default 30 days)
+- `medium_window`: Primary estimate (default 90 days)
+- `long_window`: Stability baseline (default 180 days)
+
+Adaptive features:
+- Windows shrink in high volatility periods
+- Expand during low volatility
+- Minimum size constraints for stability
+- Weighted combination based on regime
+
+### Cycle Position
+```python
+def get_cycle_position(date, halving_dates):
+    """
+    Calculate position in halving cycle (0 to 1).
+    0 = halving event
+    1 = just before next halving
+    """
+```
+
+Position calculation:
+- Linear interpolation between halvings
+- Special handling for pre-first-halving
+- Extension mechanism for future cycles
+- Built-in boundary condition handling
+
+### Price Projection
+```python
+def project_prices(df, days_forward=365, simulations=1000,
+                  confidence_levels=[0.95, 0.68]):
+    """
+    Generate price projections with confidence intervals.
+
+    Core simulation parameters:
+    - Number of paths: 1000
+    - Confidence levels: 95% and 68%
+    - Dynamic uncertainty scaling
+    """
+```
+
+## Data Requirements
+
+### Input Data
+Minimum fields:
+- Date
+- Close price
+- Trading volume (optional)
+- High/Low (optional)
+
+Format requirements:
+- Daily data preferred
+- Sorted chronologically
+- No missing dates
+- Prices > 0
+
+### Training Data
+Minimum requirements:
+- 2 years for basic operation
+- 8 years recommended
+- Must include at least one cycle transition
+- Should span multiple market regimes
+
+## Error Handling
+
+### Data Validation
+- Missing value detection and interpolation
+- Outlier identification
+- Zero/negative price handling
+- Volume anomaly detection
+
+### Runtime Guards
+- Minimum data length checks
+- Window size validation
+- Numerical stability checks
+- Regime transition handling
+
+### Fallback Mechanisms
+1. Simple volatility calculation
+2. Default uncertainty estimates
+3. Conservative parameter sets
+4. Standard cycle assumption
+
+## Memory and Performance
+
+### Optimization Features
+- Efficient numpy operations
+- Vectorized calculations where possible
+- Smart data windowing
+- Caching of intermediate results
+
+### Resource Usage
+Typical requirements for 10-year dataset:
+- Memory: ~100MB
+- CPU: ~2-5 seconds per projection
+- Storage: Negligible
+
+### Parallelization
+- Multiprocessing support for backtests
+- Independent path simulation
+- Multiple period analysis
+- Backtest parallelization
+
+# Development History & Evolution
+
+## Major Versions
+
+### Version 1.0 (Initial Implementation)
+- Basic log return analysis
+- Fixed volatility windows
+- Simple cycle position calculation
+- Base Monte Carlo simulation
+
+### Version 2.0 (Market Structure)
+- Added era-based adjustments
+- Improved cycle handling
+- Multiple timeframe volatility
+- Enhanced Monte Carlo engine
+
+### Version 3.0 (Current)
+- Adaptive volatility windows
+- Dynamic uncertainty scaling
+- Improved regime detection
+- Enhanced confidence interval calibration
+
+## Key Improvements
+
+### Volatility Estimation
+1. **Fixed → Adaptive Windows**
+   - Initial: Fixed 30/90/180 day windows
+   - Current: Dynamic sizing based on regime
+   - Result: Better regime transition handling
+
+2. **Uncertainty Calibration**
+   - Initial: Fixed scaling factors
+   - Current: Market-aware dynamic scaling
+   - Result: More reliable confidence intervals
+
+3. **Era Recognition**
+   - Initial: Single model for all periods
+   - Current: Era-specific adjustments
+   - Result: Better handling of market evolution
+
+### Simulation Engine
+1. **Path Generation**
+   - Initial: Basic random walks
+   - Current: Regime-aware path simulation
+   - Result: More realistic price trajectories
+
+2. **Confidence Intervals**
+   - Initial: Fixed width
+   - Current: Dynamic, asymmetric intervals
+   - Result: Better calibrated uncertainty
+
+## Failed Experiments
+
+### 1. Complex Regime Detection
+- Attempted multiple indicator fusion
+- Added excessive complexity
+- Reduced model stability
+- Reverted to simpler approach
+
+### 2. Machine Learning Integration
+- Tested neural network components
+- Reduced interpretability
+- Inconsistent improvements
+- Kept traditional statistical approach
+
+### 3. External Factor Integration
+- Tried incorporating macro indicators
+- Added noise to projections
+- Complicated parameter estimation
+- Maintained focus on price dynamics
+
+## Recent Improvements (2024)
+
+### Adaptive Volatility Windows
+- Implementation: Dynamic window sizing
+- Purpose: Better regime handling
+- Results:
+  - Improved 95% CI coverage to ~95%
+  - Better stress period handling
+  - More reliable uncertainty estimates
+
+### Performance Metrics
+Normal Periods:
+- MAPE: 39.9%
+- RMSE: $12,007
+- 95% CI Coverage: 95.9%
+- 68% CI Coverage: 72.5%
+
+Stress Periods:
+- MAPE: 32.8%
+- RMSE: $12,794
+- 95% CI Coverage: 95.2%
+- 68% CI Coverage: 76.1%
+
+## Future Directions
+
+### Short Term
+1. Fine-tune adaptive parameters
+2. Improve transition period handling
+3. Enhanced backtest framework
+4. Additional regime indicators
+
+### Medium Term
+1. Cycle strength indicators
+2. Volume analysis integration
+3. Improved documentation
+4. Performance optimization
+
+### Long Term
+1. Real-time adaptation framework
+2. Advanced regime detection
+3. Market microstructure integration
+4. External API integration

justfile

@@ -0,0 +1,11 @@
+run *name:
+    python3 ./model.py -n "{{ name }}"
+
+fmt:
+    black ./*.py
+
+lint:
+    ruff check ./model.py
+
+clean:
+    rm -rf output

prices.csv

@@ -1,5 +1,39 @@
 "Date","Price","Open","High","Low","Vol.","Change %"
-"11/15/2024","88,096.2","87,297.2","88,348.6","87,276.2","109.10K","0.92%"
+"12/19/2024","97,466.1","100,190.5","102,778.8","95,672.2","200.57K","-2.73%"
+"12/18/2024","100,197.8","106,140.7","106,477.6","100,049.3","187.49K","-5.60%"
+"12/17/2024","106,138.9","106,053.4","108,244.9","105,350.6","149.37K","0.08%"
+"12/16/2024","106,057.6","104,479.2","107,767.6","103,351.7","232.66K","1.55%"
+"12/15/2024","104,443.0","101,417.9","105,120.9","101,234.9","133.99K","2.98%"
+"12/14/2024","101,417.7","101,423.7","102,633.0","100,626.3","105.31K","-0.01%"
+"12/13/2024","101,426.2","100,008.3","101,891.2","99,214.2","162.40K","1.42%"
+"12/12/2024","100,009.9","101,126.3","102,495.3","99,334.5","188.96K","-1.10%"
+"12/11/2024","101,126.2","96,603.2","101,877.1","95,689.5","250.02K","4.69%"
+"12/10/2024","96,600.3","97,311.5","98,237.8","94,304.5","271.45K","-0.78%"
+"12/09/2024","97,359.4","101,129.7","101,198.6","94,395.8","321.01K","-3.72%"
+"12/08/2024","101,115.8","99,837.0","101,339.9","98,713.9","128.93K","1.28%"
+"12/07/2024","99,837.0","99,757.2","100,441.8","98,969.0","118.44K","0.08%"
+"12/06/2024","99,760.7","96,968.9","101,908.5","96,452.0","213.49K","2.88%"
+"12/05/2024","96,970.8","98,624.8","103,719.4","92,510.4","245.79K","-1.69%"
+"12/04/2024","98,635.8","95,900.1","99,030.7","94,625.3","113.77K","2.85%"
+"12/03/2024","95,898.7","95,843.9","96,245.0","93,646.1","95.08K","0.05%"
+"12/02/2024","95,846.6","97,199.3","98,082.0","94,510.6","95.80K","-1.39%"
+"12/01/2024","97,201.4","96,404.7","97,813.9","95,721.8","42.90K","0.83%"
+"11/30/2024","96,405.7","97,438.5","97,461.9","96,119.6","36.92K","-1.06%"
+"11/29/2024","97,438.0","95,645.6","98,599.8","95,405.7","76.54K","1.87%"
+"11/28/2024","95,645.5","95,881.4","96,549.0","94,648.8","68.17K","-0.25%"
+"11/27/2024","95,886.6","91,975.5","97,219.8","91,795.5","106.39K","4.25%"
+"11/26/2024","91,977.6","93,046.5","94,978.8","90,817.2","131.88K","-1.17%"
+"11/25/2024","93,064.4","97,948.8","98,872.4","92,875.3","124.98K","-4.97%"
+"11/24/2024","97,935.2","97,696.4","98,552.6","95,765.4","77.04K","0.24%"
+"11/23/2024","97,699.0","98,927.2","98,927.2","97,180.9","72.17K","-1.24%"
+"11/22/2024","98,929.7","98,381.2","99,617.4","97,182.2","132.23K","0.56%"
+"11/21/2024","98,374.5","94,308.7","98,937.2","94,063.8","179.07K","4.32%"
+"11/20/2024","94,303.9","92,252.6","94,836.1","91,517.4","121.46K","2.24%"
+"11/19/2024","92,239.2","90,482.1","93,918.5","90,378.7","146.04K","1.95%"
+"11/18/2024","90,478.7","89,843.3","92,567.1","89,410.4","150.47K","0.72%"
+"11/17/2024","89,833.7","90,569.4","91,441.6","88,736.1","81.24K","-0.78%"
+"11/16/2024","90,539.1","91,035.3","91,748.9","90,079.0","79.82K","-0.56%"
+"11/15/2024","91,046.5","87,297.2","91,833.1","87,118.8","158.19K","4.30%"
 "11/14/2024","87,294.0","90,424.8","91,726.7","86,740.2","153.34K","-3.46%"
 "11/13/2024","90,422.2","87,971.2","93,226.6","86,168.4","257.57K","2.82%"
 "11/12/2024","87,941.3","88,665.0","89,929.6","85,122.2","288.68K","-0.82%"