Improve forecast accuracy iteratively

Improving a forecasting playbook on a domain-specific dataset without web research (airgapped)

• · train dataset: [{ id, question, strike, category, is_deadline, direction, ladder_rung, trivial_bound, outcome }]
• · holdout dataset: same structure, outcome hidden
• · gen1_playbook_md: initial forecasting doctrine
• · generations: N rounds of improve cycle

• · generational playbooks (improved after each round)
• · train Brier score per generation
• · calibration analysis per forecast type
• · systematic errors identified and addressed
• · holdout Brier score (final test)

• · agent forecaster (per generation, consumes playbook)
• · verifier agent (scores against train, computes Brier + calibration)
• · diagnostician agent (identifies systematic errors, rewrites playbook)
• · deterministic Brier scoring (authoritative, leak-proof)

• · train/holdout data split clean (outcome NEVER shown to forecaster)
• · base playbook in markdown format
• · Brier scoring deterministic (no RNG)

• · train data leaks outcome to forecaster (taints all rounds)
• · calibration overfits to train (holdout performance drops)
• · playbook converges to local optimum (no further improvement)
• · systematic error diagnosis misses root cause

• · Train/holdout split enforced; outcome hidden from forecaster
• · 3 schemas: FORECAST_SCHEMA, VERIFIER_SCHEMA, DIAG_SCHEMA
• · Deterministic Brier scoring (authoritative gate)
• · Calibration analysis per forecast type (bin-by-ladder-rung, etc.)
• · N generations configurable; measured on held-out set
• · No web research (airgapped, leak-proof)