Methodology

Rules

• Questions are registered before they close, each with an https resolution source. Predictions are accepted only before the close time and are timestamped server-side.
• Questions, predictions and resolutions are appended to a SHA-256 hash chain. Anyone can recompute it at /api/v1/ledger/verify.
• A resolved question is final. Resolution requires an outcome (0 or 1) and an evidence URL. Voided questions are excluded from scoring.
• No synthetic, demo or back-filled data. Empty means empty.
• Jev is queried with a fixed prompt template (noul-v1): the question context (or text) as state and the instruction "Will this resolve YES? <question>". The model version Jev reports is recorded with each prediction.
• Leaderboards default to first-party runs. Third-party submissions are labelled third_party and are self-attested.

Metrics

• Brier score, log loss, expected calibration error (10 equal-width bins), Wilson 95% intervals per bin, Brier skill versus the sample base rate.
• Models are ranked only at 30 or more resolved predictions; samples under 100 are labelled "limited".
• Recalibration is isotonic regression (pool adjacent violators) fitted on resolved outcomes only. It returns the nearest-block Wilson interval and the supporting sample size, and refuses to answer below 30 samples.
• Drift compares the latest window against earlier history; a flag requires both a material Brier increase and a Welch z-score above 2.33 on per-prediction squared errors. ECE is shown but not used to flag, since it is biased on small windows.

Limits

Calibration measured here is specific to the question types registered in the ledger and does not transfer automatically to your data. For your own distribution use the paid label-based endpoint with your own outcomes.