WEATHER PROPSBOARDMATCHUPSMODELSSTATIONSREPLAYTHE LEAGUE
← Model league
FORECAST RESEARCH / PROSPECTIVE SHADOW SEASON

Did the forecast get better?

Compare model skill, test confidence, and measure which updates added useful information before the weather arrived.

D1 COMMON COHORT276 station-days23 target datesAll 5 models · finalized observations
CONFIDENCE / ICON

Does 80% mean 80%?

0%0%20%20%40%40%60%60%80%80%100%100%Mean forecast 6.0%; observed 14.1%; 64 cases, 21 daysn=64Mean forecast 30.4%; observed 63.3%; 30 cases, 16 daysn=30Mean forecast 49.1%; observed 74.2%; 31 cases, 16 daysn=31Mean forecast 68.4%; observed 82.4%; 34 cases, 19 daysn=34Mean forecast 94.8%; observed 100.0%; 117 cases, 23 daysn=117MEAN FORECAST PROBABILITYOBSERVED FREQUENCY

Points on the diagonal are calibrated in this sample. Bars are exploratory 95% intervals when enough date blocks exist. A small bin can look convincing by chance.

PROBABILITY DISTRIBUTION

How often does it commit?

640–203020–403140–603460–8011780–100NUMBER OF FORECASTSISSUED PROBABILITY BIN (%)

Sharpness measures how forecasts use the probability range. It does not establish accuracy. The final bin includes 100%; the others exclude their upper edge.

Reliability and probability-bin data
BINNDATESMEAN FORECASTOBSERVED95% INTERVAL
0%–20%64216.0%14.1%+0.0563 to +0.2465
20%–40%301630.4%63.3%+0.4800 to +0.7858
40%–60%311649.1%74.2%+0.5946 to +0.9167
60%–80%341968.4%82.4%+0.6399 to +0.9546
80%–100%1172394.8%100.0%Only one outcome observed
PAIRED SCORES / SAME EVENTS

Who forecast it better?

ROW − COLUMNAIFSEPSWN2HRESICON
AIFS—-0.0559-0.0350-0.0263-0.0343
EPS+0.0559—+0.0209+0.0296+0.0216
WN2+0.0350-0.0209—+0.0087+0.0007
HRES+0.0263-0.0296-0.0087—-0.0081
ICON+0.0343-0.0216-0.0007+0.0081—

Each cell is row-model Brier minus column-model Brier. Negative favors the row; positive favors the column. All cells use the same complete cohort, including byes and untraded forecasts. These are forecast comparisons, not trading wins.

Pair counts, W/L/T, and uncertainty
PAIRMEAN DIFFERENCE95% INTERVALCASESDATES / BLOCKSFIRST MODEL W–L–T
AIFS / EPS-0.0559-0.0925 to -0.025227623 / 10136–133–7
AIFS / WN2-0.0350-0.0656 to -0.010127623 / 10124–138–14
AIFS / HRES-0.0263-0.0446 to -0.010027623 / 10157–112–7
AIFS / ICON-0.0343-0.0547 to -0.015527623 / 10150–116–10
EPS / WN2+0.0209-0.0196 to +0.065427623 / 10134–132–10
EPS / HRES+0.0296+0.0041 to +0.059527623 / 10151–115–10
EPS / ICON+0.0216-0.0111 to +0.057827623 / 10136–134–6
WN2 / HRES+0.0087-0.0289 to +0.048527623 / 10148–120–8
WN2 / ICON+0.0007-0.0395 to +0.044627623 / 10139–127–10
HRES / ICON-0.0081-0.0212 to +0.005827623 / 10118–151–7
ICON / FORECAST HORIZON

Skill as the day approaches

00.250.50.751D10.1113 · n=276D30.0929 · n=252D50.1478 · n=228BRIER SCORE ↓FORECAST DEADLINE

Each horizon has its own common cohort. Use the paired update analysis below to measure improvement on identical events.

HORIZON DATA

Scores and sample sizes

LEADBRIER ↓95% INTERVALNDATES
D10.1113+0.0936 to +0.130727623
D30.0929+0.0766 to +0.112425221
D50.1478Insufficient date blocks22819

Forecasts use the frozen Gaussian approximation in the current season. There are no native ensemble or climatology skill claims in these results.

FIXED CHECKPOINTS / D5 → D1

Useful updates, costly revisions

MODELBRIER IMPROVEMENT ↑95% INTERVALMEAN CHANGEIMPROVED / WORSE / SAMECOMMON CASES
AIFS+0.0341Insufficient date blocks-0.6 pp119 / 44 / 5168
EPS+0.0149Insufficient date blocks+2.8 pp114 / 47 / 7168
WN2+0.0391Insufficient date blocks-1.1 pp132 / 34 / 2168
HRES+0.0474Insufficient date blocks-0.8 pp127 / 40 / 1168
ICON+0.0187Insufficient date blocks-14.1 pp96 / 69 / 3168

Improvement = old Brier − new Brier; positive means the later forecast was better. Both checkpoints must contain all models, identical contract terms, and the same finalized observation. This diagnostic never adds a bonus to accuracy standings or changes a position.

Inspect paired checkpoint receipts · latest 30 cases
STATION / DAYOLD → NEWIMPROVEMENTRESULTRECEIPTS
KLGA · 2026100415.1% → 6.7%+0.0183NOD5 / D1
KMIA · 2026100466.6% → 39.9%-0.2495YESD5 / D1
KMSP · 2026100456.8% → 37.8%-0.2009YESD5 / D1
KORD · 2026100472.1% → 64.4%-0.0490YESD5 / D1
KSEA · 2026100493.9% → 94.1%+0.0002YESD5 / D1
KSLC · 2026100499.3% → 100.0%+0.0000YESD5 / D1
KATL · 2026100516.3% → 64.4%-0.3878NOD5 / D1
KBOS · 2026100577.0% → 88.8%+0.0403YESD5 / D1
KDEN · 2026100599.2% → 100.0%+0.0001YESD5 / D1
KDFW · 2026100553.4% → 14.1%-0.5210YESD5 / D1
KIAH · 2026100537.2% → 57.9%+0.2174YESD5 / D1
KLAX · 20261005100.0% → 100.0%+0.0000YESD5 / D1
KLGA · 2026100564.8% → 59.0%-0.0442YESD5 / D1
KMIA · 2026100564.1% → 43.2%-0.1932YESD5 / D1
KMSP · 2026100515.1% → 75.2%+0.6595YESD5 / D1
KORD · 2026100534.0% → 18.2%-0.2328YESD5 / D1
KSEA · 2026100596.0% → 82.6%-0.0288YESD5 / D1
KSLC · 2026100599.8% → 100.0%+0.0000YESD5 / D1
KATL · 2026100666.9% → 44.4%-0.1994YESD5 / D1
KBOS · 2026100622.9% → 6.7%+0.0480NOD5 / D1
KDEN · 2026100699.0% → 100.0%+0.0001YESD5 / D1
KDFW · 2026100654.6% → 20.6%-0.4252YESD5 / D1
KIAH · 2026100676.2% → 76.1%-0.0008YESD5 / D1
KLAX · 20261006100.0% → 100.0%+0.0000YESD5 / D1
KLGA · 2026100619.9% → 2.5%+0.0391NOD5 / D1
KMIA · 2026100644.8% → 34.6%-0.1232YESD5 / D1
KMSP · 2026100677.1% → 99.7%+0.0524YESD5 / D1
KORD · 2026100672.5% → 83.2%+0.0477YESD5 / D1
KSEA · 2026100681.3% → 61.2%-0.1159YESD5 / D1
KSLC · 2026100697.0% → 100.0%+0.0009YESD5 / D1
DEADLINES REACHED / SELECTED LEAD AND REGION

Who showed up?

STATIONDUEAIFSEPSWN2HRESICONMISSEDAWAITING WORKER
KATL2828 / 2828 / 2824 / 2828 / 2827 / 2800
KBOS2828 / 2828 / 2824 / 2828 / 2827 / 2800
KDEN2828 / 2828 / 2824 / 2828 / 2827 / 2800
KDFW2828 / 2828 / 2824 / 2828 / 2827 / 2800
KIAH2828 / 2828 / 2824 / 2828 / 2827 / 2800
KLAX2828 / 2828 / 2824 / 2828 / 2827 / 2800
KLGA2828 / 2828 / 2824 / 2828 / 2827 / 2800
KMIA2828 / 2828 / 2824 / 2828 / 2827 / 2800
KMSP2828 / 2828 / 2824 / 2828 / 2827 / 2800
KORD2828 / 2828 / 2824 / 2828 / 2827 / 2800
KSEA2828 / 2828 / 2824 / 2828 / 2827 / 2800
KSLC2828 / 2828 / 2824 / 2828 / 2827 / 2800

Each cell is locked model entries / published contests whose deadline has passed. Missed execution windows stay in the denominator. Future contests do not. Missing entries remain unavailable rather than becoming a probability or an artificial loss.

SEASON-WIDE / TRIAL EVIDENCE

How much do we know?

FINALIZED TARGET DAYS28 / 14

28 dates have a complete model cohort

NEXT CUTOFF INTAKENo pending cutoff

All published cutoffs processed

FORECAST RECEIPTS54851

Last new capture: Oct 06 · 17:45 UTC

0 missed contests · 0 due contests awaiting the worker · 3 observation corrections recorded.

The 14-day target measures accumulated evidence, not approval to promote the trial. Model-version identity and calibration provenance still need qualification. Intake is a current eligibility check, not a locked entry or proof that a background worker is running. Repeated unchanged forecasts do not create new capture timestamps.

SOURCE PROVENANCE / CORROBORATING EVIDENCE

The publication behind the forecasts

858 verified publications · 539,375 forecast-to-publication links. Each link records when the evidence was observed; it never replaces the forecast’s original receipt time.

Oct 04 · 15:04 UTC · Verified Oct 04 · 15:12 UTC · 137 links
Publication and calibration evidence ↗ · Source menu · Frozen publication

Oct 04 · 14:52 UTC · Verified Oct 04 · 14:56 UTC · 126 links
Publication and calibration evidence ↗ · Source menu · Frozen publication

Oct 04 · 13:52 UTC · Verified Oct 04 · 14:08 UTC · 127 links
Publication and calibration evidence ↗ · Source menu · Frozen publication

The full menu hash, station forecast values, source cycles, publication time, and frozen projection must agree. Calibration artifact fingerprints are supporting evidence; model-release versions remain unknown and this season’s policy remains shadow-v1.

All publication evidence (JSON) ↗

How to read the uncertainty

Intervals use 1,000 deterministic bootstrap resamples of fixed three-calendar-day blocks, keeping every station and each paired comparison together. Bounds appear only after at least 14 distinct target dates and eight occupied blocks. This accounts for some shared weather and short-term persistence, not every source of dependence.

Intervals are exploratory, unadjusted for multiple comparisons, and can be unstable in a short season. They describe this cohort and its missing-data selection; they do not prove universal model superiority. No synthetic or reconstructed results are mixed into this desk.

Calibration interpretation follows the Met Office reliability guide. See the WMO verification guidance for paired block-bootstrap comparisons.

Download these research data (JSON) ↗

Journal sequence 1989 · 24f5e6442c9a9823160931c40991aa100ae9facd1d7f2807b5b0018b24b9024e