We built a calibrated weather model to beat Kalshi's temperature markets. The price was better at every hour.
Four months of forecasts, 731 settled days, a model fit the way forecasters fit them, a pass mark written first. The book beat it at all twelve decision times. The whole fail, in the open.
Weather is the one Kalshi market where someone has shown a real record with a model. We had the other half of the recipe sitting on disk: four months of the public weather ensembles, the National Weather Service forecast, Kalshi's own prices and every settlement, for six cities. So we built the model the textbooks say you build, wrote the pass mark first, and tested it against the book.
The short version. Our calibrated model was better than the raw forecasts and worse than Kalshi's price at every hour we checked. The pre-registered rule said: if the model cannot match the market on the validation month, do not even open the holdout. It could not, so we did not. This is a fail, and here is the whole of it.
What we tried before, and why this was different
In July we traded the raw forecast ensembles against the book on paper: wherever the models disagreed with the price, buy. It lost at every size of disagreement, and lost more the bigger the disagreement. We closed it. The book already knows what the public models say.
The thing we had not done is what forecasters call post-processing: take the raw ensembles, learn each station's biases from months of actual settlements, and widen or narrow the forecast to match how wrong it has really been. That is standard meteorology, it is what a trader with a real record would do, and by November we had enough settled days to do it.
What we built
Six cities (Chicago, New York, Austin, Miami, Denver, Philadelphia), 731 settled days, July 16 to November 17. For each day: 82 forecast members (ECMWF and the American GEFS) pulled every six hours, the NWS point forecast, and a snapshot of Kalshi's bucket prices at twelve decision times from the afternoon before to the afternoon of. The model fits each city's bias and spread on the settlements themselves, the exact outcomes the book is graded on. We fit on July and August and scored September. October and November were sealed as the holdout, to be opened only if September looked at least even.
Three independent reviewers went through the code and the plan before any score was trusted, and found no leakage and no scoring error. They did find six design flaws that would have decided the result for the wrong reasons, and we fixed all six before re-running.
The result
Brier score, which is how you grade probabilities. Lower is better. Two-sided quotes only.
| Decision time | Kalshi price | Raw forecast | Our model |
|---|---|---|---|
| Afternoon before (3 PM ET) | 0.115 | 0.162 | 0.132 |
| Morning of (9 AM ET) | 0.136 | 0.195 | 0.154 |
| Afternoon of (3 PM ET) | 0.131 | 0.255 | 0.210 |
Calibration closed about a third of the gap between the raw forecasts and the price. The price was still ahead at all twelve decision times, and in five of six cities (Miami was a tie). On the buckets that actually trade, between 5 and 95 cents, the gap was the same.
The trading simulation is the part that would fool you. Buying wherever the model disagreed with the price by more than five cents after the fee made about a cent a contract across 974 simulated trades. The honest confidence interval on that cent runs from minus two to plus four. The second half of the month lost. A model that is worse than the price can still show a small positive over 180 days of clustered bets, and the reviewers found that the first version's whole profit came from 57 trades in one price band. Noise.
Why the price wins
The book gets better through the day. Our forecasts, pulled four times a day from public models, do not. By the afternoon of the target day the market is scoring far ahead of anything we can build from the morning's run. The simplest reading is that the people quoting these markets are watching the station's live observations and the sky, and a public ensemble, however well calibrated, is not news to them.
What this does not say
It does not say weather on Kalshi cannot be beaten. The record that exists was built on observations and station knowledge, not on calibrating public models. It does say that the specific idea most people would try first, calibrate the free forecasts and trade the disagreement, is refuted on four months of our own tape, after a hostile review, under a bar written before we looked.
The holdout months stay sealed. If we try again it will be a new written test with different inputs: live hourly observations decided inside the day, or a model that uses the price itself as a starting point and asks only whether the forecast adds anything on top. That is a smaller claim, and probably the right one.
None of this is advice. It is the record.