models-markets
Models and Markets
Election prediction helps party officials, campaign operatives, and journalists interpret campaigns in a quantitative manner. Uncertainty is key to a useful election prediction.
The forecast model has become a staple of political punditry. Popularized by the data journalist at FiveThirtyEight, the forecasting model is a statistical tool used to incorporate a number of quantitative inputs and produce a probabilistic view of all possible outcomes.
Prediction markets can be used to generate similarly probabilistic views of election outcomes by utilizing the economic forces of price discovery and risk aversion to overcome the ideological bias of self-interested traders on a binary options exchange.
Can markets predict elections better than the models? I propose a null hypothesis of no difference in the mean Brier score of forecasting models and prediction markets for competitive races in the 2018 U.S. Congressional midterm elections.
Reproduce
All public input data has been saved on the internet archive and can be accessed through their wayback machine.
Data manipulation is done using the R language and packages from the tidyverse ecosystem.
The R scripts in the /code directory can be run in sequential
order to reproduce the results.
Data
Forecasting Models
I will be using the FiveThirtyEight “classic” model to represent the best capabilities of statistical election forecasting. FiveThirtyEight has a track record of accuracy over the last decade.
As Nate Silver explains, most forecasting models (1) “take lots of polls, perform various types of adjustments to them, and then blend them with other kinds of empirically useful indicators to forecast each race”. Importantly, they (2) “account for the uncertainty in the forecast and simulate the election thousands of times” to generate a probabilistic forecast.
The classic model incorporates three types of inputs, primarily direct and imputed polling as well as fundamentals factors like incumbency and the generic ballot.
FiveThirtyEight publishes two files with daily top-level predictions:
Together, there are 110,404 daily “classic” model prediction from 470 races with 13 variables.
Prediction Markets
Prediction markets generate probabilistic forecasts by crowd-sourcing the collection of data from self-interested and risk averse traders. The efficient market hypothesis holds that asset prices reflect all available information (including forecasting models).
PredictIt is an exchange run by Victoria University of Wellington, New Zealand. The site offers a continuous double-auction exchange, where traders buy and sell shares of futures contracts tied to election outcomes. As a trader’s perception of probabilities changes, they can sell those shares. The market equilibrium price updates accordingly to reflect current probability. As outcomes become more likely, prices rise as demand for shares increases.
PredictIt provided the price history in
DailyMarketData.csv. Together, there
are 44,711 daily market prices from 118 races with 11 variables.
Compare
The FiveThirtyEight model and PredictIt markets data sets were joined using the date and a unique race code. The data was then pivoted to a long format, which allows us to compare each method against the ultimate binary results of the race.
| Date | Race | Method | Probability | Dem. Favorite? | Won? | Correct? | Brier score |
|---|---|---|---|---|---|---|---|
| 2018-08-01 | AZ-S1 | market | 0.660 | TRUE | TRUE | TRUE | 0.116 |
| 2018-08-01 | AZ-S1 | model | 0.738 | TRUE | TRUE | TRUE | 0.069 |
| 2018-08-01 | CA-12 | market | 0.910 | TRUE | TRUE | TRUE | 0.008 |
| 2018-08-01 | CA-12 | model | 1.000 | TRUE | TRUE | TRUE | 0.000 |
| 2018-08-01 | CA-22 | market | 0.300 | FALSE | FALSE | TRUE | 0.090 |
| 2018-08-01 | CA-22 | model | 0.049 | FALSE | FALSE | TRUE | 0.002 |
| 2018-08-01 | CA-25 | market | 0.610 | TRUE | TRUE | TRUE | 0.152 |
| 2018-08-01 | CA-25 | model | 0.745 | TRUE | TRUE | TRUE | 0.065 |
| 2018-08-01 | CA-39 | market | 0.610 | TRUE | TRUE | TRUE | 0.152 |
| 2018-08-01 | CA-39 | model | 0.377 | FALSE | TRUE | FALSE | 0.388 |
Here we can see how each each race was predicted by each method, highlighted by the race results.
A probabilistic prediction should find that events with a 60% probability occur 60% of the time. Here we see how many of each method’s predictions occurred that frequently. Predictions with a 60% probability that occurred 85% of the time are underconfident and vice versa.
The Brier score allows for probabilistic forecasts to be meaningfully tested with mean squared error. The Brier score rewards skillful predictions, with a 100% probability earning a score of zero if correct. Using this test, there is no statistically significant difference in the respective skill scores of each predictive method.
| Test statistic | df | P value | Alternative hypothesis |
|---|---|---|---|
| -0.34 | 16942 | 0.7338 | two.sided |
Welch Two Sample t-test: brier by method (continued below)
| mean in group market | mean in group model |
|---|---|
| 0.1084 | 0.1091 |
