Hello,

I see on the economist site that there's a model output file. It looks like it's refreshed daily.
https://cdn.economistdatateam.com/us-2020-forecast/data/president/economist_model_output.zip

I'm wondering if there's a historical archive of the state top line results
state_averages_and_predictions_topline.csv only contains the model output from the most recent date.

Thank you!

Really grateful you guys had shared this. Such an audacious attempt to forecast something so uncertain into the future.

Just to confirm, after reading the "How it works" document explaining the methodology I am trying to locate the corresponding code that generated the beta prior for the popular vote, but could find any.

Maybe it's down to my limited understanding of stan, but I really didn't see any elastic net code either.

Bests,
Paul

Hi, I keep receiving the following error on line 250 and am unsure on how to resolve it as I cannot locate the "historical_prior_simulations.rds" file in the data folder.

Error:
In gzfile(file, "rb") :
cannot open compressed file 'data/historical_prior_simulations.rds', probable reason 'No such file or directory'

Thank you!

Would it be easy for the team to add the estimates and data (polls etc.) from the Senate and House models? Or maybe that should be in a separate repo, if you decide to publish.

I am running final_2008.R script (in RStudio on a mac). I believe the model is using Rstan as opposed to cmdstanr - therefore I comment out lines 561 to 571 and un-comment 555-559.

When I run the model I get the following error:

Error in mod$fit_ptr() : 
  Exception: variable does not exist; processing stage=data initialization; variable name=poll_mode_state; base type=int  (in 'modela29f20816b55_poll_model_2020' at line 14)

failed to create the sampler; sampling not done  

If I go to line 519-522 where these variables have been commented out and change it I get the following error:

Error in eval(ei, envir) : object 'poll_mode_state' not found

I am clearly not using the model correctly. Can someone help? Apologies, I am sure this is a basic issue

Hello, thanks for making this available. Is there a .csv with the current year polls in it? I see only 2016 back.

I think you're sorting by the 2-letter state abbreviation instead of the state name. Seems like a bad practice when you display the state name but sort by the abbreviation.

Would it be possible to provide a file with the model output from the 2008-2016 backtests? I know I could just run the files myself but that would be time-consuming. It would be helpful if people could see the backtesting results to better compare this model to others.

#15

Hi,

Just wanted to double check that the polls in 2008 are prior to the election or do they cover the post-election period as well?

There are some polls that fall in December 2008. I just want to make sure this isn't a typo.

An example is below:

-- | CNN | 1019 | 12/15/08 | 12/17/08 | 43 | 47 | 7 | 3 | Live Phone | Registered Voters

EDIT: I just double checked RCP (https://www.realclearpolitics.com/epolls/2008/president/us/general_election_mccain_vs_obama-225.html#polls), looks like some 2008 polls were conducted in 2007, so in your data some of the national polls have the wrong year.

Thanks,
Zoe

I love the fact that this repo is being shared and The Economist is taking an open source approach to this. BRAVO!!!

That said, publicizing the code / data is one thing, making it ready for the community to digest and improve is another thing. So before I post issues, I'd love if you could clarify a few questions of mine:

1. What is the goal / purpose of this being open sourced?
2. What is the goal / purpose for this codebase?

Answering my questions would help me understand the pros/cons of this. For example, if the goal is to accurately predict an outcome given all available data, this model should include more recent data, no? Furthermore, if the goal of this to be open sourced is to get community involvement, this codebase could use some better structure for people to separate "standard code" from "domain specific" code to help focus efforts.

For example, please correct me if I'm wrong, but after diving in a little, it looks like the data feeding this model is all 2016 and before? (This would mean there is no way to correct for specific new candidates that have never influenced the data before).

I'd love any feedback you have on this as the importance of this codebase and the influence of The Economist warrant this being taken seriously.

Towards transparency! Thank you The Economist!

I am really thankful that The Economist has brought onboard Andrew Gelman to model the probability of the likely outcome to the 2020 election. I am struck by the Stan code and its the similarity between Simon Jackman's work on election prediction, namely his state-space model for the 2004 Australian federal election published in the Australian Journal of Political Science and more poignantly for the American case, his modelling efforts for the 2012 Presidential election. Linzer may have published the model implemented in the Journal of the American Statistical Association in 2013 but Jackman implemented it in 2012 for the election and got 50 out of 50 states. Linzer's model as far as I can tell is Jackman's 2006 article with the hierarchical structure in his textbook. The intellectual origins are Jackman's; he published in 2006 the state-space model and spelt out in two chapters of his textbooks state-space modelling of vote intention and exchangeability for state-level predictions with hierarchical models. Linzer was visiting Stanford in 2012-2013, yet only completed his PhD at UCLA in 2008 and had mostly published on latent class analysis until 2012; I wonder how much cross-fertilisation there was. Sure Linzer got the publication while visiting (Jackman at) Stanford, but it seems to me like there was a little healthy competition at Stanford in 2012 worth acknowledging.

From my first inspection of the model, perhaps with the exception of the Cholesky factor decomposition that the Bayesian probabilistic language, Stan, has made far easier to implement over the preceding modelling language, JAGS, the model Gelman has used is basically the same as Jackman's modelling efforts and can be found in chapters 8 and 9 of his textbook, Bayesian Analysis for the Social Sciences (2009). I understand it's hard to improve on something simple and excellent ... but Gelman has his own textbooks - most notably Bayesian Data Analysis (Third Edition; 2013) - with lots of nifty tools and tricks of the trade that I would have loved to have seen expressed in this modelling effort. Gelman is the king of priors, I am pretty sure I'm not alone in wishing he had more artistic/scientific licence to perform something a bit more novel, incorporating for example estimates from Multilevel Regression and Postratification analyses he has worked hard to popularise and made some very cool and important findings about American public opinion (in particular, partisan non-response bias), or employed a cool ass Gaussian process or spline to "exploit a sensitivity–stability trade-off" as "they stabilize estimates and predictions by making fitted models less sensitive to certain details of the data" in the hierarchical model component of the model doing a little shrinking or even have the priors be a little more creative, say with a horseshoe prior to handling the sparsity. Or even, it would have been awesome to see an ensemble of model specifications that average over the models and weighting by their performance but all of this has a cost.
I'm willing to accept that the model had to be kept simple since it is being updated daily and incorporating components that explore high dimensional space while theoretically cool to a statistician or political scientist would blow out the time it takes to analyse the data. The practicality of implementation and the questionable value add of a slightly more accurate model makes the trade-off seem appropriate.

At the time of this writing, the website reports that Biden has a 91% chance to win the presidency and Democrats a 76% chance to win the senate. Scrolling down and hovering over the histogram of senate simulations, the website reports the Democrats to have 74% chance of winning at least 51 seats. When hovering over the gray bar just to the right, the text is "There is a 10% chance the senate will be split evenly, with an 18% chance a Democrat breaks the tie."

I assume that last phrase refers to the vice president breaking ties in the senate. But then given Biden's lead, surely it should be the Democrats who have the advantage to win senate ties? My guess is that someone inverted that probability and the correct number should be 82% in favor of the Democrats, and therefore the Democrats should have a 74% + 10%*82% ≈ 82% chance to win the senate (instead of the reported 76%).

Or am I misunderstanding something?

Running the final_2016.R code, stan throws warnings:

Warning messages:
1: There were 1 divergent transitions after warmup. Increasing adapt_delta above 0.8 may help. See
http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup 
2: Examine the pairs() plot to diagnose sampling problems
 
3: The largest R-hat is NA, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat 
4: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess 
5: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess 

This is with n_iter <- 2500, 4 chains, and rstan::stan_version() tells me I'm running version 2.21.0 (I'm using R 4.0.2 on Windows 10, if that's germane; I'd be happy to supply any other versioning or hardware info).

If I run it with n_iter <- 1000 on 4 chains (as it is by default), I get the last three error messages (largest R-hat is NA, Bulk ESS too low, and Tail ESS too low).

I'd give you the resulting .rds file for the output, but it's several gigs in size.

Hi there.

I noticed today that the downloadable sheet you have of current polling data on the economist website has incorrect values for the Biden vote share on the latest RMG poll. That poll has a 46% vote share for Biden, not the 48% in your data. I don't know how much that will adjust your model - I assume mostly a wiggle since it's one poll, but wanted to alert you to the error.

If this is a deliberate adjustment to weight a partisan poll or something, ignore this issue!

The CSV extract of polls on https://projects.economist.com/us-2020-forecast/president contains malformed date values in start.date and end.date (specifically the year).

library(tidyverse)

polls <- read_csv("data/2020 US presidential election polls - all_polls.csv")

polls %>% 
  filter(start.date == "5/17/2002" | end.date %in% c("7/14/20220", "5/19/0220")) %>% 
  select(state, pollster, start.date, end.date)

# A tibble: 3 x 4
  state pollster      start.date end.date  
  <chr> <chr>         <chr>      <chr>     
1 FL    St Pete Polls 7/13/2020  7/14/20220
2 NY    Sinea College 5/17/2002  5/21/2020 
3 CA    SurveyUSA     5/18/2020  5/19/0220 

I'd like to report an easy-to-make but consequential bug. It seems the bug prevents use of all state poll data during fitting
of stan models to 2008, 2012, and 2016. I'm unsure of other effects.

Like many bugs in R scripts, this one stems from read.csv having the default option stringsAsFactors = TRUE.

Bug locations

All entries of df$index_s are reassigned value NA at

• Lines 137-8 of final_2008.R
• Lines 136-7 of final_2012.R
• Lines 135-6 of final_2016.R

Bug appearance

All instances look like
index_s = as.numeric(factor(as.character(state), levels = c('--',state_abb_list)))

Reason for bug

Because read.csv defaults to stringsAsFactors=TRUE, state_abb_list has class factor, not the anticipated class character. As a
result c('--', state_abb_list) returns c('--', '1', '2', '3', ...), not the anticipated c('--', 'AK', 'AL', 'AR', 'AZ',...).

A small additional point: It looks like states are misordered in the comment directly above the bug in the R scripts.

Hello again,

I'm a little confused about what's going on here in the find_sigma2_value function. Specifically this line

y <- MASS::mvrnorm(100000, rep(0.5,10), Sigma = cov_matrix(10, par^2, 1) )

why is the correlation set to 1 in cov_matrix? This results in the 10 columns of y being identical, and hence aggregations like apply(y, MARGIN = 2, mean) result in a constant vector, which seems to make the subsequent call to mean redundant. (EDIT - actually while we're on the subject, what's the significance of the choice of 10 here?)

I notice a few lines down when state_correlation_error is created a correlation of 0.9 is used, and similarly for state_correlation_mu_b_walk and state_correlation_mu_b_T. Is there a reason the variance is estimated using a different correlation than is used to create the covariance matrices themselves?

A related question, why is the mean of y set to 0.5? Applying inv.logit to standard normal samples would result in transformed samples with mean 0.5, but on the logit scale does mean 0 make more sense? (I don't think I understand what's going on here properly yet, so I might be wrong, the question is basically just motivated by symmetry 🙂)

As of today the model on the economist.com website estimates the win probability of candidate biden / president trump at 96 % / 4 % respectively. At the same time the graph on the estimation of delegates shows that the 95 % confidence interval for both candidates goes across the line of 270 Delegates. From my understanding of statistics and the US Election system this seems to conflict:

... to the Economist for sharing and to the authors.

Let's pretend this is a enhancement issue:

what about adding some interactive code to switch States?

Really stylish!

R

I would love to be able to download a file with the results of the simulations. I am curious to see how states are correlated with each other in your model. Thank You. I love your model. It's the best, most open model in existence. I hope more data journalists follow your lead.

I'm trying to run the scripts/model/poll_model_2020.stan. After compiling it with cmdstan, I keep getting the error:

$ ./poll_model_2020  method=sample  num_samples=1000 num_warmup=1000
method = sample (Default)
  sample
    num_samples = 1000 (Default)
    num_warmup = 1000 (Default)
    save_warmup = 0 (Default)
    thin = 1 (Default)
    adapt
      engaged = 1 (Default)
      gamma = 0.050000000000000003 (Default)
      delta = 0.80000000000000004 (Default)
      kappa = 0.75 (Default)
      t0 = 10 (Default)
      init_buffer = 75 (Default)
      term_buffer = 50 (Default)
      window = 25 (Default)
    algorithm = hmc (Default)
      hmc
        engine = nuts (Default)
          nuts
            max_depth = 10 (Default)
        metric = diag_e (Default)
        metric_file =  (Default)
        stepsize = 1 (Default)
        stepsize_jitter = 0 (Default)
    num_chains = 1 (Default)
id = 1 (Default)
data
  file =  (Default)
init = 2 (Default)
random
  seed = 4125478090 (Default)
output
  file = output.csv (Default)
  diagnostic_file =  (Default)
  refresh = 100 (Default)
  sig_figs = -1 (Default)
  profile_file = profile.csv (Default)
num_threads = 1 (Default)

Exception: variable does not exist; processing stage=data initialization; variable name=N_national_polls; base type=int (in '../us-potus-model/scripts/model/poll_model_2020.stan', line 2, column 2 to column 23)

My sense is that there is probably some data file that I'd have to feed into this, but, being a complete novice with stan, I'm not sure where to find it. If I use

$ ./poll_model_2020  method=sample  num_samples=1000 num_warmup=1000 data file=../../data/all_polls.csv

I still get the same error.

Any thoughts? Coming from https://statmodeling.stat.columbia.edu/2023/06/14/the-economist-is-hiring-a-political-data-scientist-to-do-election-modeling/, where Andrew mentioned that getting this model to run would be "trivial".

I'm not quite sure what model you use to adjust for pollster and mode effects. But I noticed that the population variable in your data set has the following levels:
"a", "rv", "lv", "rv ", "lv ".
Meaning there are some polls where the population had an extra space. You can't see this but it appears when you run:
levels(data$population). Where data is the data frame of the polling data.
Here are the affected polls:
state pollster sponsor start.date end.date entry.date.time..et. number.of.observations population
374 -- YouGov Economist 2020-08-30 2020-09-01 9/2/2020 8:00:00 1207 rv
448 NH Saint Anselm College 2020-08-15 2020-08-17 8/20/2020 19:21:00 1042 rv
mode biden trump biden_margin other undecided
374 Online 51 40 11 2 4
448 Online 51 43 8 NA NA
url include notes
374 https://today.yougov.com TRUE 63 days
448 https://htv-prod-media.s3.amazonaws.com/files/saintanselmpollaugust20-1597956431.pdf TRUE

state                     pollster sponsor start.date   end.date entry.date.time..et. number.of.observations

662 -- Redfield & Wilton Strategies 2020-07-09 2020-07-09 7/13/2020 14:23:00 1853
population mode biden trump biden_margin other undecided
662 lv Online 48 40 8 3 10
url include notes
662 https://redfieldandwiltonstrategies.com/latest-usa-voting-intention-july-9/ TRUE

Hello,

Is the use of poll_mode_* variables for the poll_pop_* parameters here intentional or is it a typo?

EDIT - On a related note, it looks like there is a typo (or at least redundancy) in the definition of poll_pop_state which uses index_p rather than index_pop and hence is identical to poll_state, defined a few lines earlier.

It looks like all references to the mu_b_T_model_estimation_error parameter are commented out in the poll_model_2020.stan model as well as poll_model_2020_no_mode_adjustment.stan. Presumably this variable should be removed from the parameters declaration. I'm not sure if this is the current Stan behavior, but there are at least some cases from past versions where unused parameters can negatively impact model behavior through unintended impacts on the posterior.

I'm not super confident I'm fully understanding the logic which is going into state_data / state_cov / state_correlation, however it seems like the lines:

state_data <- read.csv("data/potus_results_76_16.csv") state_data <- state_data %>% select(year, state, dem) %>% group_by(state) %>% mutate(dem = dem ) %>% #mutate(dem = dem - lag(dem)) %>% select(state,variable=year,value=dem) %>% ungroup() %>% na.omit() %>% filter(variable == 2016)

(lines 99-107) in final_2016.R

Mean we use the 2016 election results to create these correlations. I would assume we'd want to use the 2012 results as our prior?

Related to this, is there a file for recreating the 2020 results? Or should I modify final_2016 for 2020?

This is great. Are there files missing though?

e.g. out <- read_rds(sprintf('models/backtest_2008/stan_model_%s.rds',RUN_DATE))

cannot open compressed file 'models/backtest_2008/stan_model_2008-11-03.rds', probable reason 'No such file or directory'Error in gzfile(file, "rb") : cannot open the connection