Reading the tea leaves: The interesting practice of predicting Supreme Court opinions

Fivethirtyeight.com has recently taken a look at the various methods of predicting Supreme Court opinions. Because there are no polls to consult and Supreme Court opinions are complex, making a prediction about how the Court will rule is much more complicated than simply calling an election. There are currently several different models for making these predictions, including complex computer algorithms, high-level statistical analysis, a crowd-sourced prediction machine called FantasySCOTUS and plain-old human predictions by people with varying degrees of legal knowledge.

Most of the methods the article references have had a success rate that hovers somewhere between 70-80%. Statistical analysis using a "classification tree" method, which pulled data from ten years of prior cases had about a 75% success rate for the 2002-2003 SCOTUS term, while a poll of expert law professors was about 59% accurate for that same time period. These predictions were compiled and analyzed in a 2004 study, undertaken by academics in the legal and political science fields.

Josh Blackman, a law professor at South Texas College of Law has participated in the creation of two Supreme Court predictor models. Blackman, along with two colleagues, created a computer model called {Marshall}+ that makes predictions based on over 60 years of precedent data. This model has been about 70% accurate. Blackman is also the force behind FantasySCOTUS.com, which is a website designed in the vein of fantasy sports. Players choose which justices will decide which way in certain cases and earn points for their accuracy, providing incentive for them to get it right (especially this year, when Thomson Reuters has put up a $10,000 prize). This model has also had about a 70% accuracy rate among serious players.

Interestingly, one of the more successful predictors is Jacob Berlove, a Cincinnati native currently residing in Queens, NY, who has no formal legal training. Berlove became interested in the SCOTUS while reading the Cincinnati Enquirer as a child. He has studied past Supreme Court decisions and closely follows oral arguments, analyzing both the style and content of the justices questions and comments to make his predictions, and has had about an 80% success rate in predicting the outcomes of cases. Berlove has also won the FantasySCOTUS three years in a row.

Fivethirtyeight.com offers a breakdown of predictions from various sources for both the October and November sessions of the Court. Although many of the sources have similar levels of demonstrated accuracy, there is not necessarily much agreement about which cases will turn which way, at least for the October session.