Above the Law has a post today, which provides an excellent illustration of the pitfalls to relying upon markets (and "crowdsourcing") to answer specific questions:
American Needle, a significant antitrust case, asked whether or not the NFL and its member teams counted as one entity for the purposes of merchandise licensing. The court, in a unanimous decision, reversed the 7th Circuit’s ruling that they were one entity and exempt from the Sherman Antitrust Act. Only 37% of our members predicted the reversal, and out of approximately 200 predictions, 24 got the split correct also. The SMRs for the justices indicate there was going to be a consensus, even if the direction was wrong. One easy explanation is that the majority of users thought that baseball’s antitrust exemptions would be precedential instead of a characteristic peculiar to America’s Pastime. Even in that consideration however, there is not much of an opportunity for partisan division in this opinion, although our statistics do show that there was no particular support of monopolies among the different justices and ideologies.
I don't know any expert in antitrust law who expected the Supreme Court to uphold the Seventh Circuit in American Needle. None. FantasyScotus is not an efficient market.
So what about all those studies that show that a crowd's estimate of the number of jellybeans in a jar is better than almost any individuals?
Those studies are examples of what mathematicians call the Central Limit Theorem in action. Many startup companies that proposed to change the world through their crowdsourcing of solutions would do well to understand this theorem more deeply. Frankly, most wouldn't exist if their founders did.
Here's a non-mathematician's summary of the Central Limit Theorem: A series of measurements of the same thing, made independent of each other, will be normally distributed (i.e., gaussian, or bell-shaped) with error of the mean inversely proportional to the square-root of the number of measurements.
In the case of the jelly beans, it's easy to see how the CLT applies. The way the experiment is normally conducted, each student looks at the same jar of jelly beans, makes an estimate, and writes his or her answer down on a piece of paper, which is then tallied up along with everybody else's. The measurements (i.e., the individual estimates) are identically distributed because basically everybody's view of the jar is the same, and everybody's eyes and visual processing work basically the same. The measurements are independent so long as none of the students talk to each other before giving an answer.
Independence is particularly important to achieving a good result in the jelly bean experiment. Without it, the answers become correlated, potentially deviating far from the actual number of jelly beans. This could happen, for example, if one member of the group were persuasive to others, but wrong.
(In fact, something very much like this happened inside the financial industry, when analysts had to price difficult-to-value mortgage back securities and credit derivatives. Some influential analysts started using VaR, ratings agencies priced super-senior tranches at AAA, and then everybody followed suit. As a result, the market prices were tightly correlated such that the whole market collapsed when a single set of assets dropped in price. Regulators would do well to think about how regulatory systems could ensure statistical independence in valuations without overburdening market actors with transactions costs.)
Thus, the problems for FantasyScotus could have resulted from at least two different causes. First, the individuals who participated may not have been taking measurements correctly or of the same thing. (For example, they may not have known anything about antitrust law, or may not have read any of the relevant opinions.) Second, even if some individual did take a reasonable (but off the mark) measurement, other market actors may have adopted that measurement as the correct measurement rather than making their own measurements independently.
Could these problems be overcome? Suppose the market were constituted by experts, each of whom was capable of making a correct measurement. And suppose these experts were not permitted to talk to each before betting on the outcome.
On my view, this is an impossible experiment. Almost by definition, experts arrive at their opinions through a process of communicating and understanding other expert opinions, which eliminates the possibility of (at least complete) statistical independence as a practical matter.
In fact, when it comes to legal opinions, this is probably a good thing for society. Society benefits greatly from the fact that different people from different backgrounds are more likely than not to interpret the law in the same (or at least similar way) when faced with a particular, never-before-decided set of facts.
Consensus is a public good in other fields too, although the details are interesting and complex. Mathematicians might be understood as the most rigid. But mathematical formalism is simultaneously the most abstract, and thus the most susceptible to multiple interpretations -- a fact that physicists make good use of. Conversely, literary criticism seems to lack any definite form whatsoever. But at least to this outside observer, the field of scholars active in this domain are more insular and elitist than even economists. Perhaps we all need stability and consistency in our work, and that stability comes from social networks if it can't be found within our individual neural networks.
Recent Comments