How many nickels can fit, in dollar value, in a 1.5 liter (1.58QT) Evian water bottle?

When Professor Jack Soll of Duke University’s Fuqua School of Business asks this question in class, the responses range from $20 to $2,000, with many far from the correct answer ($69).

But something “amazing” happens if you take the “crowd average,” Soll said.

“If you average everyone's individual guesses, they average out to $70,” he said in a talk on Fuqua’s LinkedIn page. 

“The crowd only misses by just $1, even though individuals are wildly inaccurate.” 

Perhaps someone in the crowd guessed correctly, he said, but it is difficult to know in advance who that person will be.

The experiment seems to suggest that in many circumstances, relying on the “wisdom of crowds” has benefits over trusting individual judgement, because the errors individual people make will likely cancel each other out, Soll said. 

But there are caveats, he added. “It doesn’t always work.”

Five takeaways about the wisdom of crowds

1. The benefit of the crowd starts working with just two people

   In the Evian experiment, a randomly selected individual’s average error is $45 (either $45 above or below the correct answer of $69), Soll said. But if you just add a second person, you will get the highest increase in accuracy compared with adding even more people (adding a second person reduces the average error from $45 to $35; adding a third brings it down from $35 to $30; and so on).

   “There are diminishing marginal benefits as individuals are added to the crowd,” Soll said. “The accuracy gain from one to two people is the same as the gain from 10 to 50 people.”

2. Wisdom of crowds depends on cognitive diversity

   The effectiveness of the wisdom of the crowd relies on different opinions “bracketing” the truth: some are significantly higher, some are lower than the correct answer, and their average is close to the truth.

   However, “bracketing” is only possible if the people providing opinions have different perspectives, approach their task differently, using different information and different training, Soll said. 

   “With diversity, you tend to get more accurate estimates,” he said.

3. Embrace disagreement

   In ongoing research by Soll, his Fuqua colleague Rick Larrick and Fuqua PhD student Bruce Y. Mei, the researchers asked the participants to imagine owning a retail company. 

   Participants were randomly assigned to four consultants who would help them predict the value of the company next year. Some were assigned to a diversified group of consultants (finance, consumer insights, operations, investor networks), some to a homogenous group (say, four experts in consumer insights).

   Some of the participants only observed the expertise of the consultants, some only saw their predictions, and some saw both. 

   The researchers then asked participants to rate the extent to which they would rely on the consultants’ predictions.

   The study found that those who only saw the consultants’ expertise gave higher ratings to diverse groups of consultants. However, those who only saw the predictions (the diverse groups, as expected, provided a wider range of answers on the company’s value in a year) responded that they were more willing to trust consultants with similar answers. 

   “And this is something of a problem to solve,” Soll said. “On the one hand, people tend to say, ‘I want them to have different perspectives, because cognitive diversity is valuable. However, they fail to anticipate its consequences, which is that when you bring in people who have diverse backgrounds, they tend to disagree with one another.”

   “But you really need disagreement to get the benefit of the wisdom of crowds. Disagreement is the engine of the wisdom of crowds,” he said.

4. Don’t “anchor” your crowd

   In a published paper Soll co-authored with Larrick and Fuqua Ph.D. student Jessica Reif, the researchers asked participants to estimate the height of the Chicago Tribune building. They then asked participants what their goal would be if they were to seek a second opinion — to get unbiased advice or to look smart and competent?

   Participants also chose a question from a predetermined list. Some of the questions included an estimate (an “anchor”, such as, “I think that the building is 650 feet. What do you think?”), while others didn’t include anchors.

   The experiment showed that 40% of the participants included an anchor. Specifically, three quarters of those who said their goal in asking for advice was to appear competent included their own opinion in the question. On the other hand, only a quarter of those who endorsed the goal of avoiding influence included an anchor.

   “When people include anchors, they want to look smart,” Soll said. “But there is a way to achieve both goals: if you want to look smart but also avoid influencing your advisor, you could say, ‘I thought about this and I have an answer in mind, but I want to hear what you have to say.’”

5. Consider a “select crowd”

   Although averaging the opinion of a crowd is often a better strategy than relying on an individual opinion, there is another option that may be more effective in certain settings: the “select crowd.”

   In the paper, “The wisdom of select crowds,” Soll, Larrick and Albert E. Manness found that when you can rank the opinions based on how they performed in the past, the best strategy would be to only average the “top judges” opinions.

   “This is really meant for cases where you might have 30, 40, maybe 100 people you could potentially ask,” Soll said.

   In the paper, the researchers studied data from a Philadelphia Fed survey, which collects forecasts of economic indicators by professional economists. The survey collects predictions about inflation, national Gross Domestic Product (GDP), unemployment rate and more, going back to 1968.

   The researchers compared forecast accuracy of random economists versus the average forecast of the crowd and found that “you are always better off with a larger crowd,” Soll said. 

   However, they also introduced a ranking of the best performers based on the accuracy of their forecasts in the previous 5 quarters. As a result, they found that if you only consider the forecast of the “10 best judges,” their accuracy is higher than that of the crowd. If you keep adding forecasters, Soll said, the accuracy starts going down.

   “You don't need a large crowd, necessarily,” he said. “If you know which people have done well in the past, or you have some information about how good of a forecaster they are, you could do better than just averaging the whole crowd.”

Embrace disagreement within organizations

The benefit of crowd’s wisdom doesn’t only apply to averaging quantitative opinions, Soll said, but also to collaborative work within organizations.

Teams won’t benefit from the wisdom of the crowd if we don’t embrace diversity of opinions, Soll said.

“Knowledge and information is distributed across individuals, and to the extent that we discourage disagreement, we will prevent us from fully capitalizing on all wisdom,” he said

Oftentimes, the message that comes from the organizational leadership is that certain norms are ingrained in the culture and that the organization has a fixed way of operating, Soll said.

But managers should always make people feel comfortable sharing their unique perspectives and their knowledge, he said, even at the cost of uncomfortable disagreement.

“We're often afraid of stepping on people's toes and arguing with people. But it turns out disagreements are essential and critical, if you want to get the most of unique perspectives and reap the benefits of the wisdom of crowds,” Soll said.