Crowdsourced Forecasting

In complex organizational environments, decision-makers face two linked challenges: understanding the factors that will shape future outcomes, and quantifying the uncertainty around them. Decomposition and scenario analysis give you the structure to understand unfolding dynamics and what signals drive them. Crowdsourced forecasting gives you a way to measure and assign probabilities to those signals, turning fuzzy intuition into clear, actionable foresight.

Background

A growing body of research shows that aggregating forecasts from diverse individuals can dramatically improve accuracy and reduce bias. A landmark example is the Aggregative Contingent Estimation (ACE) Program, sponsored by the Intelligence Advanced Research Projects Activity (IARPA). In ACE's multi-year forecasting tournament, teams of researchers and thousands of participants competed to predict global events ranging from geopolitical shifts to leadership changes. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, decisively outperformed both other research teams and the U.S. intelligence community's own benchmarks, demonstrating that well-structured crowd forecasting can deliver more accurate and timely intelligence than traditional expert analysis.

Crowd forecasting works because it harnesses a wide range of perspectives, experiences, and information sources. When managed effectively, this diversity leads to more reliable predictions, as errors and biases tend to cancel each other out.

Why Crowdsourced Forecasting Matters

Forecasting fills the gap between qualitative understanding and quantitative decision-making. It transforms the understanding of drivers and scenarios into hard probabilities, allowing organizations to focus attention and resources on the most probable futures. Key findings from the research literature include:

  • Aggregating independent judgments reduces bias and increases accuracy
  • Crowd forecasting is effective across domains, from geopolitics to operations management and public health
  • Larger, more diverse groups tend to produce better forecasts, provided the process is well-managed

Forecasting is best understood as one step in an iterative cycle. Decomposition reveals what to forecast by identifying the key drivers and signals that matter most. Scenario analysis frames the range of plausible outcomes. Forecasting assigns likelihoods to those outcomes, quantifying risk and opportunity so teams can prioritize actions and resources. As new information emerges and forecasts are updated, organizations can revisit their decomposition and scenarios, refining their understanding over time.

The Forecasting Process

  1. Select a forecast question — After running a decomposition and scenarios, create forecast questions based on your analysis. For example: "Will a major European country introduce new legislation restricting the export of advanced semiconductor technology to non-EU countries by 31 December 2026?"
  2. Invite a diverse set of forecasters — Include colleagues from different functions (legal affairs, government relations, operations, business development) to capture different perspectives and mitigate individual biases.
  3. Collect consensus forecasts and rationales — Review rationales to identify contrarian perspectives. Look for viewpoints that contradict the prevailing consensus, as these often surface overlooked risks.
  4. Iterate and update — Forecasters can submit updated forecasts as new information emerges. Establish a regular cadence for reporting forecasting results to leadership and incorporate forecasts into routine planning.

Tips for Making Good Forecasts

  • Start with the outside view: Rather than evaluating the particular circumstances of the case at hand, first look to the base rate that a particular outcome occurs in similar situations. This helps ground your forecast and prevents overconfidence.
  • Adjust with the inside view: After identifying the base rate, evaluate what you know about this particular circumstance that makes it unique and adjust your forecast accordingly.
  • Seek contradictory information: Play devil's advocate by reading articles or opinions that take the opposing perspective. This helps counter confirmation bias and overconfidence.
  • Conduct a pre-mortem: Before settling on a final forecast, ask yourself: if my forecast turns out to be wrong, what factors might have contributed to that unexpected result?

Crowdsourced Forecasting in Hinsley

Step 1: Preparation

Enter your strategic question into Hinsley and execute a decomposition to identify key drivers and trackable indicators. This foundational step helps you scope and prioritize your forecasting approach.

Step 2: Question Selection

Navigate to the Crowdsource Forecasts tab where the system suggests forecast questions derived from your decomposition. You can select a suggested question or create your own.

Step 3: Invitation & Data Collection

Enter email addresses of forecasters you wish to invite. Participants receive email invitations and can submit probabilistic forecasts without requiring a Hinsley account. They retain the ability to update their forecasts by resubmitting at the provided link.

Step 4: Review & Analysis

View consensus probability in the Crowd Forecast tab once submissions arrive. Individual forecasts and their supporting rationales appear below the aggregate result.

Since forecasters can continuously update submissions over time, teams gain real-time insights for decision-making by periodically prompting participants to refresh their estimates.

References

  1. Philip Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (Crown, 2016).
  2. Achal Bassamboo et al., "The Wisdom of Crowds in Operations: Forecasting Using Prediction Markets" (Harvard Business School, 2019).
  3. Sarah Scoles, "Psychology of Intelligence Analysis" (Popular Science, 2023).
  4. Kim Armstrong, "Crowding Out Falsehoods" (Association for Psychological Science, 2024).