Crowdsourced Forecasting

Background

A growing body of research shows that aggregating forecasts from diverse individuals can dramatically improve accuracy and reduce bias. A landmark example is the Aggregative Contingent Estimation (ACE) Program, sponsored by the Intelligence Advanced Research Projects Activity (IARPA). The Good Judgment Project, led by Philip Tetlock and Barbara Mellers, decisively outperformed both other research teams and the U.S. intelligence community's own benchmarks. Cultivate Labs, the team behind Hinsley, built and ran the software platforms that underpinned much of this research.

Crowd forecasting works because it harnesses a wide range of perspectives, experiences, and information sources. When managed effectively, this diversity leads to more reliable predictions, as errors and biases tend to cancel each other out.

Why Crowdsourced Forecasting Matters

Forecasting fills the gap between qualitative understanding and quantitative decision-making. Key findings from the research literature include:

• Aggregating independent judgments reduces bias and increases accuracy
• Crowd forecasting is effective across domains, from geopolitics to operations management and competitive intelligence
• Larger, more diverse groups tend to produce better forecasts, provided the process is well-managed
• Expressing judgments as probabilities eliminates the ambiguity of words like "likely" or "possible" — everyone sees the same number, making it easier to compare views, track changes over time, and hold forecasts accountable

The Forecasting Process

1. Select a forecast question - After running a decomposition and scenarios, create forecast questions based on your analysis.
2. Invite a diverse set of forecasters - Hinsley allows you to collect forecasts from colleagues both inside and outside your organization, ensuring diverse perspectives. You can also add Hinsley's AI Forecasting to the mix, combining human and AI judgment for even stronger forecasts.
3. Collect consensus forecasts and rationales - Review rationales to identify contrarian perspectives that often surface overlooked risks.
4. Iterate and update - Forecasters can submit updated forecasts as new information emerges.

Tips for Making Good Forecasts

Hinsley is especially useful when forecasting is a team exercise rather than a solo one. Analysts can invite internal stakeholders, collect outside judgment when needed, and compare how different rationales move the aggregate over time.

• Start with the outside view: Look to the base rate that a particular outcome occurs in similar situations.
• Adjust with the inside view: Evaluate what makes this particular circumstance unique.
• Seek contradictory information: Read opinions that take the opposing perspective.
• Conduct a pre-mortem: Ask yourself what factors might contribute to an unexpected result.