The Forecaster Experience

This page walks through what taking part in a network looks like, from the first invitation to a scored track record.

Joining

A new forecaster arrives either from an invitation email or by visiting the network's domain directly. Either way they land in a guided onboarding flow that runs before and after they create their account.

The first few screens explain what is being asked of them. The flow covers:

  • The brief - what a forecast request is: a specific question about the world, a number between 0 and 100, a few minutes a couple of times a month.
  • Vocabulary - the bridge from the probability words analysts already use ("likely", "highly unlikely") to the numeric ranges they correspond to.
  • Creating an account - by password or single sign-on, depending on what the network offers.
  • Practice - a real forecasting interface on a practice question, including a rationale. Nothing is recorded. You can set your own practice question so it feels relevant to your network.
  • The network - why many independent estimates, aggregated, usually beat any single forecaster.
  • Calibration - what being well calibrated means, and that being uncertain when you should be uncertain is the goal.
  • Profile - picking a handle and writing a short bio.
  • Expertise - rating themselves against the network's topics, and answering any profile questions you have set.
  • Frequency - how often they want to hear from the network by email.

The flow takes around five minutes, saves progress as they go, and resumes where they left off if they drop out partway.

Home: Requests for You

A forecaster's home page is a personal queue: the questions currently waiting on their input, each framed as a request from whoever asked it. A summary line at the top tells them how many sponsors are waiting and how many requests close within the week.

A question sits in the queue until they forecast it in the current round. Once they respond it drops off - the questions list is there for browsing everything else. When a recurring question opens a new round, it returns to the queue even though they answered last time. They can sort the queue by newest request or by what closes soonest.

Alongside the queue sits a Resolving soon rail showing what is about to close and whether they have forecast it, plus their running stats and accuracy standing.

The Question Page

The question page is organized as a progression - understand, take the outside view, take the inside view, forecast:

  • Current state of play - an AI-generated situation report on what is happening now and what has changed recently, drawn from the question's source material.
  • Base rate brief - the outside view. How often things like this have happened historically, expressed as a count of comparable cases, with caveats about how good the comparison is.
  • Key sources - the evidence behind the brief, including counter-evidence, so a forecaster can check the reasoning rather than take it on trust.
  • Ask Hinsley - a research chat that answers questions about this specific question and its sources. Can be turned off per network in General settings.
  • Official clarifications - anything an admin has posted to resolve ambiguity in the wording.

The Crowd Veil

The current crowd forecast is hidden until a forecaster takes a position of their own, so that the consensus does not anchor their estimate. Once they have submitted, they can reveal the crowd and see where they sit relative to it, with a prompt to think about what they know that the crowd does not - or the reverse.

Submitting a Forecast

Forecasters set a probability - by slider, by typing a number, or by picking from the same probability words the vocabulary step introduced - and write a rationale explaining their reasoning. They can update a forecast any time while the question is open.

On a return visit the page shows what has changed since their last forecast: how far the crowd has moved, new sources gathered, whether the state of play or base rate brief has been refreshed.

Discussion

Forecasters can comment on each other's forecasts, so the reasoning is discussed rather than just the number. They can react to comments and rationales, and @-mention other members of the network to pull them into a thread. Mentions and replies generate notifications and, unless the recipient has opted out, an email.

Anyone can flag a comment or rationale they think is a problem, which sends it to the network's moderation queue.

Accuracy and Standing

When a question resolves, everyone who forecast it is scored and notified. Scores are relative: a forecaster is measured against a baseline, so the score reflects whether they beat the crowd rather than whether the question happened to be easy.

My accuracy shows a forecaster their record: their cumulative margin over the baseline plotted over time, their win rate on resolved questions, how many questions they have been scored on, and where they rank. Individual question scores are listed so they can see which calls went well.

The leaderboard ranks everyone on the network and can be filtered by topic, by time period, and by forecaster type - so a network can see how its people compare to AI forecasters on the same questions. It can be switched off entirely per network in General settings.

Voided questions never count towards anyone's accuracy, and forecasters are told so explicitly when a question they answered is voided.

Email Preferences

Every forecaster controls their own email, per network - someone on two networks can be set up differently on each.

  • Request cadence - whether new requests arrive as they are published, or batched into a weekly, fortnightly, or monthly digest. This is set during onboarding and changeable any time.
  • Per-notification opt-outs - comments on their forecasts, mentions, clarifications, scoring notices, voided questions, and new requests can each be turned off individually.
  • A master switch - turns off every email from that network at once.

Only request emails batch. Notifications in the app and the home queue are always immediate, regardless of the cadence chosen, so someone on a monthly digest still sees everything the moment they visit.

Admins Viewing the Network

Account admins can open their network and look around without holding a seat on it. They see the questions and the site as forecasters do, but cannot submit forecasts, and their home page shows a note explaining that requests and standing appear once they join as a forecaster. To take part as well as administer, they need a seat - either added from the roster, or granted automatically by the analyst access setting.