AI Forecast Variants

An AI forecast variant forecasts a question from a fixed set of evidence sources that you choose. Each variant has its own label, its own research, and its own AI forecasts, which run next to Hinsley's forecasts on the same question. Compare variants on the same questions to see how each set of sources changes the forecasts.

A variant with no sources is a baseline. Its AI models forecast from their training data only, with no research, no search tools, and no documents.

Forecasters never see variant forecasts. Variant forecasts do not change Hinsley's forecast, the crowd forecast, the question's forecast counts, or any leaderboard.

Before You Start

Only account administrators can create variants and turn them on for questions. If you do not see AI Forecast Variants in account administration, the feature has not been enabled for your account. Contact your Hinsley support representative to have it turned on.

Create a Variant

  1. Click your avatar in the top-right corner and find the Account Admin section.
  2. Click AI Forecast Variants, then New variant.
  3. Enter a Name. This label identifies the variant on question pages and in exports.
  4. Under Evidence sources, select the account evidence sources the variant may use. These are the sources listed under Evidence Sources in account administration, including any connected Google Drive repository. Leave the list empty for a baseline variant.
  5. Select On for new questions if new questions should start with this variant turned on.
  6. Click Save.

The form also shows the aggregation method that combines the variant's model forecasts into one variant forecast. It is Hinsley's current aggregation method.

What Locks After the First Run

The first time the variant runs on any question, it keeps its evidence sources, their settings, and the aggregation method. Every later run on every question uses that same configuration, so one label means one configuration across your questions. Saving the variant or turning it on for a question does not lock it. After the first run:

  • The variant's sources can no longer be changed. Create a new variant to try a different set.
  • Changes to a source's settings, such as its included domains, do not reach the variant.
  • Changes to Hinsley's aggregation method do not reach the variant.

You can still rename a locked variant and change On for new questions.

Archive a Variant

To stop a variant, open the actions menu on its row and click Archive. An archived variant stops running on every question and keeps its forecasts. Click the link to show archived variants, below the list, to find it again, and click Restore to start it again.

Turn Variants On for a Question

  1. Open the question's form, on the analyst site or on your forecaster network. This can be a new question, a draft, or an existing question.
  2. Under AI Forecast Variants, turn each variant on or off.
  3. Save the question.

Only account administrators see the variant switches. On a new question, the switches start with every variant that has On for new questions selected. A question that another user creates starts with those variants. Questions that Hinsley creates for you, and questions created through the API, start with no variants.

Turning a variant off stops its runs on that question. Its earlier forecasts stay on the question.

When Variants Run

Each turned-on variant runs on the question's AI forecast schedule, which is set on the same form. Variants run on that schedule even when Generate Hinsley AI forecasts is off for the question. Each run researches the question with the variant's sources, then each of the account's AI models forecasts from what that research found.

Each variant run counts as one AI forecast against your account's monthly AI forecast limit, the same as a Hinsley run. Usage in account administration shows how much of the limit you have used. A question with four variants and Hinsley's own AI forecasts uses five forecasts each time it runs. Variant runs do not count against any user's personal monthly AI forecast limit.

If one of a variant's sources becomes unavailable, for example when a Google Drive connection loses access, the variant's runs fail until the source is available again. The variant keeps its previous forecast. AI Forecast Variants in account administration warns you about variants with an unavailable source. A repository that stopped syncing but kept its documents stays available, so the variant keeps reading the documents it already has.

See the Results

Account administrators see a Variants tab on the question page. On a forecaster network, the tab shows to account administrators who are also network administrators. For each variant, the tab shows:

  • The variant's latest forecast, and the latest forecast of each AI model in it.
  • The evidence the variant gathered for this question, with its relevance.
  • Why the last run failed, if it did, including any unavailable sources.

Export Variant Forecasts

In account administration, open Reports and create a Questions & forecasts report. It covers your account's questions that are not in an analysis. Select AI forecast variants and AI forecast variant models under Forecasters to include variant forecasts. Each variant row carries:

  • The variant's name.
  • The AI models behind the forecast.
  • The prompt version the models were given. When the prompt changes, the version changes, so you can separate forecasts made under different prompts.

The Excel file of that report has a second sheet, Samples, with every individual sample each model produced.