Build a Labeled Forecasting Dataset from Real-World News in 10 Minutes

Foresight Data helps AI teams create model-ready training data at scale. Our pipeline transforms unstructured sources into questions and labels, no human labeling required. You can bring your own data or start with our integrated sources, such as Google News.

In this walkthrough, we’ll show you how to use the Foresight Data pipeline to generate a labeled forecasting dataset directly from the news in under 10 minutes.

The pipeline separates question generation from outcome resolution to prevent data leakage and ensure questions remain objective:

This pipeline is the data foundation of our Foresight Training methodology, which uses outcome-based supervision to train LLMs to make better predictions. You can learn more here.

In about 10 minutes, you’ll generate:

Log into the Lightning Rod dashboard and select Dataset Generation from the left navigation menu. This opens the Dataset Generation screen, where you can configure the full pipeline.

The first step is to decide what topics you want to generate questions about.

For this walkthrough, we’ll use Input: News Search.

News Search takes a search query and retrieves matching news articles. Questions will later be generated from these articles.

In the search queries field input: election.

Leave source filters, article count, and intervals at their default values.

Tip: It can be helpful to test your query in Google News first. If the results match the kinds of events you want to generate questions from, you’re on the right track. If not, adjust the terms to better reflect your topic.

Next, define the start and end dates. The start and end dates control when questions are evaluated, not which articles are included.

For this example, select a start date of January 1, 2025 and an end date of December 31, 2025. This will generate questions whose outcomes resolve in this time window.

Now we define what kinds of questions the system should generate. For this walkthrough, we’ll use:

Forward-looking questions are future-dated questions with a close data and resolution criteria (e.g. Will Microsoft announce an AI agent that can autonomously complete multi-step coding tasks by March 1, 2026?).

Tip: Increasing the Questions Per Document value identifies multiple distinct prediction questions within a single source article, resulting in more diverse training data

Now we’ll define the instructions the model will follow when generating questions. In the instructions field input:

Your question section should now look like this:

Example questions strongly influence what the system generates.

In practice, models often follow examples more closely than written instructions. Even small changes here can significantly affect dataset quality.

To edit the example questions:

Use examples like these:

When context is enabled, the system reconstructs what was known at the time each question was asked by summarizing relevant articles published before the decision into a compact context block.

For this walkthrough, we’ll leave Context off.

Because context generation is the most compute-heavy and expensive part of the pipeline, we recommend refining your questions and labels first before adding the context layer.

The next step is to enable Labels. Select the Labels toggle in the dataset configuration to turn outcome labeling on.

When Labels are enabled, a separate resolver model uses web search to find up-to-date information and determine the outcome for each question. It then produces a source-grounded label for every question.

The confidence threshold controls how certain the system must be before including a resolved outcome. Higher thresholds filter out ambiguous cases where post-event reporting is unclear.

For this walkthrough, leave the confidence threshold at its default value.

You can run the pipeline as either:

The Max questions setting caps the total number of questions across all retrieved articles.

As you change settings like max questions or labels, the cost estimate updates automatically.

For this walkthrough, start with a Test Run.

Once the run starts, a progress indicator will appear while the job runs.

Once the run completes, navigate to Datasets from the left menu.

From the dataset preview, you can:

The dataset preview also shows basic metadata, including the number of rows generated and the total cost.

Now that you’ve generated the dataset, you can plug it into your workflow to train supervised forecasting models or use it as an evaluation set to benchmark LLM prediction performance.

Or, you can refine your Project Settings to explore new use cases: