Benjamin Turtel, Paul Wilczewski, and Kris Skotheim (Lightning Rod Labs), Ville A. Satopää (INSEAD), and Philip E. Tetlock (University of Pennsylvania)

Reinforcement learning is a powerful way to shape how language models behave during post-training. The choice of reward function influences not just how well a model performs on a benchmark, but what it learns to do well.

We explored this in LLM forecasting, where models must turn complex, unstructured information into probability estimates about future events. In our new paper from Lightning Rod Labs, we post-train five versions of GPT-OSS-120B with GRPO, changing only the proper scoring rule used as the training reward. We then use bias–information–noise (BIN) analysis to examine how each reward changed the model's forecasting behavior.

The results reveal an important distinction. Several models achieved nearly identical aggregate forecasting scores, but their improvements came from different sources. Brier-score training produced greater gains in distinguishing likely events from unlikely ones, while log-score training produced larger reductions in noise.

Figure 1: BIN decomposition of forecasting improvements across five reward functions, relative to the base model. Positive values indicate improvements in Brier score; negative values indicate deterioration.

Similar scores, different behaviors. The Brier- and log-trained models achieved nearly identical held-out Brier scores (0.1648 versus 0.1653), but BIN analysis revealed nearly three times as much estimated information gain from Brier training and substantially greater noise reduction from log training.

To understand why, we need to look at how different scoring rules shape training incentives—and how BIN separates the different sources of forecasting improvement.

Proper scoring rules as training objectives

A strictly proper scoring rule rewards a forecaster, in expectation, for reporting its true probability estimate. Different rules share this property but assign different rewards to individual forecasts. Two common examples are the Brier score and the log score.

The Brier score squares the forecast error, so twice the error means four times the penalty:

Brier score = (p − y)²

where p is the predicted probability and y is the binary outcome. Lower Brier scores are better, so during reinforcement learning we use the negative Brier score as the reward.

The log score is:

Log score = y log(p) + (1 − y) log(1 − p)

Higher scores are better. Unlike Brier, the log score penalizes extremely confident mistakes very sharply.

How scoring rules change training incentives

The choice of scoring rule changes which forecast improvements receive the strongest rewards during training.

Figure 2 shows this for two pairs of forecasts, assuming the event resolves Yes: a toss-up (38% versus 62%) and a confident miss (1.8% versus 4.7%). Each pair represents the same-sized step in log-odds, but the reward gaps differ.

Under Brier reward, the toss-up produces a much larger reward gap (0.24 versus 0.06). Under log reward, the pattern reverses: correcting the confident miss produces the larger gap (0.97 versus 0.50).

The bottom panels show why: Brier's sensitivity peaks around 50%, while log reward is most sensitive to confidently wrong forecasts.

Both rules reward better forecasts, but Brier emphasizes improvements near the middle of the probability range, while log reward emphasizes correcting confident mistakes. This shapes which forecast improvements GRPO prioritizes during training.

Figure 2: Reward when the event happens under Brier (left) and log (right) scoring, against forecast probability on a log-odds-spaced axis. Markers show two equal-sized steps in log-odds; the bottom row shows each reward’s derivative with respect to log-odds.

Separating bias, information, and noise

How can models with similar scores be different kinds of forecasters?

Consider three hypothetical forecasters: the Agnostic, the Optimist, and the Erratic. The Agnostic assigns every event a 50% probability. The Optimist distinguishes the outcomes, but assigns probabilities that are too high on average. The Erratic is right on average, tending toward 60% on events that happen and 40% on events that don’t, but each forecast jumps 30 percentage points up or down at random. On the four events below, all three receive the same Brier score of 0.25.

The same score hides different problems. Under some conditions, bias and noise can be reduced without changing what a forecaster knows. Information cannot be added this way.

The Optimist’s bias negated its advantage in distinguishing outcomes. If the size of the bias can be estimated on target events drawn from the same distribution as the ones being forecast, a constant downward adjustment can improve its score. Lowering all of the Optimist’s forecasts by 30 percentage points changes them to 60% on the Yes outcomes and 40% on the No outcomes. Its Brier score improves from 0.25 to 0.16, a 36% reduction in error, without changing the ranking. Correcting the bias makes the useful information already present in its forecasts visible in the score.

The Erratic has no constant offset to remove: its forecasts average 50%, the base rate. Its problem is noise, variation that doesn’t track the outcome, which sometimes puts 70% on an event that doesn’t happen. If the errors are independent across attempts, averaging several forecasts of the same event cancels them, and the expected score approaches the same 0.16 (about 0.21 with two forecasts, 0.18 with four). For a more detailed treatment of how aggregation affects bias, information, and noise, see Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model by Satopää et al.

Neither adjustment helps the Agnostic. Shifting its forecasts cannot create a distinction among events, because it still assigns every question the same probability, and averaging identical forecasts changes nothing. Calibration and averaging can only reveal information already present in a forecaster’s outputs. They cannot add any.

Figure 3: Three hypothetical forecasters on four events. Left: their probabilities for the Yes and No events. Right: each Brier score as a baseline minus information, plus bias and noise, before and after the fix.

The three forecasters are therefore held back by different things: the Optimist’s score by bias, the Erratic’s by noise, and the Agnostic’s by a lack of information. The bias–information–noise decomposition, or BIN, makes this separation formal, splitting differences in average Brier score into three components:

  • Bias captures whether forecasts systematically run too high or too low, as with the Optimist.

  • Information captures the strength of the outcome-relevant signal in the forecasts, which the Agnostic lacks entirely.

  • Noise captures the remaining variance that does not track the outcome, as with the Erratic.

We could see each forecaster’s problem by inspection above, but BIN is estimated across a set of forecasts rather than from any single prediction. It tells us whether a model’s score improved because its forecasts contain more outcome-relevant information, or because they became less biased and less noisy.

Training the same model with five rewards

In our experiment, we use our platform to generate 8,041 binary forecasting questions from resolved news events across politics, geopolitics, economics, business, science, and sports. Each asks whether an event will happen within the next 7 to 90 days. We use 7,076 questions for training and hold out 965 for evaluation. Of these, 27.0% of training questions and 28.5% of evaluation questions resolve Yes.

Each example follows Lightning Rod’s Future-as-Label method. The pipeline separates question generation, resolution, and forecasting context so that the model only sees information that would have been available at the prediction date.

Specifically, we take a document, such as a news article, and use its publication date as the prediction date. We generate a question from that document about what will happen next. Lightning Rod builds context using only information available before that date. During training, the model sees only the question and that context, and produces a probability forecast. We use later reporting to determine what actually happened.

This gives us objective labels at scale while restricting the supplied forecasting context to information available at the prediction date.

Examples include:

Will the Trump administration officially announce a suspension or termination of the 50% tariffs on imports from India before March 1, 2026?

Will André Ventura win the Portuguese presidential runoff election held on February 8, 2026?

Will the U.S. Consumer Price Index for All Urban Consumers (CPI-U) show a year-over-year increase of at least 4.0% in the Bureau of Labor Statistics’ March 2026 report?

We post-train GPT-OSS-120B with Dr. GRPO and rank-32 LoRA adapters. We hold the model, data, and training configuration fixed while varying only the scoring rule, so that we can isolate how different rewards shape forecasting behavior.

For each question, the model produces eight reasoning traces, each ending in a probability forecast. The scoring rule assigns a reward to each forecast, and Dr. GRPO reinforces traces based on their relative rewards within the group.

We compare five strictly proper scoring rules, which differ in where they apply the strongest training pressure:

  • Log score gives the largest penalties to confident mistakes, such as assigning a tiny probability to an event that happens.

  • Brier score uses squared error, so confident misses cost more, but not as much as log.

  • Spherical score gives more weight than Brier to differences near 50%.

  • Beta(2,8) emphasizes differences between small probabilities, like 1% and 5%.

  • Beta(8,8) concentrates its weight tightly around 50%, where the model is most uncertain.

Figure 4: Rewards when the event occurs (y = 1), rescaled to −1 to 0. Differences in reward shape the relative training advantages under Dr. GRPO.

Over 200 training steps, differences in reward shape which reasoning traces receive the strongest reinforcement and, consequently, which forecasting behaviors the model learns.

Similar scores, different forecasting profiles

To compare the trained forecasters, we sample five forecasts for each held-out question, giving 4,825 forecast–outcome pairs. Alongside log and Brier scores, we report expected calibration error (ECE), the gap between stated probabilities and observed frequencies, and AUC-ROC, how often an event that happens gets a higher probability than one that doesn't. Every trained model has better observed scores than the base model on the four metrics.

Reward

Log score ↑

Brier ↓

ECE ↓

AUC-ROC ↑

Log

−0.5132

0.1653

0.0434

0.7407

Brier

−0.5185

0.1648

0.0568

0.7511

Spherical

−0.5273

0.1680

0.0538

0.7353

Beta(2,8)

−0.5223

0.1677

0.0449

0.7372

Beta(8,8)

−0.5536

0.1731

0.0954

0.7366

Base

−0.5585

0.1861

0.1099

0.7248

Table 1: Held-out forecasting performance. Arrows show whether higher or lower is better.

The Brier-trained model has the best Brier score and AUC-ROC. The log-trained model has the best log score and lowest ECE. But the aggregate differences among log, Brier, spherical, and Beta(2,8) are small: their Brier scores range from 0.1648 to 0.1680, less than 2% apart, and within the variation we see across training seeds.

The aggregate metrics tell us how well each model forecasts, but not how it improved. BIN reveals what those metrics conceal: different reward functions produce different patterns of improvement in bias, information, and noise, even when overall forecasting performance is nearly identical.

Reward

Bias

Information

Noise

Log

6.06

1.29

2.60

Brier

5.75

3.70

0.44

Spherical

6.35

2.16

−0.38

Beta(2,8)

6.63

1.90

0.36

Beta(8,8)

3.73

3.68

−1.57

Table 2: BIN contributions relative to the base model (%). Positive values improve Brier performance; negative values offset it. BIN contributions are estimated from fitted scores, so their sum may differ from the observed Brier-score improvement.

The clearest contrast is between Brier and log. Their observed Brier scores differ by just 0.0005, yet the Brier-trained model shows nearly three times as much estimated information gain: 3.70% versus 1.29%. The log-trained model instead shows much larger estimated noise reduction: 2.60% versus 0.44%. One interpretation is that log training produces a forecaster less prone to extreme errors. Because log reward heavily penalizes confident misses, the model has a stronger incentive to avoid them. A three-seed replication preserved this qualitative distinction: Brier training showed greater estimated information gains, while log training showed larger noise reductions. The beta rewards produce different profiles again. Beta(2,8) shows the largest estimated reduction in bias, while Beta(8,8) nearly matches Brier on information but adds noise.

Why information matters

We use an LLM for forecasting because we want it to extract signal from rich, unstructured evidence—news, reports, explanations, and details that interact with one another.

In a BIN analysis this signal is the information component of the decomposition. For many use cases it is the property you care about most. More information helps a forecaster distinguish likely events from unlikely ones, which matters for prioritization, screening, and classification. Ranking also depends on noise, which BIN estimates separately.

Bias and noise can often be mitigated after training. Calibration can correct systematic shifts in probability without changing the ranking, and ensembling repeated forecasts can reduce noise. Bias and noise still matter, especially when decisions depend on absolute probabilities or on a single forecast, but neither recalibrating nor averaging forecasts can recover a signal that is absent from the forecasts.

Forecasting leaderboards rank models on aggregate scores. These scores summarize performance across many forecasts, but do not separate the contribution of information from bias and noise, so they can't tell us how much information a model is contributing. If a model can't distinguish likely events from unlikely ones, a much cheaper model, or even the base rate, can come close to its aggregate score.

The right reward depends on how forecasts will be used. For applications that prioritize distinguishing likely events from unlikely ones, information-rich forecasts may be especially valuable. Where decisions depend directly on probability estimates, bias and noise also matter. More broadly, our results show that reward design shapes which capabilities improve during post-training—and that evaluating those improvements separately can reveal differences that aggregate metrics miss.

Read the full paper

The full paper is How Proper Scoring Rules Shape LLM Forecasting, by Benjamin Turtel, Paul Wilczewski, Kris Skotheim, Ville A. Satopää, and Philip E. Tetlock.

@article{turtel2026scoringrules,
  title   = {How Proper Scoring Rules Shape LLM Forecasting},
  author  = {Turtel, Benjamin and Wilczewski, Paul and Skotheim, Kris and Satop{\"a}{\"a}, Ville A. and Tetlock, Philip E.},
  journal = {arXiv preprint arXiv:2608.28482},
  year    = {2026}
}