Automated Oracle: How Forecasting Agents Approach Expert-Level Prediction
A survey of architectures, benchmarks, and what cheap calibrated estimates make possible
In 2023, the Forecasting Research Institute tested GPT-4 on a battery of prediction questions. It performed at roughly the level of an untrained crowd. Now, a little over two years later, scaffolded agent systems are approaching the top 2% of human experts.
That progression has a structural explanation. Forecasting is one of the few reasoning tasks where the world provides its own training signal. Make a prediction, wait, check. Time creates free supervision at scale.
Additionally, forecasting has a known, trainable gap between average performance and expert performance. Tetlock’s superforecasters derived their advantage largely from protocol: decompose problems, anchor on base rates, update incrementally, and treat confidence as a variable. A protocol that is specified can, in principle, be automated and trained.
Superforecaster judgment is economically costly to obtain. Automated systems approaching comparable calibration cost orders of magnitude less. At that price point, forecasting starts to look less like a standalone capability and more like a primitive, a building block for agent control, coordination, and decision-making under uncertainty.
A Brief Primer on Metrics
Before examining how these systems work, it helps to understand how forecasting performance is measured.
Brier score is the standard metric for forecasting skill:
brier_score = (1/N) Σ(pᵢ - oᵢ)²
Lower is better. A perfect oracle scores 0, while predicting 50% on everything yields 0.25. For context, superforecasters achieve ~0.096 on ForecastBench, while the general public sits around 0.121.
The elite end of the range is compressed. The gap between good and expert is 0.025. The gap between random and good is 0.129. Getting from bad to decent is comparatively easy. Getting from decent to great is really, really hard.
Brier rewards both calibration (are your 70% predictions right 70% of the time?) and resolution (can you distinguish hard cases from easy ones?). But it weights all questions equally. A forecaster who nails 1% tail risks gets less credit than one who’s marginally better at calling 50/50 coin flips.
Even given these limitations, Brier score and its variants remain the most commonly used metric in the forecasting literature.
Log score penalizes confident wrong predictions severely:
log_score = y·log(p) + (1-y)·log(1-p)
A single 99%-confident mistake can dominate the entire score. This asymmetry makes log score useful for training but problematic for evaluation on small samples. While log score is strictly proper (honest reporting maximizes expected score), its high variance means a risk-averse forecaster evaluating a small set of questions may hedge toward 50% to reduce the chance of a catastrophic realized score.
Expected Calibration Error (ECE) directly measures calibration by comparing stated confidence to empirical accuracy. But ECE ignores resolution entirely. A model predicting 50% on everything has perfect calibration and zero usefulness.
Peer scores and economic returns measure relative performance. Peer scores are zero-sum: useful for tournaments, meaningless for capability claims. Prophet Arena’s economic returns simulate betting against live markets. Interestingly, Brier and returns don’t correlate reliably. Models with mediocre accuracy showed strong returns on sports upsets, capturing edge precisely where Brier penalized them.
What you optimize for shapes what you find.
How Forecasting Systems Work
The systems approaching expert-level performance share a common architecture: reasoning, retrieval, aggregation, and calibration.
Prompting Barely Matters
Prompt engineering improves most LLM tasks. Forecasting is an exception.
Schoenegger, Jones, Tetlock, and Mellers (2025) tested 38 distinct prompts across four frontier models on 100 binary questions: chain-of-thought, few-shot, self-consistency, base rate anchoring, superforecaster personas, and more. After correcting for multiple comparisons, no prompt significantly improved accuracy over a minimal control.
Two theoretically grounded approaches, Bayesian reasoning and propose-evaluate-select, actually reduced accuracy across all models, likely because the model produces text that looks like Bayesian updating without performing genuine probabilistic reasoning.
Similarly, Pratt et al. tested five sophisticated strategies on PaLM 2 and found the simplest prompt outperformed all alternatives (although this might be due to an implicit model bias). Mutschlechner and Jatowt found that few-shot examples consistently degraded performance across multiple model families.
Why? Forecasting may not decompose into verbalizable reasoning steps: chain-of-thought works for math because mathematical reasoning is a sequence of steps, but forecasting might be more like pattern matching across a vast space of analogies, where forcing that into a linear chain disrupts rather than aids it. Alternatively, LLMs may already apply implicit chain-of-thought reasoning, making explicit prompts redundant. Or accuracy may be dominated by information access rather than reasoning quality.
The one exception: reference-based prompts such as frequency framing and base rate anchoring showed modest benefits. These share a structure: they orient the model toward outside-view thinking without prescribing analytical procedures. “Out of 100 similar cases, how many times would this occur?” outperforms “decompose the problem, weigh evidence, and apply Bayes’ rule.”
If you’re going to engineer prompts at all, anchor rather than elaborate.
Search & Retrieval: The Actual Lever
Unlike prompt engineering, search and retrieval is the highest-leverage architectural component.
Removing retrieval from Halawi et al.’s system added a +0.020 Brier penalty. The AIA Forecaster reports a 3.6× reduction in Brier score with search enabled on a live market benchmark (from 0.361 to 0.100), where removing search produced performance worse than always predicting 50%. Chandak et al. found retrieval adds 9-18% accuracy across model families.
But retrieval for forecasting is harder than standard RAG. The temporal dimension introduces unique constraints, and naive approaches can hurt more than help.
The Basic Pipeline
The foundational architecture, established by Halawi et al. (2024), has three stages:
Query decomposition. Break forecasting questions into targeted sub-queries rather than searching the question directly. This significantly outperforms single broad searches.
Source retrieval & ranking. Query news APIs and rank results by relevance. The Metaculus Q2 benchmark found no clear winner among search providers (AskNews, Perplexity, and Exa performed comparably) which suggests the retrieval source matters less than what you do with the results.
Summarization to context. Compress retrieved articles to fit context windows. Necessary due to context window sizes but unfortunately lossy.
Going Agentic
The AIA Forecaster advances beyond static pipelines by having the LLM independently determine search strategies and iteratively refine queries based on previous results. Rather than executing a fixed retrieval plan, the agent decides what information it needs, searches, evaluates what it found, and searches again if gaps remain.
This matters for two reasons. First, different questions need different information, and fixed pipelines can’t adjust. Second, the agent can recognize when it has insufficient information and continue searching rather than forecasting from an incomplete picture.
The AIA system also adds a supervisor agent that monitors disagreements among multiple forecasters. When forecasts diverge substantially, the supervisor triggers additional targeted searches to resolve the ambiguity in a process of deliberative reconciliation rather than blind averaging.
Filtering Beats Volume
Wang et al. (2024) found unfiltered news injection degrades performance compared to no news at all. Noise, incorrect causal attributions, and token overload from irrelevant articles overwhelm the signal from relevant ones.
Their solution is a two-agent filtering architecture where a Reasoning Agent categorizes news by impact type and duration, and an Evaluation Agent reviews prediction errors against the news that informed them, iteratively refining what counts as relevant. The system learns which news types are actually predictive rather than merely related.
In short, retrieval for forecasting requires more curation than standard RAG. Information must be temporally valid, causally relevant, and compressed enough to leave room for reasoning.
The winning bot from Metaculus Q2 2025 illustrates this well. Its architecture separates outside view research (base rates, reference classes) from inside view research (case-specific evidence). Combining both produces better-calibrated forecasts than either alone.
Aggregation: The Effectiveness of Averaging
Single-run forecasts are unreliable. Even frontier models exhibit high variance across runs.
The fix is simple and old: combine multiple forecasts and let independent errors cancel. Schoenegger et al. (2024) found that an ensemble of 12 diverse LLMs achieved Brier scores statistically indistinguishable from human crowds despite several individual models performing near chance. The AIA Forecaster calls ensembling non-optional. The Metaculus Q2 winning bot used median aggregation across five model predictions.
What should you aggregate? Three levels appear in the literature, each adding value:
Multiple samples from one model. Query the same model multiple times with temperature > 0. Cheap and effective. Halawi et al. used 3 samples; the AIA system uses 10.
Multiple models. Combine predictions from different architectures. Model diversity matters. Correlated errors don’t cancel. The Metaculus winning bot combined Sonnet 3.7/4, o4-mini, and o3.
Human + AI combinations. Halawi et al. found that a 4:1 crowd-to-system weighting achieved 0.146 Brier, better than either system alone (0.179) or crowd alone (0.149). Humans and LLMs are wrong about different things, making their errors complementary.
How should you aggregate? Simple mean works. Median is more robust to outliers. Trimmed mean (removing extremes before averaging) outperformed both in Halawi et al.’s tests. The AIA system’s supervisor approach identifies why forecasts disagree and triggers additional research. Schoenegger et al. tried asking GPT-4 to intelligently update its forecast given human predictions, and it underperformed simply averaging the numbers together.
The typical pipeline runs: generate multiple forecasts → aggregate → calibrate. Calibration is fit on aggregated prediction distributions. Reduce variance first, correct bias second.
The limitation is aggregation only helps when errors are partially independent. If all models share the same blind spots (e.g. same training data, same architectural biases) averaging won’t save you. Averaging reduces variance but not shared ignorance.
Calibration: Fixing What Models Get Wrong
Raw LLM probability outputs are systematically miscalibrated. The direction varies: sometimes hedging toward the middle, sometimes overconfident, sometimes exhibiting negativity bias.
Halawi et al. (2024) found that models rarely output extreme probabilities: values below 0.1 or above 0.9 are conspicuously absent. This likely stems from RLHF training, which penalizes confident wrong answers more than cautious correct ones. The result: models that hedge when evidence is strong.
Schoenegger et al. (2024) found the opposite: overconfidence and acquiescence bias, with models predicting 57% average probability despite only 45% of events resolving positively. Pratt et al. (2024) documented negativity bias: PaLM 2 achieved 0.050 Brier on negative outcomes but a catastrophic 0.341 on positives. The model was mostly guessing “probably not” and looking good because most things don’t happen.
These contradictions (hedging vs. overconfidence, acquiescence vs. negativity) suggest that miscalibration direction depends on prompt structure and dataset composition, not on any intrinsic architectural property. What’s consistent is that some recalibration is always needed.
Granularity also matters: superforecasters express probabilities at 1% precision, while typical forecasters round to the nearest 10% (Mellers et al. (2015)). A model limited to expressing 0.5, 0.6, or 0.7 will underperform one that can express 0.53, even if both hold identical beliefs. Coarse-graining alone caps achievable calibration.
The simplest effective fix is Platt scaling according to the AIA design, a logistic regression fitted on validation data:
P_calibrated = 1 / (1 + exp(A·f + B))
This is mathematically equivalent to log-odds extremization: it stretches hedged predictions toward the poles where evidence warrants confidence. The key requirement is a validation set of resolved questions. Given that, Platt scaling is cheap and addresses the RLHF hedging problem directly. For systems without resources for training-time interventions, this is the minimum viable calibration technique.
Training: Where the Real Gains Live
Halawi et al. pioneered a self-supervised approach: generate many candidate reasonings, select those where the model beats crowd predictions, filter out overconfident examples, and fine-tune. Improvement was modest (Brier -0.007) but consistent.
Turtel et al. (2025) extended this with Direct Preference Optimization. They generated pairs of reasoning traces, ranked them by proximity to resolved outcomes, and trained models to prefer accurate traces. Result: 14B models matching GPT-4o. Fine-tuned models made more high-confidence correct predictions (33-36% with Brier < 0.05 vs 21-23% for base models).
RL with Proper Scoring Rules
Turtel et al. (2026) trained Qwen3-32B using Group Relative Policy Optimization with log score reward.
The log score is a strictly proper scoring rule, uniquely maximized when the model reports its true beliefs. Real-world event resolution provides verifiable labels at scale without human annotation. The intro’s point restated in training terms: time creates free supervision.
On a synthetic benchmark, ECE dropped from 0.173 to 0.060, a 65% improvement; on the out-of-distribution Metaculus benchmark, calibration error roughly halved. The 32B model outperformed the 235B base model despite 7× fewer parameters.
A related effort from the same group applied negative Brier score as reward with a modified GRPO that preserved large reward signals for well-calibrated predictions. That version achieved ECE of 0.054, compared to 0.093 for OpenAI’s o1 on the same benchmark.
Chandak et al. (2025) found that combining accuracy and Brier score in the reward function outperformed either alone. Their OpenForecaster-8B matched much larger proprietary models. More notably, calibration improvements generalized to out-of-distribution benchmarks: MMLU-Pro, GPQA-Diamond, SimpleQA. Forecasting-tuned models may be better calibrated across the board, including tasks well outside the prediction domain. If this result replicates, it would suggest that training a model to be honest about what it knows in one domain transfers broadly, arguably the most consequential finding in this literature, though it currently rests on a single study.
The Failure Modes
RL-based training can go wrong in documented ways. Turtel et al. catalogued several:
Probability collapse. Standard GRPO caused 39% of predictions to collapse to 0% or 100%. The reward for confidently correct dominated the penalty for confidently wrong, producing degenerate behavior. The reward landscape has sharp attractors at the extremes.
Output degeneration. Without guardrails, models exhibited language-switching, gibberish, and format violations. Required mitigations included gibberish filters, token-length limits, and early stopping.
Reward hacking. Models exploit edge cases in scoring rules. Careful reward shaping with baseline-subtracted advantages and modified GRPO with preserved reward magnitude is the fix.
The same training dynamics that improve calibration can destroy it. RL for forecasting requires careful engineering, but when done right, the gains dwarf every other intervention in the literature.
Why Training Beats Post-Hoc Correction
Platt scaling operates on a fixed model distribution. It can map 0.6 → 0.8, but only if that miscalibration is consistent across questions. Training reshapes the model’s actual beliefs. The log score reward penalizes overconfidence asymmetrically: predicting 0.99 when the outcome is 0 is devastatingly expensive, which incentivizes the model to express appropriate uncertainty during generation, not just have it corrected afterward.
The empirical gap reflects this: training-time methods (65% ECE reduction) substantially outperform post-hoc methods. But they require resolved questions, compute, and careful engineering. For deployments without those resources, Platt scaling remains the practical choice.
Benchmark Evaluation
Static benchmarks, questions gathered at a single point and held indefinitely, atrophy as resolved events leak into pretraining corpora. A benchmark created in 2023 doesn’t test the same thing in 2025. Dynamic benchmarks attempt to solve this by continually adding fresh questions with unknown outcomes. They vary substantially in design, and the choice of benchmark shapes what conclusions you can draw.
ForecastBench draws 1,000 questions from 9 sources including Metaculus, Polymarket, and Manifold. It updates nightly, using only unresolved questions. Its breadth makes it the most widely cited leaderboard, but it skews toward binary questions and overweights Security & Defense topics.
The Metaculus AI Benchmark runs quarterly tournaments where bots compete directly against Pro Forecasters on shared questions. Unlike ForecastBench, it includes non-binary and multiple-choice formats, and it measures head-to-head performance rather than aggregate accuracy. This makes it a harder, more diverse test.
MarketLiquid (from the AIA paper) offers the hardest test: 1,610 questions drawn exclusively from high-liquidity prediction markets, where consensus prices reflect real money and sustained attention rather than thin participation. More than half of Manifold questions have fewer than 12 forecasters; MarketLiquid filters these out.
Prophet Arena sources live questions from Kalshi and evaluates models on both Brier score and Average Return, a simulated metric measuring profitability when betting against market prices. These can diverge: a model that tracks market consensus scores well on Brier but earns nothing, while one that’s occasionally right when the market is most wrong can profit despite worse calibration. Model rankings shift depending on which metric is used. Uniquely, Prophet Arena holds retrieval fixed, providing identical news and market data to every model. But as the AIA results demonstrate, search quality may be the dominant factor in real-world forecasting performance.
FutureX is the largest and most diverse live benchmark, sourcing from 195 websites across politics, economics, sports, and technology. Unlike the others, it goes well beyond binary questions to include ranking tasks (predict the order of a future chart), numerical predictions (estimate a stock price), and open-ended questions with no predefined options. It evaluates 25 models including agentic systems with search tools, making it the benchmark most directly aimed at testing forecasting agents rather than base LLMs.
PROPHET is less a competing benchmark than a methodological corrective. It introduces Causal Intervened Likelihood (CIL), a statistical filter that screens out questions where the available evidence is insufficient to support an informed prediction. The premise is that some benchmark questions are inherently non-inferable, so including them rewards luck rather than skill and adds noise to evaluation. CIL hasn’t been widely adopted yet, but the underlying concern applies to every benchmark above.
The Results
ForecastBench shows a clear hierarchy: superforecasters (0.096 Brier) >> best LLM (0.122) ≈ general public (0.121). The 27% gap between experts and models is statistically significant (p<0.001). LLMs struggle disproportionately with combination questions, where the performance gap exceeds 0.054 Brier points, suggesting current architectures handle independent predictions better than dependency reasoning.
The AIA Forecaster is the first system to close this gap, achieving 0.108 Brier on ForecastBench, with the difference from superforecasters not reaching statistical significance (p=0.15). This required substantial engineering: agentic search, 10-forecast ensembles, supervisor reconciliation, and Platt scaling calibration.
Metaculus tells a different story. In Q2 2025, 96 bots competed on 348 questions alongside 10 Pro Forecasters. Pros beat the top-10 bot team decisively (p=0.00001). Every individual pro ranked higher than every bot.
The gap appears to be widening rather than closing, though this trend is confounded by the introduction of non-binary questions in Q1. The apparent widening may partly reflect a composition effect from harder question types rather than a capability trend. Bots struggle most with multiple-choice (head-to-head: -32.9) versus binary (-14.8).
On MarketLiquid, even AIA underperforms market consensus (0.126 vs 0.111 Brier). Though an ensemble of AIA + market beats either alone (0.106), providing diversifying signal even when the model underperforms on average.
One heavily-engineered system has reached parity on one benchmark under one metric. Live tournaments show experts maintaining a decisive, and possibly growing, lead. Several factors may explain the divergence:
Question diversity. Metaculus includes more non-binary formats where bots underperform severely.
Benchmark optimization. AIA’s architecture was tuned on ForecastBench-style questions; it may not generalize.
Evaluation mode. Leaderboards reward average performance; tournaments reward beating specific competitors on shared questions.
Human adaptation. Metaculus pros have access to AI tools and may be improving faster than the bots.
Research lag. The Q2 2025 tournament predates many recent techniques.
Superforecaster-level depends on where you look.
Why It’s Hard to Know
These results are also harder to interpret than they appear. Temporal leakage, models accessing information about outcomes they shouldn’t know, contaminates evaluations in ways that are difficult to detect and harder to prevent. Paleka et al. (2025) catalogued the failure modes:
Question leakage. When Dai et al. generated forecasting questions from news articles, weak classifiers achieved >80% accuracy without any retrieval. The question phrasing itself encoded the answer.
Retrieval leakage. Search “January 6” with a pre-2020 filter and you’ll surface articles about the Capitol building and election certification, all contaminated by post-2021 associations. “Wuhan” searches before COVID prominently feature the Institute of Virology. The past, as presented by modern search engines, is not actually the past.
Knowledge cutoff leakage. GPT-4o claims an October 2023 cutoff but reveals November 2023 knowledge when prompted creatively. Models know more than they admit, and we can’t reliably bound what they’ve seen.
Piggybacking. Many benchmark questions originate from prediction markets where human forecasts are public. Models may retrieve and copy existing probabilities rather than reasoning independently. Matching human performance becomes circular.
There’s a complementary problem: many questions are simply unanswerable. PROPHET estimates whether a question can actually be answered from available evidence using CIL. When supportive articles existed in the corpus, CIL-guided retrieval significantly improved Brier scores; when they didn’t, retrieval provided negligible gains. Aggregate benchmark performance conflates reasoning ability with question answerability. A model might score poorly not because it reasons badly, but because half its test set is unanswerable.
Paleka et al. recommend sensible mitigations: use older embedding models for retrieval that couldn’t have encoded future information; evaluate across multiple disjoint temporal periods; prefer live evaluation over retrospective benchmarks; report Brier, calibration, and log score simultaneously rather than cherry-picking.
What This Makes Possible
Legibility and Forecasting as Interface
The most consequential implication of the research may be legibility rather than predictive accuracy. A probability estimate that’s explicit, even if mediocre, is more useful than an intuition that’s better but opaque.
The human augmentation finding illustrates this: forecasters using LLM assistants improved accuracy by 24-28%. A deliberately noisy, overconfident assistant produced comparable gains to a well-calibrated one. The gain came not from the quality of the model’s predictions, but from forcing beliefs into explicit form. When a model says “65% based on these three factors,” the human can push back, adjust for missed context, notice when their gut disagrees and ask why. Implicit beliefs resist all of this.
This points toward a broader shift in how humans instruct agents. Currently, humans specify procedures: do X, then Y, then Z. But the actual goal is usually a state of the world, not a sequence of actions. Calibrated forecasting enables a different contract: specify a target probability distribution (e.g. achieve a state where P(deployment succeeds) > 85%) and let the agent search for actions that move the forecast. The agent’s job becomes finding trajectories through action-space that shift probabilities. The human retains control over what matters without prescribing how.
Cheap legible estimates also enable contracts that currently don’t get written. Drop the cost of a calibrated probability estimate by orders of magnitude and the long tail of small, novel-risk contracts becomes writable. The same economics apply to prediction markets: most questions are too niche to attract human liquidity, but forecasting agents could serve as automated market makers, providing continuous pricing for the long tail of questions nobody bothers to trade on.
The tradeoff: “73% probability” sounds more rigorous than “I think probably yes” even when it’s no better grounded. People anchor on numbers and forget the uncertainty around them. The gain from legibility only materializes in cultures that engage critically with estimates rather than treating them as oracles.
Control Signals for Autonomous Agents
Agents currently lack a principled “should I continue?” signal. Without calibrated uncertainty over their own trajectories, they can’t distinguish hard but progressing from impossible and thrashing. Calibrated forecasting fills this gap. Estimate P(success | continue) vs. P(success | switch) vs. P(success | escalate), and resource allocation follows from expected returns rather than heuristics. Concretely: use a forecasting model as back-pressure in an agentic loop. Generate a candidate output, forecast its probability of meeting the target, iterate until the forecast exceeds a threshold.
The same logic applies to information acquisition. Before reading a document or calling an API, an agent could forecast how much that information would shift its estimate. If the expected shift is negligible, skip it. If large, prioritize it. PROPHET’s CIL metric is the closest existing implementation, estimating whether a question is even answerable from available evidence. Route hard problems to expensive pipelines, handle easy ones cheaply, abstain when the evidence can’t support a useful answer.
These signals also enable delegation. A supervisor agent could forecast subagent success probability before assigning work, routing tasks based on forecasted comparative advantage. Combined with calibration track records, this creates a reputation primitive: agents whose self-assessments prove accurate earn autonomy. Those with poor self-calibration get oversight.
But self-evaluation appears fundamentally harder than evaluation of external events. Schoenegger et al. (2025) found that asking models to evaluate competing hypotheses reduced accuracy. External forecasting and self-assessment probably require different training regimes: outcome-based RL on agent trajectories with success/failure labels, not prediction market questions.
Coordination Between Agents
Multi-agent systems currently either enforce consensus (losing information) or tolerate opaque disagreement (losing coordination). Calibrated forecasts offer a third option.
A probability estimate is an extremely compressed representation of all available evidence. A vector of 50 calibrated probabilities communicates more decision-relevant information than a 10,000-token reasoning trace, at a fraction of the bandwidth. For multi-agent systems, forecasts become a coordination primitive: enough shared state to act jointly without full epistemic alignment.
Two agents with explicit estimates can identify where they disagree, by how much, and what evidence would change each position.
At the network level, forecasts can serve as an orchestration protocol. In a distributed swarm, agents broadcast forecasted success probabilities on available tasks. Work flows toward agents with the highest, and historically best-calibrated, confidence. Downstream agents dependent on upstream outputs decide whether to wait or pursue alternatives based on forecasted completion probabilities. This replaces centralized task queues with decentralized allocation where forecasts function as bids. Shared beliefs also create the substrate for resource exchange: agents forecasting their own demand could negotiate spare compute or API access based on predicted utilization, a form of futures trading applied to computational resources.
The trap is treating forecast exchange as bargaining. The structure only helps if agents trace differences to different information or different models. Splitting the difference defeats the purpose.
Systemic Effects
If everyone uses similar forecasting systems trained on similar data, errors become correlated. The 2008 financial crisis happened partly because everyone used the same risk models. Ubiquitous AI forecasting could create analogous fragility. Everything looks fine until a shared blind spot triggers correlated failure simultaneously. Diversity of forecasting approaches may matter more than average accuracy.
There is also a potential for some reflexivity. When forecasts influence the things they forecast, outcomes become endogenous. An agent forecasting its own failure might withdraw effort. An agent forecasting success might commit resources. Knowing when a forecast should be insulated from its own effects is a design question that barely exists.
Retrieval-heavy architectures introduce an adversarial surface. If forecasting agents drive real decisions, there are incentives to manipulate them. Poison the information environment and you poison the forecasts downstream. Robustness to information-environment manipulation is largely unexplored.
One partial counterweight: well-calibrated forecasters generate anomaly signals as a byproduct. When a 95%-confidence prediction fails, it’s evidence that something structural has changed. Tracking these failures turns forecasting into a sensor for regime change.
Open Research Questions
Calibration transfer. Chandak et al. found that forecasting-tuned models showed improved calibration on MMLU-Pro, GPQA-Diamond, and SimpleQA. If this replicates, it suggests proper scoring rule RL produces domain-general epistemic honesty.
Retrieval contamination. Can retrieval pipelines be designed that provably avoid temporal leakage, or is contamination-free evaluation on historical questions fundamentally intractable?
Domain and horizon decomposition. Are gains concentrated in domains with structured data (finance, sports) while lagging in unstructured ones (geopolitics, technology)? How does accuracy degrade as prediction horizons lengthen? Do the architectural components contribute differently at different timescales and in different domains?
RL reward design. What is the space of reward functions that produce well-calibrated forecasters without degenerate behavior? Is there a principled way to balance accuracy incentives against calibration incentives, or does this require domain-specific tuning?
Scaling laws for forecasting. Do forecasting gains from training scale differently than general capabilities? Is there a point of diminishing returns, and does it depend on training data diversity or volume of resolved questions?
Retrieval filtering. Can filtering be learned end-to-end rather than hand-designed? What signals reliably distinguish causally relevant information from merely topically related noise?
Aggregation theory. Simple mean, median, and trimmed mean all work, but why they work is underspecified. Is there an optimal aggregation strategy conditioned on model diversity, or does it not matter much below some correlation threshold?
Non-binary question formats. Is this an architectural limitation (current systems are optimized for binary probability output), a training data limitation (most resolved questions are binary), or something deeper about how LLMs represent uncertainty over structured outcome spaces?
Benchmark answerability. Can CIL or similar filters be standardized across benchmarks? What fraction of existing benchmark questions are effectively unanswerable, and how much does this distort published results?
Synthetic data. Prediction markets and forecasting platforms generate tens of thousands of resolved questions per year. Is that enough for RL training at scale, or does the field hit a data bottleneck? Can synthetic question generation fill the gap?
Update dynamics. How should forecasting agents handle sequential updating: when to re-forecast, how to weight new evidence against prior estimates, and how to avoid anchoring on their own earlier predictions? The training regimes described in this literature optimize for single-shot accuracy, not update quality.
Self-assessment versus external forecasting. Can models trained on outcome-based RL for external events transfer that calibration to assessments of their own uncertainty and capability? If not, what training signal works for self-assessment?
Calibrated probability estimates are cheap to produce and getting cheaper, and if that holds, it matters more than any benchmark result in this piece.
Superforecaster-quality judgment costs hundreds of dollars per question-hour or robust liquidity in prediction markets. A system at 80% of that quality for fractions of a cent changes what gets forecast at all. The long tail of decisions that currently get made on intuition starts getting explicit probability estimates. Even mediocre-but-explicit beats good-but-implicit for downstream reasoning, coordination, and accountability.
The hard problems follow from improved accuracy. Correlated errors across systems sharing training data and retrieval sources. Reflexive dynamics when forecasts influence their own outcomes. Adversarial manipulation of the information environments these systems depend on.
The capability trajectory, meanwhile, has a structural tailwind that most AI domains lack: the training signal renews itself. Every prediction resolves. Every resolution is a free label.
The systems are still behind human experts in live competition, and productization lags the academic research. But they are improving in a domain that generates its own curriculum.
That combination tends to resolve in one direction.