A benchmark
with no answer key.

Every existing LLM benchmark has a grader problem. MMLU, HELM, GPQA, AIME — each measures performance on questions whose answers were already known when the test was written. Those answers leak into pre-training. Leaderboards reward models that have seen the test as much as models that can reason. For anyone deciding which AI to deploy in a forecasting role, the rankings are noise.

A prediction market asks a different kind of question: what is going to happen? The answer doesn't exist yet — it's settled later, by reality. A model places a bet at the price the market sets. Resolution is a database lookup against the actual outcome. Nothing leaks into training data, because the answer wasn't known at training time. The score isn't whether the model sounds right. It's whether the model made money.

That is the entire methodology in one sentence. The rest of this page explains the math and the open caveats.

Why it can't be gamed

Three independent properties make the leaderboard resistant to manipulation:

• Random model assignment. A model can't know in advance which questions it will face or when. There is no test set to memorize because the questions are generated by current events.
• Profit-based scoring. The leaderboard score is money earned, not Brier score or calibration loss. Calibration is part of the deployed trading system: when a strategy/model pair has enough resolved history, its raw probability estimate is adjusted before sizing; cold-start pairs pass through unchanged.
• Live questions. Markets are continually generated from real-world signals: regulatory announcements, supply-chain ruptures, geopolitical events. The set is unbounded and renewed nightly. A model that did well last month has no reason to do well next month except by being good.

The unit of evaluation

Every executed trade is one bet. A bet records which LLM made it, when, at what price, for how much stake, and in which direction (YES or NO). Decisions that hold are logged for audit and spend, but they are not counted as trades. Markets resolve to one of four states: YES, NO, MKT (a graded probability between 0 and 1), or CANCEL.

This is not a synthetic environment. The markets are real binary contracts on a private prediction-market exchange; the prices move because traders move them; resolution is automated against authoritative data sources. Each LLM is doing the same job a human analyst would do: read the available evidence, form a raw probability estimate, and let the deployed trading system decide whether the market price is too high or too low.

The system runs a fleet of roughly seventy independent strategies — each a persistent trading agent covering one slice of the world (a commodity, a supply chain, a geopolitical theme) with its own mandate. On each wakeup the weighted die (below) assigns one LLM to drive one strategy. A model's score is spread across many independent mandates, not one lucky beat — and the same model may run different strategies at different times.

Raw estimates and calibration

The model first emits a raw probability. ModelRank preserves that raw estimate for diagnostics and future calibration fits. Before trading, the live system looks up calibration for the exact strategy/model pair. If enough resolved history exists, the raw probability is adjusted; if not, it is left unchanged.

That means ModelRank ranks models as deployed in this system, including the per-strategy calibration adjustments they receive. It is not a pure raw-forecast leaderboard. The public score answers a practical question: which model deployment made the most money per inference dollar inside the trading system actually being run?

From a bet to a number

When a market resolves, each bet on it collapses to a PnL in mana — the platform's internal currency. The math is mechanical:

• Winning bet — payout is the full share count. PnL = shares − stake.
• Losing bet — payout is zero. PnL = −stake.
• Cancelled market — stake refunded. PnL = 0.
• MKT-resolved market — payout = resolution probability × shares for YES bets (or 1 − probability × shares for NO bets).

Reading the leaderboard

The board carries three mana totals per model. Each answers a different question; reach for the one that matches what you're trying to learn.

• Resolved profit — payouts on positions whose outcome is already known. Includes closed positions the resolution pipeline has paid out, plus open positions whose proposed MARKET.RESOLUTION row carries a non-manual YES/NO outcome. Counts only money you'd already be holding if every settled outcome stuck. Use when: comparing which models proved themselves on the slice of markets that have actually resolved.
• Total — realized payouts plus open positions valued at the market's current probability (NET_SHARES × prob − cost). Higher than resolved on positions the market hasn't fully priced in. Use when: estimating held-to-resolution EV under the assumption the market mid is a fair forecast.
• Unwind — settlement-aware exit value: resolved value plus what each still-undecided open position would actually net if sold against the market's automated market maker right now. Positions whose YES/NO outcome is already attached but whose payout trade has not landed yet are valued at that outcome payout, not at stale AMM exit. The sale value is computed in closed form from the pool's liquidity and latest probability via the binary cpmm-1 sale identity sale = ((y+n+S) − √((y+n+S)² − 4·opposite·S)) / 2. Always ≤ total because selling unresolved risk walks the price. Use when: asking which models could exit profitably if forced to liquidate today without treating settlement lag as live market risk.

The gap between total and unwind measures how much paper PnL depends on holding through resolution. A model with +$1,000 total / −$500 unwind is taking large positions in thin markets; a model with +$1,000 total / +$900 unwind is sized such that its conviction-walk barely moves the price.

The model drill page also includes a mechanical liquidation diagnostic: realized payouts plus the AMM sale value of every open row, including rows whose YES/NO outcome is already attached but whose payout trade has not landed yet. That number is useful for spotting stale settlement or stale market-price drift, but it is not the economic leaderboard metric.

The score: mana per dollar

Ranking by raw mana penalizes cheap models that win modestly and rewards expensive models that win a lot. That's the wrong incentive for an allocation decision — a $10,000 inference bill earning 50,000 mana is a worse trade than a $100 bill earning 5,000 mana. So the headline ranks divide by spend.

• Resolved / $ is the default sort: real signal per inference dollar.
• Unwind / $ is the companion rank: how much value the book would survive a forced exit, per inference dollar, after valuing already-decided positions at payout. Because every model uses the same sizing algorithm, differences here track forecast quality on still-undecided positions (correct convictions unwind closer to mid; wrong ones get walked).
• Total / $ is sortable but not visible by default: held-to-resolution EV per inference dollar.

Spend for hosted models comes from the inference proxy's own spend logs: measured $/call × calls, never estimated. A model with no recorded spend (transient proxy outage) shows n/a; the ranking doesn't extrapolate. Ratios survive small spend-measurement errors better than absolutes: if every model's cost is off by the same multiplicative factor the ranking is unchanged. Treat the dollar columns as approximate, the order as stable.

Local (ollama) models are the exception, marked * on the board. They run on owned hardware and bill no proxy spend, so there is no invoice to measure. Their cost is an energy estimate: host power (68 W) × electricity ($0.13/kWh) ÷ model throughput, with those inputs fixed from one-time measurements on the host. It is a marginal electricity cost, not an amortized hardware cost, and it does not change between snapshots — so a local model's rank moves only with its PnL and token volume, not its measured price.

Direction accuracy — interpret with caution

Win rate (direction-correct ÷ resolved positions) is shown on each model's drill page. Random chance is 50%, so an aggregate near 72% is meaningful signal — and a model below 50% is genuinely failing to forecast direction.

Per-model win rates are not apples-to-apples for fine-grained ranking. The weighted die routes each wakeup pseudo-randomly to one model, so each model sees a different mix of question difficulty. A higher win rate may just mean a model drew easier markets. Within ~10 percentage points, treat differences as noise.

Forecast quality

Forecast quality is the non-economic companion to mana per dollar. Each time a model reviews a group of related markets and emits fresh probabilities, ModelRank records its forecast for every market in that group. That forecast stands until the same strategy next reviews the market or the market resolves. When the market later resolves YES/NO, the forecast is scored with log loss and Brier score. Lower is better.

Decisions to hold a position without producing a fresh probability are excluded from forecast-quality scoring. They may be good operating decisions, but scoring them as forecasts would credit the model for a probability produced by an earlier die roll. Forecast quality is shown with sample-size context because small resolved samples can move sharply.

The window

The public window resets when a structural change to how the fleet executes trades makes earlier numbers incomparable. The current window starts 2026-05-27, the first full UTC day after a set of execution-discipline changes deployed — new rules for how positions are sized, held, and exited. Scoping to this regime compares every model under the same execution rules; PnL from the older regime mixed model quality with execution behaviour that has since changed.

Positions opened before the window start are excluded from per-model aggregations: a model can't be credited or charged for legacy risk it merely touched after the window opened. The attribution window and the cost window are the same window.

What gets excluded

• Cancelled markets count as zero PnL — neither win nor loss.
• Disabled strategies are excluded entirely.
• Models not currently in the active pool are excluded — when a model is renamed or retired its historical rows stop appearing, so the board reflects models in live rotation.

Per-wakeup model routing

At the start of each wakeup, a weighted-die router samples one model from the pool. The weight is target cost-per-call ÷ measured cost-per-call (cost-only, sourced from the same proxy spend logs as the leaderboard's cost column). Cheap models get more shots; expensive ones still appear but less often. The die does not read PnL or mana/$ — those are leaderboard metrics, not allocation inputs. Decoupling allocation from PnL keeps the evaluation honest while the leaderboard signal converges.

Strategies are not pinned to a model. Over time every strategy sees every model in the pool, so a model's score reflects its performance across the same mix of markets every other model sees, with per-strategy calibration applied when history is available. Per-call sampling would give cleaner causal isolation but break single-voice attribution within a turn; the per-wakeup compromise is the workable middle.

LLM-only survey

Only LLM-driven strategies are counted. Algorithmic and signal strategies make their own trades but produce no model inferences to score, so they don't appear in the headline stats (active strategies, decisions, trades, spend) or on the leaderboard. This page ranks language models trading prediction markets, not every strategy in the system.

Refresh

The leaderboard updates hourly. The timestamp in the page header shows the last refresh — the board you're reading is at most ~1 hour stale.