Data, before
the model.
Every execution claim begins with a source. Our research inputs combine market identity, observed trades, order-book history, and final settlement evidence—each with its own clock and limitations.
In this chapter
The source map.#
The upstream market-data project owns collection, storage, and publication. Backtest Lab consumes selected historical records or frozen catalogs; a replay does not silently refresh its inputs from the latest market state. A dataset's availability is determined for the requested market and interval.
| Dataset | Original source | Role in a replay |
|---|---|---|
| Markets & tokens | Polymarket Gamma metadata, official CLOB identity, and protocol mappings | Bind markets, conditions, outcome indices, and raw token IDs. |
| On-chain trades | Polygon exchange-contract OrderFilled logs | Construct economic trade prints and preserve source identities. |
| Block timestamps | Polygon block-header evidence | Map block coordinates to historical UTC time. |
| Native L2 | Polymarket CLOB market WebSocket; identified REST snapshots where applicable | Reconstruct the visible price-level order book and its updates. |
| Resolution & payout | Canonical Oracle, adapter, and settlement-contract events | Determine which tokens can be redeemed, for how much, and when. |
| External signals | Caller-supplied, separately identified reference feeds | Supply strategy features with observation time and provenance. |
Prices, sizes, and identities are normalized for computation while the original evidence remains distinguishable. A WebSocket trade message, an on-chain fill log, and a simulated order fill are three different records; combining them as independent executed volume would double-count activity.
First, identify the asset.#
The market catalog provides questions, slugs, descriptions, categories, scheduled dates, observed status, and outcome mappings. Gamma is a principal source of descriptive metadata. Official CLOB and protocol records also supply identities, including cases without a Gamma market.
A replay joins data using the market, condition, and original outcome token. Titles and labels are presentation fields. “Yes,” “No,” “Up,” “Down,” and a team's name do not establish token identity or its payout position. A source-list index must not be substituted for a verified protocol outcome index.
- Token IDs stay lossless. Large integer identifiers are carried as strings, not JavaScript numbers.
- Rules have an observation time. A current status or fee setting cannot be assumed to have been visible throughout history.
- Scheduled end is metadata. It does not establish the actual stop-trading time, the final Oracle event, or the time cash becomes redeemable.
- Special structures keep their actual mapping. Multi-outcome, negative-risk, and combinatorial assets cannot be relabeled as an ordinary binary pair without evidence.
For a point-in-time strategy, use the relevant historical metadata observation where available. A valid token mapping today is useful for identity, but does not by itself prove when that mapping first became available to the strategy.
From logs to a trade tape.#
Historical exchange-contract logs supply block and transaction coordinates, participants, token identifiers, raw asset amounts, and fees. The upstream primary store retains an event body together with its source versions and conflicts. Backtest readers can reconstruct the required original records without treating a compatibility source name as a second independent observation.
- 01 / SOURCEExchange logChain, transaction, log index, and integer amounts.
- 02 / SEMANTICSResolve legsIdentity, economic direction, and conservation.
- 03 / TIMEJoin clocksBlock time and available transaction ordering.
- 04 / INPUTFreeze printsReplay records with source references.
Why a raw event is not yet a replay trade
One transaction can emit multiple fill logs. Ordinary trades and mint/merge arrangements have different asset flows; taker summaries and passive legs can describe overlapping economic activity. Simply summing every log or interpreting a source side code as aggressor direction can produce an incorrect tape.
The paired-tick builder checks participants, condition/outcome mapping, and integer asset conservation under its event-semantics contract. It accounts for valid ticks and quarantined records in a build receipt. The separate active-summary research reader consumes taker summaries without stacking the passive legs on top. These input constructions have distinct semantics and must retain their own provenance.
What the engine needs
A replay print carries market/token identity, price, share quantity, block/time coordinates, direction evidence, and traceable trade/transaction/log identifiers. The catalog retains canonical ordering and atomic transaction or trade-group boundaries. Unknown direction, conflicting amounts, or an invalid identity is not repaired by guessing from the price move.
Execution price is collateral per share; quantity is shares; cash and fees are collateral. Raw chain amounts retain their integer and contract-specific units before conversion. The public tutorials label their configured collateral as USDC. A fee observed in an old log is not automatically the correct fee for a hypothetical order under a different schedule.
Reconstruct the book we observed.#
Native historical L2 comes from recorded Polymarket market-channel observations. The archive retains snapshots, price-level changes, trade messages, and relevant rule or connection events. Source event time and local receive time are preserved separately. The current upstream message families are described in Polymarket's real-time data reference; historical archives retain their own schema versions.
| Observation | Meaning in the archive | Replay treatment |
|---|---|---|
| Book snapshot | Price/size levels for one token at an observation boundary | Establish a source's baseline; respect the retained depth. |
| Price change | The new size at a price level, not an additive volume delta | Replace that level's size; zero removes it. Apply the complete message atomically. |
| Trade message | An observed trade broadcast | Use its source meaning; do not count it again as an independent on-chain trade. |
| Best quote / tick change | Top-of-book information or a price-step rule update | Validate quotes or update rules; a quote does not supply missing depth. |
| Connection / frame evidence | Source session, sequence, and message boundaries | Separate streams and assess whether updates can be applied. |
Multiple feeds keep separate histories
Where a union archive shares identical message bodies, observation receipts retain the original source, connection, generation, frame, and receive time. Each source needs its own usable baseline. An uninitialized feed's delta cannot edit another feed's book. A shared body is a storage optimization, not proof that two observations occurred at the same instant.
The reader advances only through complete, visible messages. A future snapshot cannot repair an earlier decision. A late snapshot may help recovery only from the time it was actually received and only when its relation to already observed updates can be established. Unsupported old records that lack atomic update identity cannot silently become complete modern evidence.
Depth and history are bounded
A retained five-level archive proves, at most, its retained price levels. It cannot answer a request for deeper liquidity by filling in the missing ladder. Cached slices and restored archives inherit the original source's limits. Price-level L2 also does not reveal every individual resting order or a hypothetical order's exact queue position.
Keep the clocks separate.#
| Clock | What it describes | What it does not establish |
|---|---|---|
| Block / event time | When the chain block or upstream event occurred | When a local strategy received it. |
| Source receive time | When a particular collector observed the message | When every downstream consumer had received it. |
| Decision time | The information cutoff at which the strategy acts | Immediate arrival at the simulated venue. |
| Arrival / response time | When an order becomes eligible and when execution feedback reaches the strategy | Permission to use later information in an earlier decision. |
OrderFilled time joins use stored block-header timestamps. Missing exact time is not replaced with ingestion time. If a research path explicitly uses interpolation, it must retain the interpolated provenance rather than claim exact header evidence. Where transaction ordering is unproven, a block-atomic policy preserves that uncertainty instead of inventing a within-block sequence.
Native L2 point-in-time queries use the source's receive cutoff. They establish what the observed source could provide, not what a separate archive consumer necessarily knew at that moment. Run time filters are explicit; catalog windows use UTC half-open intervals [start, end), while individual snapshot and lifecycle boundaries follow their documented contracts.
A book update stamped 12:00:00.000 but received at 12:00:00.180 is unavailable to a receive-time decision at 12:00:00.100. A later, more complete archive does not change that decision's information set.
Settlement comes from protocol evidence.#
The settlement input associates a market and its actual settlement condition with final protocol events. It retains chain/contract/transaction/log identity, the final event time, and token-specific payout fractions. UMA proposals and disputes provide lifecycle context; they are not interchangeable with a final payout event. See the upstream resolution overview for that broader lifecycle.
The framework reads historical evidence as of the requested cutoff. A current resolved flag, a scheduled end date, or an outcome label does not authorize historical redemption. A final event after the cutoff remains a later event. Missing token mappings and conflicting payout evidence remain unresolved.
Verified payout uses exact numerator/denominator evidence per token. Binary wins, fractional or cancellation payouts, non-binary outcomes, and supported negative-risk or combination mappings keep their actual semantics. The reader does not force an incomplete set of assets into an invented Yes/No vector.
Oracle finality authorizes a research-account redemption under the run's policy. The resulting simulated cash movement is distinct from an on-chain wallet redemption receipt, its gas costs, or a production execution service.
Signals, marks, and synthetic fixtures.#
External signals
A strategy can consume an explicitly supplied reference feed, such as an external spot-price series, together with market observations. The signal contract records source, observation time, latency assumption, payload hash, and relevant market/token identity. Provider event time alone does not certify local availability. External reference data is not bundled automatically with every trade or L2 catalog.
Price marks
Marks value inventory; they do not create fills. A run may use explicit historical marks or, in supported dynamic trade-tape replay, the last causally observed trade for each asset. Carried marks retain their source, timestamp, age, and stale status. They are neither executable quotes nor Oracle payout values. Bars published by the separate Prices domain are also derived data, not a reconstruction of missing L2 depth.
Synthetic tutorials
The Tutorials page's small trade-tape and PML2 examples are generated from synthetic inputs by the repository's offline examples. They exercise execution and accounting without a historical data dependency. Their source label, fee assumption, and incomplete valuation are visible. Synthetic fixtures cannot establish historical data coverage or strategy profitability.
Freeze the input, preserve its identity.#
Trade catalogs contain normalized Parquet records plus a manifest binding the schema, source pin, profile identity, strategy identity, and file checksums. Orders and initial-account settings are frozen alongside the experiment. Archive-based PML2 input similarly binds selected source files, token/window scope, and reconstruction evidence.
- Select a bounded scope. Record the requested assets, dates or blocks, and actual source files.
- Validate the transformation. Preserve event identity, units, direction, atomic boundaries, and any rejected or quarantined records.
- Pin inputs and assumptions. Retain the catalog manifest, content hashes, model configuration, account, fees, and strategy settings.
- Bind the result. Store a run receipt linking that exact input to its orders, fills, ledger, report, and checkpoint.
A resumed run verifies the original identity. Replacing files or attaching new data to an old checkpoint would create a different experiment. Read caches and columnar indexes accelerate access to frozen input; they do not expand historical coverage or make future rows available sooner.
Coverage belongs to the run.#
Requested range, observed records, reconstruction readiness, and source completeness are separate measurements. A file checksum proves that a file has not changed. It does not prove that the upstream feed delivered every event. An available API or a cache hit says even less about a historical interval.
A useful result records the requested and observed dates, included markets/tokens, source versions, known gaps, exclusions, and the denominator behind any coverage percentage. Research can use a bounded, incomplete dataset if those limits are explicit and the required business evidence remains valid. It cannot present eighteen observed days as a complete six-month history.
- Missing data stays missing. A missing baseline or invalid input is not an empty but tradable book.
- No fill has a different meaning. A valid observation with no eligible liquidity is an execution outcome, not a data failure.
- Do not smooth away real events. Price jumps may reflect news or settlement; filtering, clipping, and forward-filling need an explicit research rationale.
- Old receipts retain their scope. A saved successful run proves its recorded experiment, not current storage availability or a fresh run under changed code.
Follow the source.#
- Polymarket · Market data overview — upstream public data interfaces.
- Polymarket · Real-time data — current book, price-change, and trade message definitions.
- Polymarket · Resolution — resolution rules and Oracle lifecycle.
- Backtest Lab · Replay frameworks — how these observations become execution and account results.
Repository basis: docs/architecture.md, docs/reference/execution.md, docs/reference/accounting.md, and the canonical trade catalog, OrderFilled, native archive, and settlement readers. Upstream lineage was checked against market-data's dataset catalog and OrderFilled consolidation contract. Protocol documentation describes the source; Backtest Lab's frozen schema and run receipt describe a particular experiment.