Market Data Pipeline Architecture: Ingestion, Time-Series Storage, Corporate Actions, and Survivorship Bias

Three layers, never collapsed

The architectural decision that determines whether you can trust your data years from now is separating raw capture from derived views. Every team that overwrites raw prices with adjusted ones eventually needs the originals — to reconcile with a broker statement, to debug a bad fill, to reprocess after a vendor corrects a split ratio — and cannot get them back.

• Raw layer: exactly what the venue or vendor sent, immutable, append-only, with the receipt timestamp and the source's own timestamp stored separately. Never edited, never adjusted.
• Normalized layer: unified schema across venues — symbol mapping resolved, timezones in UTC, prices in a fixed-point decimal type, sizes as integers.
• Derived layer: adjusted prices, bars at every resolution, indicators. Fully reproducible from the layers below by a deterministic job.

Store money as fixed-point decimals or integer minor units, not floats. A float price that round-trips through JSON becomes 12.229999999999999, and once that value is aggregated across a million rows the error is real. This is not pedantry — it is the most common data bug in retail-built trading systems.

Ingestion: assume the socket lies

A websocket feed will disconnect, silently stall, deliver out of order, and replay duplicates. Each of those needs an explicit mechanism, because the failure mode of ignoring them is not a crash — it is quietly missing data that your backtest later interprets as a flat market.

// Sequence-gap detection: the check that catches silent data loss.
function onMessage(msg: Tick) {
  const expected = lastSeq.get(msg.symbol);
  if (expected !== undefined && msg.seq !== expected + 1) {
    // Do not paper over it. Record the hole, then backfill via REST.
    gaps.record(msg.symbol, expected + 1, msg.seq - 1);
    void backfill(msg.symbol, expected + 1, msg.seq - 1);
  }
  lastSeq.set(msg.symbol, msg.seq);
  buffer.push(msg);
}

// Staleness watchdog: a socket that stops sending looks identical
// to a market with no trades. Heartbeats distinguish them.
setInterval(() => {
  if (Date.now() - lastHeartbeat > STALL_MS) reconnectWithBackoff();
}, 1_000);

• Persist a gap record for every detected hole and expose the count as a monitored metric — unexplained gaps are the leading indicator of bad research results.
• Idempotent writes keyed on (symbol, venue, sequence) so a replay after reconnect cannot double-count volume.
• Buffer and batch to storage; per-tick inserts will not keep up and will fall behind exactly when the market is most volatile.
• Keep both timestamps. Venue time tells you when it happened; receipt time tells you when you knew, and only the second one is legitimate for backtesting.

Ticks or bars — a storage decision with consequences

Choose by what you will actually test. Storing full tick data because it feels rigorous, then only ever running daily strategies, buys a large bill and a slow research loop. The reverse — storing minute bars and later needing to model queue position — means recollecting history you may not be able to buy retroactively. Decide from the strategy, and keep raw ticks only for the symbols you genuinely trade intraday.

Use a time-series-aware store: TimescaleDB hypertables on Postgres give you compression, continuous aggregates for bar rollups, and ordinary SQL, which matters more than raw benchmark numbers when a researcher needs to join prices against fundamentals at 11pm.

Corporate actions: the silent invalidator

A stock that splits four-for-one drops 75% overnight in raw prices. Every momentum signal, every stop-loss backtest, and every volatility estimate that reads unadjusted history sees a crash that never happened. Handling this well is the difference between research that transfers to live trading and research that does not.

• Maintain a corporate actions table — splits, dividends, spin-offs, mergers, ticker changes — with an ex-date and a ratio, sourced independently from your price vendor where possible.
• Store an adjustment factor per symbol per day, and compute adjusted prices as raw multiplied by the cumulative factor. Never mutate raw rows.
• Adjust volume in the inverse direction to price, or your liquidity filters break across every split boundary.
• Reprocess forward when a vendor issues a correction. Because the derived layer is reproducible, this is a job you run, not an incident.
• Map symbols through a permanent internal identifier. Tickers are recycled — a backtest that follows a ticker rather than an entity will happily switch companies mid-series.

Survivorship and point-in-time correctness

If your universe is 'stocks in the index today' and you test back ten years, you have selected for companies that survived. Results will look excellent and will not reproduce. The fix is point-in-time data: for any historical date, you must be able to reconstruct exactly what was known and listed on that date, including the companies that later delisted.

• Store index membership with effective date ranges, and include delisted securities with their final state.
• Bitemporal fundamentals: keep both the period a figure describes and the date it was first published. Earnings restated later must not appear in a backtest before the restatement existed.
• For crypto, the equivalent traps are delisted pairs, exchanges that shut down, and tokens that migrated contracts — the same discipline applies with less vendor support.
• Test your own data: pick a historical date, reconstruct the universe, and check it against an archived source. Most pipelines fail this the first time.