Most traders start recording their own ticks because vendor history is a separate heavy product capped at whatever lookback the vendor kept. That is not the best reason. A vendor file records the market; your capture records your feed, meaning its conflation, its dropouts, its symbol mapping and its latency. When fills keep landing worse than the backtest promised, only the second file explains why, because no vendor file contains your clock.
A usable self-recorded tick capture stores six fields per event: the specific contract identifier including month, the event type, price, size, the venue-reported timestamp, and the local receipt timestamp, both stamps in UTC nanoseconds. Recording both timestamps is what makes the file worth building, because their difference is the actual feed latency and no vendor archive contains it. Write append-only, fsync on a fixed cadence, and emit an explicit gap marker on every disconnect, because a recording with undetectable holes produces confident wrong conclusions.
Why record your own tick data at all?
Record your own data when the question is about your feed rather than about the market. Vendors answer only the second question. A cleaned historical file has been normalised, deduplicated, gap-repaired and stamped with exchange time, which is right for studying an instrument and wrong for studying your own system: every artefact that would have explained your slippage was polished out. Your feed may conflate, your vendor may aggregate before sending, your platform may remap symbols at roll. A self-capture keeps all of it, because it records what arrived at your socket.
That sets the honest boundary. If your feed conflates, meaning it delivers periodic snapshots of book state instead of every update, your recording is a faithful record of your feed and an incomplete record of the market: right for diagnosing your own execution, wrong for reconstructing microstructure. Confirm your feed's behaviour with the vendor rather than assuming, and read how conflation and throttling actually work first. Depth is bounded the same way, since you record only what you subscribed to, which is the practical difference between Level 1 and Level 2 futures data.
What exactly do you record per event?
Six fields per event, and the common failure is recording fewer: the instrument identifier with its explicit contract month, the event type, the price, the size, the venue-reported timestamp if the feed supplies one, and your own receipt timestamp. Both timestamps as integer nanoseconds since the Unix epoch in UTC, never a local-time string. Add a seventh that costs nothing, a feed-source identifier, because the same product reaches you differently through different vendors and a file that does not name its pipe cannot be compared against another later.
Store the identifier as the actual contract, for example ESZ6, never a continuous alias like ES1!. A continuous alias silently changes its underlying contract at each roll, and back-adjusted continuous series alter historical prices outright, so a recording keyed on the alias cannot later be re-resolved to what actually traded. CME month codes are stable: F January, G February, H March, J April, K May, M June, N July, Q August, U September, V October, X November, Z December. Equity index futures run the quarterly cycle H, M, U, Z.
Event type is the field people drop, and platform APIs show why that is fatal. MetaQuotes splits tick retrieval into distinct request types in the MQL5 CopyTicks documentation: one flag returns ticks with Bid and/or Ask changes, another ticks with changes in Last and Volume, and a third all ticks. The same page states that for any type of request, the values of the previous tick are added to the remaining fields of the MqlTick structure. So a stored row carrying a bid, an ask and a last price does not say which of the three was the actual event. Without an explicit event-type flag you can never separate a quote update from a print, and every statistic afterwards inherits that ambiguity.
Why two timestamps are non-negotiable
The venue timestamp says when the market did something. The receipt timestamp says when your system learned about it. Their difference is your feed latency, it cannot be reconstructed from either stamp alone, and no vendor file contains it because no vendor knows when your machine received anything.
Name the hops and the measurement becomes obvious. A price event is created at the matching engine, stamped at the venue gateway on its way out, crosses the public network, passes through your vendor's aggregation or conflation layer, arrives at your network card, and is read by your process. The venue stamp lands near the start of that chain, your receipt stamp at the end, and everything between them is what you are measuring. Store one end only and the measurement is gone forever.
Stamp the receipt time as close to the socket read as possible: before parsing, before queueing, before any business logic touches the buffer. A stamp taken after your decoder runs measures your decoder, not your feed, and that is the most common way a hand-built capture produces a latency number that is real and meaningless. Kernel or network-card hardware timestamping, where supported, removes your own scheduler from the measurement.
Clock choice is the second trap. The stored UTC stamp needs a settable wall-clock source to stay comparable to the venue stamp, and that clock takes discontinuous jumps and NTP frequency adjustments. A monotonic clock is immune to jumps and correct for intervals inside your process, but has no relationship to UTC. Use both, for different jobs.
Negative feed latency is not a fast feed. It is a broken clock, and it invalidates every number in the file.
Build the clock alarm into the analysis. Suppose true venue-to-application latency is 1.2 ms while your host clock runs 5 ms behind true UTC. The computed value is 1.2 - 5 = -3.8 ms. Negative latency is physically impossible, so every negative value is a clock-discipline failure or a parsing bug, and you count how often it fires. With 5 ms of clock error, a genuine 1.2 ms measurement sits inside noise more than four times its own size. Software NTP discipline on a well-connected host generally lands in the low milliseconds and PTP with hardware support far tighter, but measure the offset your host actually holds rather than trusting the category.
What format survives a year of this?
Not CSV and not JSON. Both store numbers as text and repeat field names or quoting on every record, the wrong shape for millions of near-identical rows.
The size penalty is arithmetic, not opinion. A fixed-width binary record holding instrument id as uint32 (4 bytes), event type as uint8 (1), price as a scaled int64 (8), size as uint32 (4), venue timestamp as int64 nanoseconds (8) and receipt timestamp as int64 nanoseconds (8) totals 4 + 1 + 8 + 4 + 8 + 8 = 33 bytes written packed. Serialise it packed rather than dumping a native struct, because a compiler laying those fields out with natural alignment pads the record to 40 bytes and that padding buys nothing on disk. The same event as a CSV line, ESZ6,Q,6412.25,17,2026-12-15T13:30:00.123456789Z,2026-12-15T13:30:00.124913204Z plus a newline, is 80 bytes. That is 80 / 33 = 2.42x the packed binary size before compression, and still 80 / 40 = 2.0x against the padded layout, on every event.
Delta-encoding the timestamps pays again. Consecutive events arrive microseconds apart, so store each timestamp as a delta from the previous record. Replacing two int64 fields (16 bytes) with two int32 deltas (8 bytes) takes the 33-byte record to 25 bytes, a saving of 8 / 33 = 24.24%. Check the range: a signed int32 of nanoseconds spans 2,147,483,647 ns, about 2.147 seconds, and an unsigned int32 spans 4,294,967,295 ns, about 4.295 seconds. Any inter-event gap longer than that, an overnight halt or a disconnect, overflows the delta, which is why you write an absolute-timestamp anchor record at every rotation and every gap marker.
| Property | CSV / JSON | Append-only binary log | Columnar (Parquet) |
|---|---|---|---|
| Bytes per event (example above) | 80 | 33 packed, 25 with deltas | Smaller again after column compression |
| Safe to write live | Yes | Yes, truncated tail still readable | No, footer written last |
| Query a time range by instrument | Full scan | Full scan or custom index | Reads only relevant column chunks |
| Compression effectiveness | Poor, values scattered across rows | Moderate | High, similar values sit adjacent |
| Right role in the pipeline | None | Capture target | Analysis target after batch conversion |
Refuse anyone's gigabytes-per-day figure: volume varies by more than an order of magnitude with instrument, session, subscribed book depth and conflation. Capture one full session for your real instrument list on your real feed, read the bytes off disk, call that B, and budget B x instruments x roughly 250 sessions per year. For shape only: a B of 1 GB across 4 contracts gives 1 x 4 x 250 = 1,000 GB of raw log before compression. That 1 GB is a placeholder for your measurement, not a claim about any instrument.
Durability, rotation, and the fsync everyone forgets
One writer per file, records only ever appended, filenames stamped with their start time and never reused. Rotate on a fixed schedule, hourly is a sane default, so a corrupted tail costs one hour rather than a session.
Durability is a syscall, not a hope. The fsync(2) man page states that fsync flushes all modified in-core data for the file referred to by the descriptor to the disk device, and that the call blocks until the device reports that the transfer has completed. Until you call it your writes sit in page cache, and a power loss or a hard kill takes them. The same page documents fdatasync as the variant that does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled, which is the right call on an append-only log where file size is the only metadata that matters.
Pick the cadence as a loss budget stated in events. At an illustrative measured 800 events per second, flushing every 2 seconds puts 800 x 2 = 1,600 events at risk in a hard crash. Flushing every 200 ms cuts that to 800 x 0.2 = 160 events but multiplies your fsync call rate by 10, costing IOPS on whatever device your trading logs share.
The fsync(2) man page is explicit that calling fsync does not necessarily ensure the entry in the directory containing the file has also reached disk, and that an explicit fsync on a file descriptor for the directory is also needed. On file rotation that is the difference between a durable new file and a file that does not exist after a crash.
One consequence decides your pipeline shape: never write Parquet live from the capture process. The Apache Parquet file format documentation states that file metadata is written after the data to allow for single pass writing, and that readers are expected to first read the file metadata to find all the column chunks they are interested in. A process that dies before that footer is written leaves a file readers cannot open at all. So the pipeline splits: the socket reader hands events to a bounded queue with a drop counter, a single writer appends them to a raw crash-safe log with periodic fdatasync and hourly rotation, and a separate batch job converts only closed segments into Parquet partitioned by date and instrument. Crash safety at write time, query speed at read time, and the stages fail independently.
Keep the capture off the hot path and, if you can, off the shared disk. On a shared volume, an fsync stall or a full disk back-pressures into the feed reader and makes you drop the very events you built the thing to record. Separate thread with a bounded queue at minimum, separate disk preferably, separate host ideally. Decide retention now rather than during the panic, and partition directories by date and instrument so pruning is a directory delete rather than a query.
Gap markers: the feature that makes the file trustworthy
A recording with undetectable holes is worse than no recording, because a silent gap looks exactly like a quiet market and the analysis that follows will be confidently wrong.
Write an explicit marker record into the same stream whenever the feed disconnects, whenever you reconnect, whenever a sequence number jumps, and whenever your bounded queue drops an event. Never interpolate across a gap. If your feed carries sequence numbers and you record 1,204,551 followed by 1,204,559, the missing message count is 1,204,559 - 1,204,551 - 1 = 7. Write a marker carrying both bounding sequence numbers, both receipt timestamps and the count 7, so later analysis excludes that window instead of reading it as zero activity.
Write a session marker at the head of every file: software build, feed vendor and version, exact subscription list, host, and clock-discipline state. A year later the file is interpretable only if it says what produced it.
Are you even allowed to record it?
Nothing in this section is legal advice. Exchange market data is licensed, the licence governs recording, storing and above all redistributing it, and receiving data does not by itself grant a right to store it.
Exchanges treat these as separately defined categories, and their published policy indexes show it. The NYSE market data policies index lists, as distinct named documents, a Non-Display Use Policy for proprietary data products and a Historical Use of Real-Time NYSE Proprietary Data Products Policy, alongside separate subscriber, vendor and academic policies. A dedicated policy for the historical use of a real-time feed is the tell: turning a live feed into stored history is a specifically addressed activity, not a free side effect of holding a subscription. Display use (a human reading a screen) and non-display use (data consumed by an automated process) are commonly defined and priced separately too, and a capture pipeline feeding a backtest is closer to the second. That is an equities venue, so treat it as evidence that the categories exist, not as your rulebook: check the definitions published by the exchange whose product you actually record.
If your data arrives through a prop firm's platform or a broker's feed, the governing agreement is theirs, not yours. Put these questions to that vendor, broker, firm or exchange in writing before you start, and to a lawyer if the answer matters commercially: may I record this feed for personal research, may I retain it after the session ends, does the answer change for derived data such as bars, and does it change for delayed rather than real-time data. This article cannot answer them for your agreement.
Handing your capture file to another trader, publishing it or posting samples is a categorically different act from keeping it on your own machine, and it is the act most likely to breach an agreement. This is not legal advice, so treat redistribution and derived-data questions as ones for the licensor or a lawyer, never as defaults.
The payoff: a latency distribution nobody will sell you
With both timestamps stored you can compute the full distribution of feed latency across a session and slice it by time of day. No cleaned vendor file can produce it, because none contain your receipt clock.
Here is the shape of the finding, with illustrative numbers you would replace with your own. A session on the ES front month yields a median receipt-minus-venue delay of 1 ms and a session-wide p99 of 5 ms, unremarkable so far. Restrict the same data to 09:30 to 09:35 ET and the p99 is 40 ms. The opening-window tail is 40 / 5 = 8x the session tail, and that is when a large share of strategies trade.
Check sample counts before believing any percentile. At 800 events per second, five minutes holds 800 x 300 = 240,000 events, so the p99 rests on 240,000 x 0.01 = 2,400 observations and the p99.9 on 240, both real percentiles. A window holding only 1,000 events gives a p99.9 determined by 1,000 x 0.001 = 1 single observation, which is not a percentile at all. Report the sample count next to every percentile or the number is decoration.
The tail translates into money. ES carries a $50 per index point multiplier and a 0.25 point minimum tick, so one tick is 0.25 x 50 = $12.50 per contract, and those are stable exchange constants. Assume, illustratively, that during the opening burst price moves roughly one tick per 40 ms, so acting on data 40 ms stale costs about one tick of adverse selection per entry. A strategy taking 8 entries in that window loses 8 x $12.50 = $100 per contract per session. Across 20 sessions that is 20 x $100 = $2,000 per contract, and at 3 contracts, 3 x $2,000 = $6,000. The staleness, the move rate and the entry count are inputs you must replace with your own measured figures; the multiplier and the tick value are not.
When not to build this
If your strategy consumes bars rather than individual events, a capture pipeline buys precision you will never spend. That is the honest tradeoff, and it disqualifies most traders who ask about this. The engineering is real (a writer, a rotation scheme, a converter, a retention policy, monitoring for silent failure) with ongoing storage cost, and the conclusion of the tick data versus bar data comparison applies directly: if bar-level simulation already answers your questions, event-level capture is resolution your model discards. The sharper objection is that the pipeline can damage the system beside it, since an fsync stall or a full volume on a shared disk propagates back into the feed reader and causes drops in the live trading path. A capture that degrades execution to study execution is a net loss.
There is also a much cheaper first step that answers most of the questions people build full captures for: record only your own order lifecycle events and the quotes at those exact instants. It is a tiny fraction of the volume, carries none of the redistribution exposure of a bulk archive, and directly answers "was I filled where I expected". Start there.
If you do build the full thing, monitor the capture itself, because a pipeline that silently stopped three weeks ago is the standard outcome. Alert on bytes written per interval falling to zero during known session hours, on the drop counter increasing, on gap markers exceeding a threshold, on disk free space and on clock offset. Budget for file count too: rotating hourly across a near-continuous futures session of roughly 23 hours produces 23 files per instrument per day, or 23 x 250 = 5,750 files per instrument per year before conversion. Verify current session hours for your product on the exchange product page before hard-coding them, since maintenance windows and holidays change. Build this when a specific question demands data nobody will sell you, not because self-collected data sounds rigorous.
Frequently asked questions
What fields do I need to record for tick data to be useful later?
Six fields per event at minimum: the instrument identifier with its explicit contract month, the event type (trade, bid update or ask update), the price, the size, the venue-reported timestamp, and your own receipt timestamp. Store both timestamps as integer nanoseconds since the Unix epoch in UTC. Add a feed-source identifier as a seventh field so the file can later be compared against a recording from a different vendor or platform.
Why do I need both the venue timestamp and my own receipt timestamp?
Because their difference is your actual feed latency, and it cannot be reconstructed later from either one alone. The venue stamp says when the market did something; the receipt stamp says when your system learned about it. No vendor historical file contains the second one, since no vendor knows when your machine received anything, which is why this measurement exists only in a self-capture.
Should I store the continuous contract symbol like ES1! or the specific month?
Store the specific contract, for example ESZ6, never a continuous alias. A continuous alias silently changes its underlying contract at each roll, and back-adjusted continuous series alter historical prices outright, so a recording keyed on the alias cannot be re-resolved to what actually traded. CME month codes are stable: F, G, H, J, K, M, N, Q, U, V, X, Z for January through December.
Is CSV good enough for storing tick data?
No, CSV runs roughly 2.4x the size of an equivalent packed binary record before compression and it compresses poorly because similar values are scattered across rows. A typical event costing 33 bytes as a packed fixed-width binary record costs 80 bytes as a CSV line, and you pay that multiplier on every event. Capture to an append-only binary log and convert to a columnar format for analysis.
Can I write Parquet directly from my capture process?
No, because Parquet writes its file metadata after the data to allow single pass writing, so a process that dies before the footer is written leaves a file readers cannot open. Use a two-stage pipeline instead: capture to an append-only raw log that stays readable up to the last complete record even when truncated, then convert closed segments to Parquet in a separate batch job. That gives crash safety at write time and query speed at read time.
Am I allowed to record market data from my prop firm's platform?
That depends entirely on the agreement governing that feed, and this is not legal advice. If the data arrives through a prop firm or broker, the agreement is theirs rather than yours, and exchanges publish separate named policies for non-display use and for historical use of a real-time feed. Ask the vendor, broker, firm or exchange in writing before you record, and ask a lawyer if the answer matters commercially.
How much disk space will a year of tick capture take?
There is no honest general answer, because volume varies by more than an order of magnitude with instrument, session, subscribed book depth and whether your feed conflates. Capture one full session for your real instrument list on your real feed, measure the bytes on disk, then multiply by instrument count and roughly 250 sessions per year. Any figure quoted without that measurement is a guess you would be budgeting storage against.
How often should I fsync a tick capture log?
Choose the interval as an explicit loss budget measured in events, not as a feeling. At 800 events per second, flushing every 2 seconds risks 1,600 events in a hard crash while flushing every 200 ms risks 160 but multiplies your fsync rate by 10 and costs IOPS on the same device. Use fdatasync for the periodic flush, and remember that the fsync(2) man page states rotating to a new file also requires an explicit fsync on a descriptor for the containing directory.
What is a gap marker and why does it matter so much?
A gap marker is an explicit record written into the capture stream whenever the feed disconnects, reconnects, jumps a sequence number, or your queue drops an event. It matters because a silent hole in a recording looks identical to a quiet market, so analysis over that window produces confident wrong conclusions. Write the bounding sequence numbers, both receipt timestamps and the missing message count, and never interpolate across the gap.
Is it worth building a capture pipeline if I trade off bars?
Usually not, because a bar-consuming strategy discards the event-level resolution the pipeline exists to preserve. The engineering, the ongoing storage cost and the operational failure modes are real, and a capture sharing a disk with your trading process can back-pressure into the feed reader and cause the drops you were trying to measure. A cheaper first step is recording only your own order lifecycle events and the quotes at those instants.