← Back to blogs

The Illusion of Time: Why Your Timestamps Lie

Every engineer has sorted events by created_at and been wrong. Physical clocks drift, NTP corrects in jumps, and two machines that disagree by 50 milliseconds will happily invert causality. If ordering matters, wall time is not a clock — it is a liability.

← Previously: The Unreliable Network: Why Two Generals Can Never Agree

1. Physical Clocks: Honest but Unreliable

Quartz oscillators drift. Temperature, voltage, and manufacturing variance push two servers apart by 10–100 ppm even when NTP is running. NTP disciplines the clock by slewing or stepping, but it tracks an external reference with asymmetric network delay — it does not give you a shared now.

In production you observe this as: Spanner's TrueTime explicitly advertises an uncertainty window [earliest, latest] instead of a point. CockroachDB's clock-offset monitoring pages when skew exceeds 500 ms. And every "sort by timestamp" query that ever double-counted an event because producer B's clock was 80 ms ahead of producer A's.

Real NTP behavior (chrony logs):
system clock wrong by 0.042s → slewed over 12s
system clock wrong by 0.311s → stepped (jump!) at 14:03:01.112
-- Any event stamped during the slew has non-monotonic physical time.

Monotonic clocks (CLOCK_MONOTONIC) solve elapsed-time measurement on one host, not ordering across hosts. For cross-node ordering you need a different primitive.

2. Causality Over Chronology: Lamport's Insight

Leslie Lamport asked a simpler question: what does "A happened before B" actually require? If A and B are on different nodes with no message between them, they are concurrent — wall-clock comparison is meaningless. If a message flows from A to B, then A causally precedes B, and any clock must respect that.

The Lamport clock is a per-node counter plus one rule:

• On local event: L := L + 1.
• On send: attach L to the message, then L := L + 1.
• On receive: L := max(L_local, L_msg) + 1.

The guarantee is one-directional but powerful: if A causally precedes B, then L(A) < L(B). The converse is not promised — concurrent events may still be ordered arbitrarily — but no causal arrow is ever inverted. That is enough to build a total order that respects causality: break ties by node ID.

Move the slider below: Increase clock drift and watch physical timestamps (red) invert the true send→receive order while Lamport timestamps (teal) stay monotonically causal below. Emit new events and messages to see max()+1 repair the ordering in real time.

3. Where Physical Time Still Matters

Lamport clocks tell you order, not when. For TTLs, GC, retention, and human-facing timestamps you still need physical time — with discipline:

• Bound the skew. CockroachDB and Spanner refuse to serve if clock offset exceeds the configured max (typically 250–500 ms). Monitor chrony tracking → System time offset per host.
• Use hybrid clocks. Hybrid Logical Clocks (HLC) combine a physical component (pt) with a logical counter (l). They stay close to wall time for readability, but guarantee hlc(e1) < hlc(e2) whenever e1 causally precedes e2 — even during NTP steps.
• Never sort cross-shard events by wall time alone. If you must merge, attach the sender's HLC/Lamport timestamp at the source and sort by that, not by arrival ingest time.

HLC timestamp = (wall_ms, logical_counter, node_id)
send: hlc = max(wall, hlc+1)  // logical ticks inside same ms
recv: hlc = max(wall, hlc_local, hlc_msg) + 1
-- Monotonic, causal, and within ~ms of physical time.

4. Vector Clocks: When You Need to Detect Concurrency

Lamport clocks collapse concurrent events into an arbitrary total order — useful for a log, dangerous for conflict detection. Vector clocks keep a per-node counter vector so you can distinguish causally ordered from concurrent. Dynamo, Riak, and Cassandra's lightweight transactions use this to know when to surface a conflict to the application instead of silently picking a winner.

The cost is size O(n) per timestamp and the operational reality that most teams do not want to resolve conflicts client-side. Prefer HLC/Lamport for ordering and reserve vector clocks for systems that explicitly embrace multi-version concurrency (shopping carts, offline sync).

5. What Clocks Cannot Fix

A perfect clock — physical or logical — does not tell you whether the node that stamped it was honest or even alive when it stamped it. Clocks order events from correct participants. They say nothing about participants that crash mid-protocol or lie.

Perfect clocks still leave the harder question untouched: whether the node that stamped the event was even correct to do so. Once a process can crash and wake with amnesia — or send a different story to each peer — ordering alone is not enough. How much that costs depends on what you decide a failure is allowed to be, which is the subject of Failure Models: The Crash, the Recovery, and the Liar →