Rows Silently Stopped Appearing — A Watermark Design Bug

An incremental pipeline pulls new and updated rows from a source orders table using a watermark on updated_at:

SELECT order_id, customer_id, amount, status, updated_at
FROM source.orders
WHERE updated_at > :last_watermark
ORDER BY updated_at;
-- after load: last_watermark = MAX(updated_at) from this pull

Three months after launch, an analyst notices that a small, consistent fraction of order status updates (roughly 0.3% of all updates) never show up in the warehouse at all — not late, not duplicated, just permanently missing. Investigation reveals the source application runs on multiple applications servers behind a load balancer, and under certain conditions a server's clock can be up to 90 seconds behind the others; rows written by a lagging server occasionally get an updated_at that is earlier than a watermark value the pipeline had already advanced past in a previous run.

1. Explain precisely why this specific class of row is permanently lost rather than merely delayed, tracing through the pipeline logic above.
2. Propose a fix, including any change to the downstream write logic that your fix requires to remain correct.
3. Your fix introduces the possibility of re-pulling some rows that were already successfully loaded in a previous run. Is that a problem? Why or why not?