Debugging a Concurrent-Write Conflict on an Iceberg Table

Your platform runs two independent pipelines writing to the same Iceberg table on S3: a streaming job that appends new events roughly every 30 seconds, and a nightly batch job that runs a MERGE to deduplicate and backfill late-arriving records. One night, the batch job's MERGE fails with a commit conflict / concurrent-modification error, and the on-call engineer isn't sure whether this indicates data corruption.

1. Explain what is actually happening mechanically when this conflict occurs, and why it is a sign the system is working correctly rather than a sign of corruption.
2. Contrast what would have happened if this same two-writer situation occurred on a plain data lake (raw Parquet files, no table format) instead of an Iceberg table.
3. Propose a concrete operational fix so this conflict doesn't repeatedly interrupt the nightly batch job.