Intermediate
Open
Pro
Choosing Between Query-Based and Log-Based CDC
You're designing ingestion for two source tables into the same warehouse:
product_catalog: ~50,000 rows, updated a few dozen times a day by internal staff, no deletes (products are markeddiscontinued, never removed). Downstream consumers (search indexing, the product page) tolerate up to 15 minutes of staleness.payment_events: tens of millions of rows/day, a small percentage are corrected or reversed within minutes of creation (and the correction involves deleting the original erroneous row, not just updating it), and a fraud-detection service downstream needs to see both new events and deletions within seconds.
A junior engineer proposes standing up Debezium log-based CDC for both tables, reasoning "it's strictly more capable, so let's just use it everywhere and be consistent."
- For each table, say whether you'd use query-based (timestamp) CDC or log-based CDC, and justify each choice against the table's actual requirements rather than "more capable is always better."
- What operational cost is the team taking on if they follow the
junior engineer's "use it everywhere" proposal, specifically for
product_catalog? - Six months later,
product_catalogrequirements change: the business wants true hard deletes when a product line is discontinued and fully removed. Does this change your answer for that table? Explain the decision process, not just the new answer.
Share this question