Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Choosing a Format for a New Event Pipeline, End to End

You're designing a pipeline for a new checkout_completed event: a web/mobile client emits the event, it flows through Kafka, and is eventually queried by analysts and a nightly revenue-reporting job. A teammate proposes using Parquet as the wire format on the Kafka topic itself, "since it's the best format and we should standardize on one format everywhere to keep things simple."

  1. Evaluate the proposal to use Parquet as the Kafka wire format. Would it technically work? What would go wrong or be inefficient?
  2. Propose the actual end-to-end format choice across the pipeline (wire format on Kafka, and storage format once landed in the lake), and justify each choice by workload shape rather than by "it's the best format."
  3. The checkout_completed schema will need to add an optional promo_code field a few months from now, after several other teams are already consuming the topic. What has to be true of that change for it to roll out safely?

Share this question

← Back to File Formats, Partitioning & Storage Layout practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.