Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Design an Experiment Assignment Service with Layers, Salts and Holdouts

You are designing the assignment service for a company with 20 million users and about 50 concurrent experiments. Requirements: the same user must always get the same arm; two experiments that edit the same checkout page must never overlap; unrelated experiments should overlap freely to use traffic; a 2 percent global holdout must be excluded from all launches; ramps from 1 percent to 50 percent must not reshuffle users already assigned; assignment must be auditable; and data scientists must be able to analyse only the users who actually reached the feature.

  1. Specify the assignment algorithm (hashing, buckets, salts).
  2. Specify layers and mutual exclusion.
  3. Explain how ramping, holdouts and re-use of users work.
  4. Specify exposure logging and which analysis population to use.
  5. List the automatic health checks the platform should run.

Share this question

← Back to Advanced Experimentation: Interference, Marketplaces & Platforms practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.