Advanced
Open
Pro
Design an Experiment Assignment Service with Layers, Salts and Holdouts
You are designing the assignment service for a company with 20 million users and about 50 concurrent experiments. Requirements: the same user must always get the same arm; two experiments that edit the same checkout page must never overlap; unrelated experiments should overlap freely to use traffic; a 2 percent global holdout must be excluded from all launches; ramps from 1 percent to 50 percent must not reshuffle users already assigned; assignment must be auditable; and data scientists must be able to analyse only the users who actually reached the feature.
- Specify the assignment algorithm (hashing, buckets, salts).
- Specify layers and mutual exclusion.
- Explain how ramping, holdouts and re-use of users work.
- Specify exposure logging and which analysis population to use.
- List the automatic health checks the platform should run.
Share this question