Define a North Star, Driver Metrics and Guardrails for a Grocery-Delivery Subscription
You join a grocery-delivery company that sells a monthly membership (flat fee, free delivery above a minimum basket). Leadership says "our metric is total orders". Growth is strong, but finance reports that membership churn is rising and support tickets per order are up. You are asked to propose how the product will be steered.
- Propose a north-star metric and justify it against "total orders". State its exact definition (grain, window, what counts as active).
- Draw the metric tree one level down: name 4 to 5 driver metrics and the team that would own each.
- List 4 guardrail metrics and say which gaming move each one catches.
- Orders per member is hard to move within a quarter and 12-month retention takes a year. Propose a proxy for the experiment OEC and describe how you would validate it before trusting it.
1. North star. "Total orders" is a volume metric that goes up with discounts, a bigger member base and low-margin one-item baskets, and it cannot tell a loyal member from a one-time promotion buyer. A better north star measures durable value delivered to paying members, for example weekly active members: paying members who placed at least one order in the trailing 7 days, optionally restricted to orders that meet the free-delivery minimum. Grain is the member (not the order or device), window 7 days in the member's local time zone, and "active" means a completed and delivered order (not an app open, which push notifications can inflate). It rises when we acquire, convert, engage and retain, and it correlates with revenue because the fee is paid by members who stay.
2. Tree. Weekly active members = retained + newly converted + resurrected. Drivers: (a) new member sign-ups per week (growth/marketing team); (b) trial-to-paid conversion and first-order activation within 7 days (onboarding/growth product); (c) orders per active member per month, driven by basket composition, assortment and substitution quality (catalog/merchandising); (d) monthly member churn, split into month-1 churn and later churn (retention/lifecycle team); (e) resurrection of lapsed members (CRM). Each branch has one owner, so a movement in the north star can be walked down to one team.
3. Guardrails. Each is paired with the gaming move it catches: on-time delivery rate and order defect rate (catches pushing volume through a stressed logistics network); refund and substitution-complaint rate (catches aggressive substitutions that keep baskets large but upset members); support contacts per 100 orders (the failure finance already sees); notification opt-out and unsubscribe rate (catches hitting order targets by spamming reminders); optionally contribution margin per order (catches buying orders with discounts).
4. Proxy and validation. Use something fast that predicts the slow outcome, for example member still active in weeks 4 to 5 after joining (a bracketed retention window, not "active on day 28 exactly") or "three or more orders in the first 30 days". Validate on history: take past cohorts and past experiments and check (a) that the proxy level correlates with 12-month retention, and, more importantly, (b) that the treatment effects on the proxy across past experiments line up with their effects on the long-run outcome; a proxy that predicts retention across users can still fail to carry a treatment's effect (for example a discount raises early orders without improving long-run retention). Re-validate when the audience changes, and keep a holdout or a long-running guardrail to catch drift. Pre-register the OEC before the experiment, and report the proxy together with the guardrails.
Share this question