Product Analytics, Metrics & LTV
This subject assumes basic SQL and descriptive statistics. The boundary: the statistics of experiments (power, SRM, CUPED, sequential tests) belong to ab-testing; LTV as a reinforcement-learning objective and delayed-reward design belong to long-term-value-and-delayed-reward-systems; UX measurement frameworks such as HEART belong to design-metrics-and-product-thinking. Here we cover definitions, computation and pitfalls: what a metric means exactly, how it is computed from raw events, and the ways a perfectly computed number still misleads.
Interviews ask this as "how would you measure the success of X?", "retention fell but DAU rose, why?" or "compute LTV for this cohort". The senior answer is precise definitions first (what counts as active, over which window, per user or per event), then a breakdown that finds which part of the number moved. Everything below that is executable was run (Python 3 with NumPy, fixed seeds) and is marked "(verified)". Rules of thumb quoted from industry practice (quick ratio above 4, LTV/CAC around 3) are folklore recalled from memory, not measured here; treat them as starting points to compare with your own business.
The Metric Hierarchy
A product is steered by a small hierarchy, not a pile of numbers:
| Layer | Question it answers | Example (subscription learning app) |
|---|---|---|
| North star | Is the product creating durable value for users that the business can monetize? | Weekly active subscribers who completed at least one lesson |
| Driver (input) metrics | Which levers move the north star, and who owns each lever? | Trial starts, activation rate, trial-to-paid conversion, lessons per active subscriber, monthly paid churn |
| Guardrail metrics | What must not get worse while we optimize the drivers? | Refund rate, crash-free sessions, p95 lesson load time, support contacts per 1,000 subscribers, notification opt-out rate |
| OEC (overall evaluation criterion) | The single combined score an experiment is judged on | A weighted, pre-registered blend or one agreed proxy, for example week-4 retained subscribers |
Leading versus lagging. Revenue, annual retention and LTV are lagging: they confirm what already happened, months later. Activation, first-week habit formation and week-4 retention are leading: they arrive fast and predict the lagging ones. You steer by leading metrics and audit by lagging ones. A metric tree makes the decomposition explicit so every team can point at the branch it owns:
graph TD
NS["North star: weekly active subscribers (WAS)"] --> ACQ["Acquisition: new trials per week"]
NS --> CONV["Conversion: trial-to-paid rate"]
NS --> ENG["Engagement: lessons per active subscriber"]
NS --> RET["Retention: monthly paid churn"]
ACQ --> A1["Traffic x signup rate"]
ACQ --> A2["Referral and partner trials"]
CONV --> C1["Activation: first lesson within 24h"]
CONV --> C2["Paywall and price test results"]
ENG --> E1["Streak and reminder reach"]
ENG --> E2["Content breadth: courses started"]
RET --> R1["Month-1 churn (largest hazard)"]
RET --> R2["Resurrection of lapsed subscribers"]
NS -.-> G["Guardrails: refunds, crashes, p95 latency, support load, opt-outs"]
Read the tree multiplicatively where you can: weekly active subscribers = existing active subscribers retained + newly converted + resurrected. That identity (growth accounting, below) is what turns a tree into diagnosis: when the north star moves, you walk down the branches until one explains the change.