Defining a Metric for a Fuzzy Feature Request from Scratch
Your VP asks: "We just launched in-app commenting on shared documents. I want to know if it's actually working — figure out how we'd measure that." There's no existing dashboard for this feature, and the VP hasn't defined what "working" means.
- Pick the HEART category (or categories) most relevant to this request and justify the choice.
- Walk through the Goals-Signals-Metrics process to turn this into one or two specific, instrumentable metrics.
- Name one guardrail metric you'd track alongside your primary metric, and explain what it would catch if the primary metric improved for the wrong reason.
- The VP says "just tell me engagement went up." Why is that answer insufficient, and what would you say instead?
1. Relevant HEART category(ies)
Adoption is the primary fit: the feature is new, and the first question is whether users are starting to use it at all. Task success is a close second — once someone starts a comment thread, whether the interaction itself works cleanly (can they find the right place to comment, resolve a thread, tag a collaborator without errors) is a separate question from whether they tried it once. Engagement (frequency of use over time) matters too, but only after Adoption and Task success establish that the feature works at all — leading with Engagement risks measuring how much people use something before confirming they can use it correctly.
2. Goals-Signals-Metrics
- Goal: users who are actively collaborating on a shared document use commenting to communicate about it, instead of falling back to email or a separate chat tool.
- Signal: a user who is an active collaborator on a document (has edited it, or is listed as a shared editor) creates or replies to a comment on that document within a session.
- Metric: % of documents with 2+ collaborators that have at least one comment thread within 7 days of being shared, and, separately, % of comment threads that receive a reply from a different collaborator within 48 hours (a proxy for the feature actually facilitating back-and-forth, not just one-off notes).
3. A guardrail metric
Comment-thread abandonment rate — % of comment threads that are started but never resolved or replied to. If the primary "documents with a comment thread" metric goes up but this guardrail also spikes, it would suggest people are starting to use commenting but hitting a dead end (can't find how to notify the right person, notifications aren't working, the thread UI is confusing) — the primary metric would look like a win while the feature is actually failing at the task-success level.
4. Why "engagement went up" is insufficient
"Engagement went up" doesn't say whether the feature works, whether it's actually replacing the workaround (email, external chat) it was meant to replace, or whether it's healthy usage vs. people starting threads that go nowhere. A better answer names the specific metric and what it implies: "38% of multi-collaborator documents got a comment thread in the first week, and 71% of those threads got a reply within 48 hours — so it's being adopted and it's facilitating real back-and-forth, not just one-off notes. Thread-abandonment rate is at 12%, which we're treating as a guardrail to watch." That answer lets the VP make a real judgment call; "engagement went up" doesn't.
Share this question