Compute and Interpret a Two-Sample t-Test
A support team trials a new ticket-triage tool. Resolution time in hours for two independent groups of tickets:
| Group | n | mean | SD |
|---|---|---|---|
| Old tool | 30 | 8.6 | 2.4 |
| New tool | 36 | 7.4 | 1.8 |
- State the hypotheses and compute Welch's t statistic and the approximate degrees of freedom. Is the result significant at \alpha = 0.05 (two-sided)? (Use t_{0.975} \approx 2.00.)
- Give a 95% confidence interval for the difference and Cohen's d. In one or two sentences, what should the team be told?
- Why is Welch's version preferable to Student's pooled t here?
1. Test statistic and df
H_0: \mu_{old} = \mu_{new}, H_1: \mu_{old} \ne \mu_{new}.
Standard error: \text{SE} = \sqrt{2.4^2/30 + 1.8^2/36} = \sqrt{5.76/30 + 3.24/36} = \sqrt{0.192 + 0.090} = \sqrt{0.282} = 0.531.
t = (8.6 - 7.4)/0.531 = 1.2/0.531 = 2.26.
Welch–Satterthwaite df: \dfrac{0.282^2}{0.192^2/29 + 0.090^2/35} = \dfrac{0.0795}{0.001271 + 0.000231} = \dfrac{0.0795}{0.001502} \approx 53.
With df ≈ 53 the two-sided critical value is about 2.00; t = 2.26 exceeds it, so p < 0.05 (about 0.03). Reject H_0: the observed 1.2-hour reduction is unlikely under "no difference".
2. CI, effect size, and the message
95% CI: 1.2 \pm 2.00 \times 0.531 = 1.2 \pm 1.06, i.e. (0.14, 2.26) hours. Pooled SD: s_p = \sqrt{(29 \cdot 5.76 + 35 \cdot 3.24)/64} = \sqrt{(167.0 + 113.4)/64} = \sqrt{4.38} = 2.09, so Cohen's d = 1.2/2.09 \approx 0.57 — a medium effect.
Message: "The new tool reduced average resolution time by about 1.2 hours (95% CI 0.1 to 2.3 hours; p ≈ 0.03). The improvement is statistically detectable and moderate in size, but with 66 tickets the true saving could plausibly be anywhere from negligible to over two hours — a larger trial would pin it down." Reporting the interval, not just the star, is what makes this an honest summary.
3. Why Welch
The sample SDs differ (2.4 vs 1.8, variance ratio ≈ 1.8) and the group sizes are unequal (30 vs 36). Student's pooled t assumes equal variances; when that fails with unequal n the test's actual Type I error rate can drift away from the nominal 5%. Welch estimates each group's variance separately and adjusts the degrees of freedom, so it is correctly calibrated whether or not variances match, and it loses almost no power when they do. There is no gain from running an equal-variance pre-test first; just use Welch.
Share this question