Explain TreeSHAP additivity to a risk committee and answer the "so we should raise income?" question

(a) Additivity. The attributions add up exactly to the difference between this prediction and the baseline (the efficiency axiom, called local accuracy in the SHAP paper): f(x)=\text{base}+\sum_j\phi_j. Here 0.99+0.56-0.17-0.18-0.22-1.62-0.01=-0.65 in log-odds. The probability is 1/(1+e^{0.65})\approx 0.34. In this lesson's run, the brute-force values summed to -0.652 against a model output of -0.652 (verified), and the probability was 0.343. Two further points: the unit is log-odds, so attributions add there and not in probability; and there is a second sense of additivity, the linearity axiom, under which the SHAP values of an ensemble are the sum of those of its trees, which is why tree ensembles are handled tree by tree.

(b) How TreeSHAP works. The Shapley formula averages marginal contributions over all 2^{d} feature subsets. For a decision tree, the expected output given a subset of known features can be computed by following both branches at splits on unknown features, weighted by training cover, and TreeSHAP (Lundberg et al., 2018 and 2020; from memory) organizes this recursion so that the whole sum is computed in polynomial time, roughly linear in the number of leaves times depth squared, instead of exponential. The two variants are path-dependent (the value function uses the trees' own cover statistics) and interventional (unknown features are drawn from a background dataset). They define different games: on this applicant, interventional values matched my brute-force enumeration to 3e-9, while path-dependent values differed (income +0.576 against +0.560; neighbourhood score +0.041 against -0.010; verified). When features are correlated, the interventional version can evaluate the model on unrealistic combinations, and the path-dependent version can spread credit onto correlated features.

(c) Not causal. A SHAP value describes how the model uses a feature to move this prediction relative to the baseline. It says nothing about what would happen to the person if the feature changed: the model learned an association from observational data, and stated income can be unverified, correlated with other income documents, or a proxy for something else. Raising a reported income might be misrepresentation, would not change the person's repayment capacity, and would break the model's assumptions once applied at scale. The right questions are causal (Causal Inference Basics) or about recourse: which actionable, verifiable changes flip the decision (in the example, a debt ratio of 0.196 or a utilization of 0.093 would; both are model facts, not promises), and whether the lender is willing to commit to that advice after retraining.

(d) The baseline is part of the explanation. With approved applicants as the background, the base value rose from 0.992 to 1.295, and the late-payments attribution became -1.699 instead of -1.624, the income attribution fell from +0.560 to +0.392 (verified). The ordering of reasons stayed the same here, but it need not: "compared with whom" changes the attribution. For adverse-action reason codes, fix the baseline, document it, version it with the model, and run a sensitivity check on the ordering of the top four reasons before the compliance team signs off.