# 9 Shapley Additive Explanations (SHAP) and Average Variable Attributions

In Chapter 7, we introduced Break-down (BD) plots, a method of assessment of local variable-importance based on the contribution of an explanatory variable to model’s prediction. We also indicated that, in the presence of interactions, the computed value of the contribution depends on the order of explanatory covariates that is used in calculations. One solution to the problem is to find an ordering in which the most important variables are placed at the beginning. Another solution, described in Chapter 8, is to identify interactions and explicitly present their contributions to the predictions.

In this chapter, we introduce yet another approach to address the ordering issue. It is based on the idea of averaging the value of a variable’s contribution over all, or a large number of, possible orderings. The idea is closely linked to ,,Shapley values’’ (Shapley 1953), developed originally for cooperative games.

The approach was first introduced in ,,An Efficient Explanation of Individual Classifications Using Game Theory’’ (Štrumbelj and Kononenko 2010) and (Štrumbelj and Kononenko 2014). It was widely adopted after the publication of the NIPS paper ,,A Unified Approach to Interpreting Model Predictions’’ (Lundberg and Lee 2017) and Python’s library SHAP (Lundberg 2019). The authors of SHAP (SHapley Additive exPlanations) introduced an efficient algorithm for tree-based models (Lundberg, Erion, and Lee 2018). They also showed that SHAP values can be presented an unification of a collection of different commonly used techniques for model explanations (Lundberg and Lee 2017).

## 9.1 Intuition

Figure 9.1 presents BD plots for ten random orderings (indicated by the order of the rows in each plot) of explanatory variables for the prediction for `johny_d`

(see Section 5.1.6) for the random-forest model (see Section 5.1.3) for the Titanic dataset. The plots show clear differences in the contributions of various variables for different orderings. The most remarkable differences can be observed for variables `fare`

and `class`

, with contributions changing the sign depending on the ordering.

To remove the influence of the ordering of the variables, we can compute an average value of the contributions.
Figure 9.2 presents the average contributions, calculated over the ten orderings presented in Figure 9.1. Red and green bars present, respectively, the negative and positive averages. Violet box-plots summarize the distribution of the contributions for each explanatory variable across different orderings. The plot indicates that the most important variables, from the point of view of the prediction for `johny_d`

are `age`

and `gender`

.

## 9.2 Method

SHapley Additive exPlanations (SHAP) are based on ,,Shapley values,’’ a concept in cooperative game theory developed by Lloyd Shapley (Shapley 1953). Note that the notation may be confusing at the first glance. Shapley values are introduced for cooperative games. SHAP is an acronym for a method designed for ML models. We will use the name Shapley values.

Consider the following problem. A coalition of players cooperates, and obtains a certain overall gain from the cooperation. Players are not identical, and different players may have different importance. Cooperation is beneficial, because it may bring more benefit than individual actions. The problem to solve is how to distribute the generated surplus among the players? The Shapley value provides one possible fair answer to this question (Shapley 1953).

Now let’s translate this problem to the context of model predictions. Explanatory variables are the players, while model \(f()\) plays the role of the coalition. The payoff from the coalition is the model prediction. The problem to solve is how to distribute the model prediction across particular variables?

The idea of using Shapley values for evaluation of local variable-importance was introduced in ,,An Efficient Explanation of Individual Classifications Using Game Theory’’ (Štrumbelj and Kononenko 2010). We define them here in the notation introduced in Section 7.2.

Let us consider a permutation \(J\) of the set of indices \(\{1,2,\ldots,p\}\) corresponding to an ordering of \(p\) explanatory variables included in model \(f()\). Denote by \(\pi(J,j)\) the set of the indices of the variables that are positioned in \(J\) before the \(j\)-th variable. Note that, if the \(j\)-th variable is placed as the first, then \(\pi(J,j) = \emptyset\). Consider the model prediction \(f(x_*)\) for a particular instance of interest \(x_*\). The Shapley value is defined as follows: \[\begin{equation} \varphi(x_*,j) = \frac{1}{p!} \sum_{J} \Delta^{j|\pi(J,j)}(x_*), \tag{9.1} \end{equation}\] where the sum is taken over all \(p!\) possible permutations (orderings of explanatory variables) and the variable-importance measure \(\Delta^{j|J}(x_*)\) was defined in Section 7.2. Essentially, \(\varphi(x_*,j)\) is the average of the variable-importance measures across all possible orderings of explanatory variables.

It is worth noting that the value of \(\Delta^{j|\pi(J,j)}(x_*)\) is constant for all ordering \(J\) that share with the same subset \(\pi(J,j)\). It follows that equation (9.1) can be expressed in an alternative form:

\[\begin{eqnarray} \varphi(x_*,j) &=& \frac 1{p!}\sum_{s=0}^{p-1} \sum_{ \substack{ S \subseteq \{1,\dots,p\}\setminus \{j\} \\ |S|=s }} \left[s!(p-1-s)! \Delta^{j|S}(x_*)\right]\nonumber\\ &=& \frac 1{p}\sum_{s=0}^{p-1} \sum_{ \substack{ S \subseteq \{1,\dots,p\}\setminus \{j\} \\ |S|=s }} \left[{{p-1}\choose{s}}^{-1} \Delta^{j|S}(x_*)\right], \tag{9.2} \end{eqnarray}\] where \(|S|\) denotes the cardinal number (size) of set \(S\) and the second sum is taken over all subsets \(S\) of explanatory variables, excluding the \(j\)-th one, of size \(s\).

Note that the number of all subsets of sizes from 0 to \(p-1\) is \(2^{p}-1\), i.e., it is much smaller than number of all permutations \(p!\). Nevertheless, for a large \(p\), it may not be feasible to compute the Shapley values from Equations (9.1) nor (9.2). In that case, an estimate based on a sample of permutations may be considered. A Monte Carlo estimator was introduced in ,,Explaining prediction models and individual predictions with feature contributions’’ (Štrumbelj and Kononenko 2014). An efficient implementation of computations of Shapley values was introduced in package SHAP (Lundberg and Lee 2017).

From the properties of Shapley values for cooperative games it follows that, in the context of predictive models, they enjoy the following properties:

- Symmetry: if two explanatory variables \(j\) and \(k\) are interchangeable, i.e., for any set of explanatory variables \(S \subseteq \{1,\dots,p\}\setminus \{j,k\}\) we have got

\[ \Delta^{j|S}(x_*) = \Delta^{k|S}(x_*), \] then their Shapley values are equal:

\[ \varphi(x_*,j) = \varphi(x_*,k). \]

- Dummy feature: if an explanatory variable \(j\) does not contribute to any prediction for any set of explanatory variables \(S \subseteq \{1,\dots,p\}\setminus \{j\}\), that is,

\[ \Delta^{j|S}(x_*) = 0, \]

then its Shapley value is equal to 0:

\[ \varphi(x_*,j) = 0. \]

Additivity: if model \(f()\) is a sum of two other models \(g()\) and \(h()\), then the Shapley value calculated for model \(f()\) is a sum of Shapley values for models \(g()\) and \(h()\).

Local accuracy: the sum of Shapley values is equal to the model prediction, that is,

\[ f(x_*) - E_X[f(X)] = \sum_{j=1}^p \varphi(x_*,j). \]

## 9.3 Example: Titanic data

Let us consider the random-forest model `titanic_rf_v6`

(see Section 5.1.3 and passenger `johny_d`

(see Section 5.1.6) as the instance of interest in the Titanic data.

Box-plots in Figure 9.3 present the distribution of the contributions \(\Delta^{j|\pi(J,j)}(x_*)\) for each explanatory variable of the model for 25 random orderings of the explanatory variables. Red and green bars represent, respectively, the negative and positive Shapley values across the orderings. It is clear that the young age of Johny D results in a positive contribution for all orderings. The Shapley value is equal to \(0.2525\). On the other hand, the effect of gender is in all cases negative, with the Shapley value equal to \(-0.0908\).

The picture for `fare`

and `class`

is more complex, as their contributions can even change the sign, depending on the ordering. While Figure 9.3 presents the Shapley values separately for each of the variables, it is worth noting that, by using the iBD plot in Section 8.3 the pair was identified as one for each an interaction effect was present. Hence, the effect of the variables should not be separated.

In most applications the detailed information about the distribution of variable contributions across the considered orderings of explanatory variables will not be necessary. Thus, one could simplify the plot by presenting only the Shapley values, as in right panel in Figure 9.3. Table 9.1 presents the Shapley values underlying this plot.

Variable | Shapley value |
---|---|

age = 8 | 0.2525 |

class = 1st | 0.0246 |

embarked = Southampton | -0.0032 |

fare = 72 | 0.0140 |

gender = male | -0.0943 |

parch = 0 | -0.0097 |

sibsp = 0 | 0.0027 |

## 9.4 Pros and cons

Shapley values provide a uniform approach to decompose model predictions into parts that can be attributed additively to different explanatory variables. In ,,A Unified Approach to Interpreting Model Predictions’’ (Lundberg and Lee 2017) it is shown that the method unifies different approaches to additive features attribution, like DeepLIFT (Shrikumar, Greenside, and Kundaje 2017), Layer-Wise Relevance Propagation (Binder et al. 2016), or LIME (Ribeiro, Singh, and Guestrin 2016). The method has got a strong formal foundation derived from the cooperative games theory. It also enjoys an efficient implementation in Python, with ports or re-implementations in R.

An important drawback of the Shapley values is that they are additive attributions of variable effects. If the model is not additive, then the Shapley values may be misleading. This issue can be seen as arising from the fact that, in the cooperative games, the goal is to distribute the payoff among payers. However, in the predictive modeling context, we want to understand how do the players affect the payoff? Thus, we are not limited to independent payoff-splits for players.

It is worth noting that, for an additive model, the approaches presented in Chapters 7, 8, and in the current one lead to same variable contributions. It is because for additive models different orderings lead to same attributions. And since Shapley values can bee seen as an average across all ordering it’s an average from identical values.

An important practical limitation of the method is that, for large models, the calculation of the Shapley values is time consuming. However, sub-sampling can be used to address the issue.

## 9.5 Code snippets for R

In this section, we use an `DALEX::variable_attribution()`

function which is a wrapper for `iBreakDown`

R package (Gosiewska and Biecek 2019a). The package covers all methods presented in this chapter. It is available on CRAN and GitHub.
Note that there are also other R packages that offer similar functionality, like `shapper`

(Gosiewska and Biecek 2019b), which is a wrapper for the Python library `SHAP`

(Lundberg 2019), and `iml`

(Molnar, Bischl, and Casalicchio 2018).

For illustration purposes, we use the `titanic_rf_v6`

random-forest model for the Titanic data developed in Section 5.1.3. Recall that it is developed to predict the probability of survival from sinking of Titanic. Instance-level explanations are calculated for a single observation: `henry`

- an 42-year-old passenger that travelled in the 1st class.

`DALEX`

explainers for the model and the `jonhy_d`

data are retrieved via `archivist`

hooks as listed in Section 5.1.8.

```
library("randomForest")
explain_rf_v6 <- archivist::aread("pbiecek/models/6ed54")
library("DALEX")
henry <- archivist::aread("pbiecek/models/e3596")
henry
```

```
## class gender age sibsp parch fare embarked
## 1 1st male 47 0 0 25 Cherbourg
```

We obtain the model prediction for this instance with the help of the `predict()’ function.

`## [1] 0.246`

With the help of function `variable_attribution()`

we can re-create Figure 9.3. The function is applied to the explainer, created with the `explain()`

function from the `DALEX`

package, and a data frame for the instance of interest. Additionally, in the `B=25`

argument we indicate that we want to select 25 random orderings of explanatory variables for which the Shapley values are to be computed. The resulting object is a data frame with variable contributions computed for every ordering. Applying the generic function `plot()`

to the object constructs the plot that includes the Shapley values and the corresponding box-plots.

To obtain a plot with only Shapley values, we can use the `show_boxplots=FALSE`

argument in the `plot()`

function call.

When we compare this plot with the `johny_d`

in Figure 9.3 the largest difference is related to effect of `age`

. Young `johny_d`

has larger than average chances of survival, much larger than 47 years old `henry`

.

The object obtained as a result of the application of function `shap()`

allows to compute other summary statistics beyond the average.

```
## min q1 median
## Random Forest: age = 47 -0.14872225 -0.115577707 -0.077651998
## Random Forest: class = 1st 0.12112732 0.142754644 0.176491618
## Random Forest: embarked = Cherbourg 0.01981876 0.034941097 0.051973267
## Random Forest: fare = 25 -0.03364295 -0.009764386 -0.009519710
## Random Forest: gender = male -0.15592478 -0.130189397 -0.125022202
## Random Forest: parch = 0 -0.02795650 -0.011817399 -0.005084730
## Random Forest: sibsp = 0 -0.03593203 -0.011618034 -0.006115541
## mean q3
## Random Forest: age = 47 -0.076695025 -0.0294435886
## Random Forest: class = 1st 0.170520888 0.1925099683
## Random Forest: embarked = Cherbourg 0.062216294 0.0964676031
## Random Forest: fare = 25 -0.004429615 0.0039927503
## Random Forest: gender = male -0.125052324 -0.1155987766
## Random Forest: parch = 0 -0.007256366 -0.0008337109
## Random Forest: sibsp = 0 -0.008613321 0.0067818305
## max
## Random Forest: age = 47 -0.021961033
## Random Forest: class = 1st 0.246304486
## Random Forest: embarked = Cherbourg 0.109760761
## Random Forest: fare = 25 0.033626643
## Random Forest: gender = male -0.101295877
## Random Forest: parch = 0 0.002820118
## Random Forest: sibsp = 0 0.007650204
```

### References

Binder, Alexander, Grégoire Montavon, Sebastian Bach, Klaus-Robert Müller, and Wojciech Samek. 2016. “Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers.” *CoRR* abs/1604.00825. http://arxiv.org/abs/1604.00825.

Gosiewska, Alicja, and Przemyslaw Biecek. 2019a. “iBreakDown: Uncertainty of Model Explanations for Non-additive Predictive Models.” https://arxiv.org/abs/1903.11420v1.

Gosiewska, Alicja, and Przemyslaw Biecek. 2019b. *shapper: Wrapper of Python Library ’shap’*. https://github.com/ModelOriented/shapper.

Lundberg, Scott. 2019. *SHAP (SHapley Additive exPlanations)*. https://github.com/slundberg/shap.

Lundberg, Scott M., Gabriel G. Erion, and Su-In Lee. 2018. “Consistent Individualized Feature Attribution for Tree Ensembles.” *CoRR* abs/1802.03888. http://arxiv.org/abs/1802.03888.

Lundberg, Scott M, and Su-In Lee. 2017. “A Unified Approach to Interpreting Model Predictions.” In *Advances in Neural Information Processing Systems 30*, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 4765–74. Curran Associates, Inc. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf.

Molnar, Christoph, Bernd Bischl, and Giuseppe Casalicchio. 2018. “iml: An R package for Interpretable Machine Learning.” *Joss* 3 (26). Journal of Open Source Software: 786. https://doi.org/10.21105/joss.00786.

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?: Explaining the Predictions of Any Classifier.” In, 1135–44. ACM Press. https://doi.org/10.1145/2939672.2939778.

Shapley, Lloyd S. 1953. “A Value for n-Person Games.” In *Contributions to the Theory of Games Ii*, edited by Harold W. Kuhn and Albert W. Tucker, 307–17. Princeton: Princeton University Press.

Shrikumar, Avanti, Peyton Greenside, and Anshul Kundaje. 2017. “Learning Important Features Through Propagating Activation Differences.” *CoRR* abs/1704.02685. http://arxiv.org/abs/1704.02685.

Štrumbelj, Erik, and Igor Kononenko. 2010. “An Efficient Explanation of Individual Classifications Using Game Theory.” *Journal of Machine Learning Research* 11 (March). JMLR.org: 1–18. http://dl.acm.org/citation.cfm?id=1756006.1756007.

Štrumbelj, Erik, and Igor Kononenko. 2014. “Explaining prediction models and individual predictions with feature contributions.” *Knowledge and Information Systems* 41 (3): 647–65. https://doi.org/10.1007/s10115-013-0679-x.