Chapter 6 Story MEPS: Explainable predictions for healthcare expenditures

Authors: Anna Kozioł (Warsaw University of Technology), Katarzyna Lorenc (Warsaw University of Technology), Piotr Podolski (University of Warsaw)

Mentors: Maciej Andrzejak (McKinsey & Company), Alicja Jośko (McKinsey & Company)

6.1 Introduction

Perhaps the most urgent problem with the current health care system in the United States is its high cost. According to the Centers for Disease Control and Prevention, during 2017 health care spending per capita averaged nearly $11,000 and total spending was $3.2 trillion, or 17.9% of GDP. This raises the natural question of the causality of high expenses and the estimation of them for a particular person.

One of the objectives of this chapter is to forecast annual spending on the health care of individuals in the United States.

There is no doubt that these forecasts are of interest to people directly related to medical expenditure, for example, insurance companies, employers, government.

How to deal with a situation when the model works well but is a so-called black box and we do not know what affects a specific result? What if the proposed models return non-intuitive results and we want to know why they are wrong?

The next and main purpose of this chapter is to address these concerns using Explanatory Model Analysis.
We will try to identify not only which features are most predictable for the results, but also the nature of the relationship (e.g. its direction and shape). We will focus on understanding the behavior of the model as a whole, as well as in a specific instant level (for specific person).

The data set comes from a study called Medical Expenditure Panel Survey (MEPS), which is sponsored by the Healthcare Quality and Research Agency. About 15,000 households are selected as a new panel of surveyed units, regularly since 1996.

Data set used for analysis is available for free on the MEPS website. The MEPS contains a representative sample of the population from the United States with two major components: the Household Component and the Insurance Component.

Household Component collects data about demographic characteristics, health conditions, health status, medical history, fees and sources of payment, access to care, satisfaction with care, health insurance coverage, income, and employment for each person surveyed.

The second component - insurance - collects data about the health insurance from private and public sector employers. The data include the number and types of private insurance schemes offered, premiums, employers’ and employees’ health insurance contributions, benefits associated with these schemes, and employer characteristics.

The data processing and analysis were carried out in Python 3.7.3 and R 3.6.1.

6.2 Model

6.2.1 Data

Agency of Healthcare Research and Quality provides an extensive database of medical expenses. Consequently, dataset selection on which we will make further analysis was an important first step. We decided to choose the two latest panels. Expenditures for treatment that we will examine in the following chapter apply to the years 2015/2016 and 2016/2017. The selected dataset contains information on over 32,000 patients, and each of them is described by 3,700 variables. We attached great importance to choosing features that would be appropriate for the prediction. The most important criterion adopted is that the variable cannot relate to expenditure associated with any treatment. For this purpose, we looked through several hundred of them and selected 387 most suitable.

As a part of the preprocessing, we removed records that were marked as Inapplicable in the expenditure column. The number of people who didn’t incur expenses is 5504, while the number of patients with “inapplicable” is 407, the percentage respectively are 17% and 1%. The following figures show the distribution of the explained variable.

FIGURE 6.1: Distribution of medical expenses

6.3 Model

6.3.1 Data

Agency of Healthcare Research and Quality provides an extensive database of medical expenses. Consequently, data set selection on which we will make further analysis was an important first step. We decided to choose the two latest panels. Expenditures for treatment that we will examine in the following chapter apply to the years 2015/2016 and 2016/2017. The selected dataset contains information on over 32,000 patients, and each of them is described by 3,700 variables. We attached great importance to choosing features that would be appropriate for the prediction. The most important criterion adopted is that the variable cannot relate to expenditure associated with any treatment. For this purpose, we looked through several hundred of them and selected 387 most suitable.

As a part of the preprocessing, we removed records that were marked as Inapplicable in the expenditure column. The number of people with such markings was 407, which is about 1% of the whole data set. The following figures show the distribution of the explained variable.

FIGURE 6.2: Distribution of medical expenses

6.3.2 Model

Among the models we have trained, the best results were achieved by Gradient Boosting. Due to the characteristics of the explained variable, we decided to check the behavior of the model after applying the logarithmic transformation to expenses. In the case of modelling the expenditure logarithm, we also used translations by 1 to avoid undesirable values of the variable. We used NNI toolkit to find the best hyperparameters To choose the best model, we compared the determination coefficient values.

The table below shows the results of the experiments. To calculate the determination coefficient in column \(R^2\) (log), we transformed logarithmically the values of expenses, and after training the model we returned to the original scale.

Values of the determination coefficient.

Model	\(R^2\)	\(R^2\) (log)
Gradient Boosting	0.43	0.44
Tuned Gradient Boosting	0.49	0.46

The best fit relying on the determination coefficient was demonstrated by a Gradient Boosting. Then, as a compromise between the size of the model and its quality, we chose the 7 most important variables. For this purpose, we ranked the significance of the variables in the model and extracted those with the highest coefficient.

Obtained variables mainly concern the number of visits to specialists. For a more diverse and interesting analysis, we have also taken into account demographic variables such as age, gender, educational background, and race, as well as some disease units.

FIGURE 6.3: Scheme of conduct with a specification of origin of variables

A review of selected variables

Variable	New Name	Description
IPNGTDY1	hospital_nights	number of nights associated with hospital discharges
OBDRVY1	phys_visits	number of office-based physician visits
HHAGDY1	home_days	agency home health provider days
DSFTNV5	feet_checked	indicate whether the respondent reported having his or her feet checked for sores or irritations
OBOTHVY1	non-phys_visits	office-based non-physican visits
PSTATS2	disposition	person disposition status
OPOTHVY1	outpatient_visits	outpatient dept non-dr visits
AGE2X	age	age of patient
RACEV2X	race	race of patient
SEX	sex	patient’s gender
HIDEG	edu	the highest degree of education attained at the time the individual entered MEPS
diab_disease	diab_disease	indicates whether the patient suffered from a diabetes disease
art_disease	art_disease	indicates whether the patient suffered from a arthritis disease
ast_disease	ast_disease	indicates whether the patient suffered from a asthma disease
press_disease	press_disease	indicates whether the patient suffered from a high pressure disease
heart_disease	heart_disease	indicates whether the patient suffered from a heart disease

In the following section we will explain the Gradient Boosting model based on 16 variables presented in the table above. The coefficient of determination of the final model is 0.5

6.4 Model Level Explainations

6.4.1 Permutation Variable Importances

In order to find out about the influence of individual variables on the prediction for each patient, we present a Permutation Variable Importances graph.

FIGURE 6.4: Permutation Variable Importances for Gradient Boosting Regressor

In the figure 6.4 we present the 6 most relevant variables. Features for which the median of Permutation Variable Importances was less than 0.01 were omitted. Undoubtedly, the most important is the variable that indicates the number of nights spent in the hospital by the patient. An interesting observation seems to us that the demographic variable - age, which initially did not have a significant impact on the prediction, on the reduced model is in the top five most important variables. The remaining demographic variables, as well as those relating to diseases, do not show a gain in relevance in a model reduced to several variables.

6.4.2 Partial Dependence Profiles

Based on previous analyzes, the number of nights spent in the hospital turned out to be the most important variable. To understand the nature of its impact on prediction in our model, it’s worth looking at the Partial Dependence Profiles. Below we present the PD plots broken down by gender.

FIGURE 6.5: Partial Dependence Profiles for number of nights spent in the hospital broken down by gender

Among patients who spent a few nights in the hospital, gender is not important for the amount of prediction. This rule begins to change after exceeding 30 nights. The PD profile for men has significantly higher values compared to the women’s profile, despite the fact that variable age doesn’t show much influence on the model. After exceeding 70 nights in the hospital, this variable does not affect the result returned by the model on average.

6.5 Instance Level Explanations

6.5.1 Business approach

In this subsection we will try to show the application of explanatory methods in the business approach. Selected observations are:

1. the person with the best estimated cost among people with results greater than 3000,
2. the person for whom the model predicted the highest cost of all.

Finding the value of characteristics that increase or decrease the final result, diagnostics of the direction of changes or oscillations of the result in case of change of characteristics describing a person may be valuable information for insurance companies or other payers for medical services. Such conclusions may also be useful for the patients, who have decided to pay for medical care themselves. In further consideration, the selected observations will be called Patient 1 and Patient 2.

Values of explanatory variables for patients.

Variable	Patient 1	Patient 2
hospital_nights	0	52
phys_visits	4	6
home_days	0	0
feet_checked	-1-inapplicable	-1-inapplicable
non-phys_visits	1	0
disposition	household	household
outpatient_visits	0	1
age	58	59
race	1-white	1-white
sex	2-female	1-man
edu	other	bachelor’s degree
diab_disease	1-yes	0-no
art_disease	1-yes	1-yes
ast_disease	0-no	1-yes
press_disease	0-no	0-no
heart_disease	1-yes	1-yes
real expenses	3882$	143457$
prediction	3886.8$	147178.5$

6.5.1.1 XIA for the best prediction

• XIA for the best prediction using Break Down Plots

Break-down plots show how the contribution of individual variables change the average model prediction to the prediction for observation.

The patient has 58 age, which alone increase average total cost by 574.177$ and the gender is female which decreases average total cost by 140. 72$. Her total number of office-based visits is 4, which increase average total cost by 574.177$. She suffers from arthritis what increase average total cost by 677.868$ but she is not diagnose to diabets, astma or high blood preasure which decrease final result. Her status of education is unknown, what increase total cost. The fact that she didn’t spend any night in the hospital decrease average total cost by 1343.271$. And also, she didn’t benefit from home medical services decrease average total cost by 386.772$.

• XIA for the best prediction using Shapley Values

To remove the influence of the random ordering of the variables in brake down results we can compute an average value of the contributions.