Recently I wrote a blog posting for Broad River Rehab, my employer, about the non-therapy ancillary (NTA) payment of the new Patient Driven Payment Model (PDPM). After going through the way the payment will work, I used ANOVA to see if there were any significant differences in the payments to urban facilities versus rural. I used this chart: (click to enlarge)

That analysis showed that that there is a statistically significant difference between the means which is likely around 0.04118. The minimum difference using the 95% confidence intervals was 0.03296.

This left me with several questions:

The difference in means is statistically significant but is it significant from the perspective of a nursing home operator? That's a higher bar.
If we choose a level of significance that matters to a typical nursing home operator, would a Bayesian analysis approach be better than ANOVA?
If the difference in the means IS actually large enough that an operator would care about it, what might be causing the difference?

These questions are a a little too esoteric for the Broad River blog, but perfect for my mine.

Let's get started!

When is a difference significant?

Case-mix index differences are often meaningless to people: is a change of 0.02 in case-mix index a big deal? 0.2? People are much more able to gauge changes when they are translated to dollars. So let's re-frame the the question in terms of dollars: How much does your Medicare Part A reimbursement have to change before it gets your attention?

Most administrators and CFOs would notice some change in Part A reimbursement. I am going to say a change of $1000 per month in one facility would start to get attention. That's $12,000 per year. Unfortunately, for some homes that's a very significant amount.

We'll use the following formula to calculate pay in a case-mix situation:

We know Pay is going to be $1000 and we'll use 30 for days in month.

We know the urban and rural rates for NTA, $78.05 and $74.56 respectively. We'll use the rural rate because it is smaller and will require a larger change in case-mix index to get a $1000 monthly change. We'll also know that if a case-mix change is significant enough to get $1000 monthly in a rural setting, it will be more than that in an urban setting.

The last piece we need is the number of patients. We'll get that from the data CMS provided with the PDPM proposal. As you can see in the histogram below, the number of Med A patients in a a facility is a non-normal distribution. We could use the mode because that is what is most likely to happen and isn't skewed by outliers. The mode for rural facilities is only 2 Med A patients. For urban facilities it is 5 Med A patients. That is surprisingly low. The overall mode is 3.

We could also use the median because half of the population falls on either side. The medians for urban, rural and overall are 9, 5 and 8 respectively. I think using the overall median is a good compromise here.

Now we have enough numbers to solve for the CMI change:

So the answer is a change of 0.0559 is large enough that most facilities would notice it. Our ANOVA analysis said we had a difference of 0.0412, which is less less than our threshold that most people would notice or care. So while it may be statistically significant, it isn't practically significant.

Would Bayesian analysis have done a better job?

If you read this blog then you know I am a fan of Bayesian analysis over traditional null hypothesis testing (NHT). There are a lot of reasons to like Bayesian testing, but the two I am most interested in here are that I can make the test more robust against outliers and I think the results are easier to understand. Let's take a look.

We can use all 15,000 facilities to create a thousands of models and then look at see what the most common models look like. This can be a long process when compared to just running an ANOVA. We'll use the same boundary we created to evaluate our ANOVA to create a "Region of Practical Equivalence" or ROPE for the analysis. Then we can see if the most likely model falls inside or outside the ROPE.

I won't go into the details of doing the Bayesian analysis here because you probably don't care, but if you do, check out this book.

Here are the results of that analysis. The most likely difference between urban and rural case-mix due to NTA is -0.0339. We would expect the difference between rural and urban to be -0.0383 and -0.0289 95% of the time. We would never expect the difference to be less than -0.0559, ever. So our Bayesian analysis shows the difference between the two populations is even less than the ANOVA. This is likely because the ANOVA assumes the populations are normal and for the Bayesian analysis I was able to use a students T distribution which is better about handling outliers.

Possible causes?

Both ANOVA and Bayesian analysis agree: the populations are definitely different but that difference is small. Now we are left to speculate about why that difference exists. Why would we do this you ask? Because understanding the reason for something like this might lead to a discovery that is more meaningful. We should never stop being curious and trying to learn.

So here are a couple of ideas:

The difference could be due to some educational component. Perhaps the rural facilities under-code certain items. For some this theory is easy to get on board with due to natural biases we might have regarding the word rural. I find this theory extremely unlikely. Why? Because of the relative uniformity of the difference across the country. There are rural areas in almost every state. It seems very unlikely that far-flung areas of the US would all be making this similar mistake. Unfortunately this theory will always be a theory because it is probably impossible to test.
A second theory is perhaps people in rural areas with serious medical conditions travel to urban areas for treatment. We'll call this "Medical Migration". I like this theory. Not just because I have heard anecdotal information to support it, but because it is possible to test. Several of the RESDAC data sets have the information we need to figure this out. (I don't currently have access to MedPAR or the LDS data.)

If you have access to RESDAC data and want to collaborate, let me know. Or if you just want to talk shop, I'm up for that too.