In the past I have railed against the methodology used by PEPPER reports to identify "outliers". I won't completely rehash that here other than to say I think the technique is flawed, even when you consider the purpose is to discourage certain behavior.

I don't want to be the guy who just points out the flaws of the system without offering up an alternative that is (hopefully) better. This posting will explore what I think is a better way.

First, we are going to assume that the main purpose of the auditing is two-fold:

We're doing focused auditing. That means we are looking at facilities that stand out from the rest for some reason. (I believe random auditing is necessary as well but that's for another posting.)
We're actually trying to find issues and not just trying to modify behavior of the average facility. We're not trying to scare everyone into doing less therapy. (I've already posted about a way to remove the incentive to provide more and more therapy.)

For this example we are going to use Medicaid data from the Commonwealth of Kentucky. We'll look at every assessment made for May 2015. (There are 15,425 if you're curious.)

What we're trying to do is identify outliers, facilities that run an unusual caseload that can't be explained by random chance. There are a lot of ways we could do this. We are going to use a simple method for two reasons: it's simple to do and it's hard to argue with a simple method.

All we need to do then is look at the case mix index for all the facilities and determine which facilities have CMIs that would be unusual given the Medicaid patients in the state.

Let's get started.

It's tempting to just look at all the CMIs, calculate the standard deviation and say anything more than three standard deviations away from the mean is an outlier. The problem is our distribution isn't normal.

Now some would say the easy thing to do here is normalize the data. In this case I think that might not be the best idea because it's hard to explain to the average person. So that means we need another way.

Here's another way that's still pretty easy to understand:

First, imagine if we took all 15,425 Medicaid patients in the state and wrote each person's RUG on a ping pong ball. We'd end up with a large hopper full of ping pong balls like they use for lotteries. Then we'd turn on our giant mixer and fan (or whatever it is lottery people do). Next a representative from each facility in the state would get a random ball with a name on it. They would continue to get random balls out of the hopper until they have the same number of balls as they do Medicaid residents. Then we'd calculate the CMI based on those random balls. What would happen?

Well, some facilities would get paid more, some less, obviously. Now imagine we did that same lottery 100,000 times. We'd get a distribution of scores. In fact, you'd think that if we did that often enough, we'd eventually see every possible score.

The range of CMIs would be greater for smaller facilities and less for larger facilities. Now we can calculate a standard deviation for all the CMIs for a given facility size and see which facilities fall outside three standard deviations of that. Here's what that simulation actually looks like:

Each blue dot represents a skilled nursing facility. The X axis is the number of Medicaid patients and the Y axis is the CMI. The orange and blue lines represent the 3 standard deviations from the mean CMI calculated during the 100,000 simulated lotteries we described before. The purple and red lines are the maximum and minimum CMI actually calculated during the simulation.

There are a lot of interesting things to see on this chart. The first and most obvious is there are facilities with CMIs outside the simulated maximum and minimum. These facilities have a patient distribution that random chance cannot explain. In other words, they are doing something (or not doing something) that changes the RUG a resident ultimately receives. This isn't an indicator of a problem necessarily but it's an obvious place to start looking.

Pros

This system is more fair to smaller facilities. Notice how the expected deviations for a smaller building are much wider. If you only have 4 Medicaid patients then you'd have to expect a greater variability in your CMI. This technique takes that into account.
Relatively easy to understand: If your facility has a distribution that's impossible to achieve by randomly assigning patients then you are obviously doing something to influence the score. There may be a good reason for it but the auditors can figure that out.
Symmetry: Notice that facilities outside the upper & lower simulations are highlighted. PEPPER professes to work this way but we all know it doesn't. PEPPER saves the red highlighting for things they don't want you to do.

Cons

There are legitimate reasons a facility might have higher CMI than we would expect to see with random assignment. Perhaps your facility has certain geographic advantages like proximity to a hospital or you have a strong relationship with an admission source. Using this technique you'd get extra scrutiny for that.
Facilities with no rehab program would see more audits. This one is interesting. The auditor should be making the argument that given the number or Medicaid patients you have you aren't doing enough therapy. (Can you imagine an auditor making that argument?) Maybe this should be a "pro". Medicaid eligible patients should get the benefits they are entitled to.
Complexity: This technique is (slightly) harder to do than the simple metrics we use now. However, when I did the simulations for this posting I did the lottery I described 20.5 million times and it only took a few minutes on my laptop.

I'm not saying this proposal is perfect, but I do think it's better than the current situation. If you paired this with some clever modifications to the rates and a strong random auditing program I think it would be an improvement.