From Analysis of Hearing Offices Using Key Risk Factors, a report from Social Security's Office of Inspector General (OIG):
We developed a model that measured variances among multiple risk factors . The model analyzes performance and outcome data among ALJs in the same office and uses five risk factors: (1) ALJ allowance rates, (2) ALJ dispositions, (3) ALJ on-the- record (OTR) decision rates, (4) ALJ dismissal rates, and (5) ALJ average processing time. While the Agency’s monitoring process identified a number of potential workload problems at the time of our review , such as ALJ -specific issues and productivity declines, our model offers another method to evaluate the performance of individual hearing offices.
Using our model and FY 2012 workload data, we identified hearing offices with the highest and lowest variance score s. We believe o utlier hearing office s provide ODAR managers with indications of potential processing issues (high-variance) as well as potential best practices (low - variance). We fou nd 4 regions had 20 percent or more of their hearing offices among the 25 high- variance offices , and 4 regions had 20 percent or more of their hearing offices among the 25 low- variance offices. In discussions with ODAR regional managers, we learned that they focused their oversight on individual ALJ performance rather than variances among ALJs in hearing office s as we do in our model.
Finally, our review of the hearing offices with the 10 highest variance scores identified an outlier ALJ who had a significant number of dispositions and OTR decisions with 1 claimant representative. We referred this case to ODAR management for additional review.No, they don't explain their model all that well nor do they give the variance score for any office, other than indicating that the Huntington, WV office had high variance scores in past years. The OIG model also identified the Huntington ALJ who was the subject of media attention as being an outlier. Of course, the model was probably tailored so that it would point to Huntington and that particular ALJ.
This proposal stems partly from internal turf issues. ODAR (particularly the AC) has been playing around with data analytics (on an amateurish level) and in so doing has succeeded in "dazzling" agency officials who have no training and little knowledge about data mining. So now OIG is getting in the game as well. Every component is now acquiring a working knowledge of the new data buzzwords and collection tools.
ReplyDeleteYeah, the people running the new quality stuff at my region, at least, have no mathematical/statistical background whatsoever, and neither do the new folks they brought on board. It's cute that you can run excel spreadsheets and all, but if you don't know how to collect and analyze data in a way that actually shows what you are looking for, you're just wasting everyone's time.
ReplyDeleteThey actually do explain the model...they took the high and low ALJ's (for each category) in each office and then measured the difference.
ReplyDeletei.e. the high FF ALJ may grant 85% FF and the low FF ALJ may only grant 20%, resulting in a "variance" of 65. They did this for 5 factors and then totaled them.
This is not statistically valid. Basically, one outlier in an office can skew the whole office. But if the goal is to identify outliers, this works. Not really useful, but what would you expect from SSA/OIG?
When are they going after the outliers on the bottom, the 80-90% denial ALJ's
ReplyDeleteConsidering the number of high payers is much greater than the number of high deniers and considering that the high payers cost the trust fund/taxpayers a significant amount of money, while high deniers do not, it seems the priority should be reining in the high payers; this would also lend more legitimacy to the disability program generally.
ReplyDeleteIn other words you didn't answer the question which is when will they start to go after the ALJs who hold mock hearings and have their own personal agenda instead of evaluating each claim on face value.
ReplyDeleteOIG tries so hard to make their 'findings' seem relevant. A year from now they'll be asking the question, 'why have hearings wait times skyrocketed and what should SSA do about it?' It's always too little, too late from OIG. As pointed out above, their analyses are amateurish indeed. The agency employs an army of analysts and managers to keep the assembly line running smoothly and with high quality results. OIG's little forays into SSA management issues do one thing only: remind agency leadership that the overseers are watching - a good thing. But these auditors are not trained consultants or strategic thinkers - they're bean counters, not decision makers.
ReplyDeleteIt's not easy to balance judicial independence and accountability. There does need to be a real way the system can flush out ALJs who perform poorly, chronically fail to follow the law, employ improper bias, etc. In the current system it is very difficult to enforce any such accountability, so it rarely happens. Unfortunately, that means the bad apples (thankfully few) can wreak a lot of damage.
ReplyDeleteIt's also desirable to protect ALJ independence, to avoid improper influence. From the outside looking in, I'd say the balance is tilted substantially in favor of independence and against accountability at present.
@12:04 PM, January 09, 2014 - Right on.
ReplyDelete@12:17 PM, January 09, 2014 - Maybe, maybe not. But no individual incorrect approval is devastating, while an incorrect denial in many cases is.
Justin