Oct 15, 2009

A 12 Minute Test For Mental Or Cognitive Disability?

Social Security's Occupational Information Development Advisory Panel (OIDAP) has been assigned the task of advising the agency on what to do about the fact that disability determination at Social Security is heavily dependent upon the Department of Labor's hopelessly out of date Dictionary of Occupational Titles (DOT).

Most, if not all, of OIDAP's report to the Commissioner of Social Security is now available. I have been going through it and will be posting about what I have found.

What I have seen consists of the reports of OIDAP's five subcommittees. I have not seen a report from the entire Panel. It is not clear to me that the entire Panel issued any final recommendations other than to endorse what its Subcommittees recommended.

I will start with the report of the Mental Cognitive Subcommittee since I find it extraordinarily jarring, but before I get to that let me give this quote from the report of the Transferable Skills Analysis Subcommittee as a backdrop:
At the inaugural meeting of the Panel, we were advised that the work of the Panel did not include recommending changes to SSA's [Social Security Administration's] disability policies; rather, we were instructed to treat SSA's disability policies as though they were "standing still." Through further conversations, it was learned that SSA intended that the focus of our recommendations be upon the OIS [Occupational Information System] itself rather than SSA policy or possible effects upon said policy. That is, the OIS we are helping to create must meet SSA's current adjudicative needs at a minimum. (page 223 of the OIDAP Report PDF)
Now, let me give a couple of excerpts from the Mental Cognitive Subcommittee Report:
The conceptual model of psychological abilities required to do work should be revised. The aims are to address shortcomings of the current model, base a revised model on scientific evidence, identify specific abilities that can be reliably assessed and tested for predictive validity, and retain elements of the current mental residual functional capacity (MRFC) that meet these criteria in order to maintain continuity where possible.

The Subcommittee recommends a series of studies to determine the reliability and predictive validity of any instruments developed to assess residual functional capacities and occupational demands as part of the OIS [Occupational Information System] Project. (page 56) ...

... [T]he Subcommittee recommends that the SSA conduct a study in which all of the revised physical and mental residual functional capacity measures are administered to a nationally representative sample of persons who have worked for at least six months ("successful incumbents") in one of the 150 to 200 most common occupations in the U.S. Economy. (page 82)
First, who told the Mental Cognitive Subcommittee that their job was to revise "conceptual models" or to come up with some test instrument to determine disability due to mental and cognitive problems? From what the Transferable Skills Subcommittee says, OIDAP was told the exact opposite, to stick to the issue of how to replace DOT.

You may well wonder what sort of test the Mental Cognitive Subcommittee thinks can be used to determine disability. You may not believe the answer. The Subcommittee was entranced with something called "g." G is a score derived from the Wonderlic employment aptitude test. That test consists of 50 questions and can be given in 12 minutes. The Subcommittee seemed to believe that g was close to being a perfect index for determining disability due to mental or cognitive problems. However, the Subcommittee thought that g was still not perfect enough for disability determination and that Social Security could and should build on g to create a test to be used to determine disability due to mental and cognitive problems. I told you that you might not believe this.

As best I can tell from reading the Subcommittee's report, they felt that any Occupational Information System (OIS) to replace the DOT would be irrelevant in determining disability due to mental and cognitive impairments. As the quote above says, they have in mind a new "conceptual model" for determining disability due to mental and cognitive impairments. Apparently, age, education and work experience are not included in their "conceptual model," even though the statutory definition of disability requires consideration of these elements.

My prediction is that someone will devise a test to simply and accurately determine mental or cognitive disability at about the same time as a machine is invented to accurately measure how much pain a person is suffering -- never.

The Mental Cognitive Subcommittee went completely out of bounds. My opinion is that their report is worthless and should be quickly repudiated by Social Security.

By the way, I have already heard an attorney who practices Social Security law say that he is considering having all his clients take the Wonderlic. I wonder what Social Security will make of it if that starts happening regularly.

Also, by the way, the Mental Cognitive Subcommittee report recommended that Social Security conduct a study of physical residual functional capacity measures as well as mental. None of the other Subcommittees recommended physical residual functional capacity measures. Does this mean that the Mental Cognitive Subcommittee was so far out of bounds that it was making a recommendation of physical residual functional capacity tests as well as a mental one? Does this mean the physical residual functional capacity measures were under consideration by other Subcommittees which decided not to recommend them? Is this a sign that OIDAP will recommend physical residual functional capacity tests at a later time?


Nancy Ortiz said...

What you said, Mr. Charles T. :)

Anonymous said...

They didn't propose to discard the mental RFC assessment altogether, but they did consider the 4 categories and 20 abilities rated on the SSA-4734-SUP form, which is the current "conceptual model." Exactly how job demands are going to be rated reliably based on that form's constraints is beyond me. As they very prominently noted, "We must first decide what abilities to assess before we decide how to assess them." They recommended overall a somewhat different 4 categories with 15 abilities for evaluation.

It was noted in the report that "it is impossible to parse psychological abilities that are essential for work into completely orthogonal dimensions", which would appear to include Dr. Schretlen's proposed g, and also included recommendations 2 and 3, which were on page 74 of that PDF. They did however, state that job complexity could be rated on a neurocognitive basis, but complexity alone was not the only demand to be rated.

I also think it was a bit disingenuous to not include the rest of the paragraph from the TSA subcommittee. The OIS should provide a platform from which SSA can develop and test revisions to its disability process and policies as the OIS data are obtained. Statistical analyses of OIS data and applied research will provide empirical bases for policy evolution that may result in proposed policy revisions that will be developed according to the Administrative Procedures Act (APA). I read this as stating that they are expected to obtain data on which SSA can change its policies regarding vocational evaluation, which seems more consistent with what the Mental Cognitive Subcommittee is doing.

I'm not saying the report is perfect, and my observations could be incorrect, but I feel you're raising an alarm over what amounts to nothing here.

Anonymous said...

Thanks for raising an alarm Charles! SSA will not likely go for any of these recommendations beyond the DOT replacement.