Preaching It However You Want

     Social Security's effort to create an Occupational Information System (OIS) to replace the Dictionary of Occupational Titles (DOT) is the most important policy development at the agency in more than thirty years. Recent activities at the agency's Occupational Information Development Advisory Panel (OIDAP), which is tasked with creating the OIS, remind me of the story of a young preacher, badly in need of a job, who is sitting before a church board. A member of the board asks, "How do you feel about same sex marriage?" The young preacher immediately replies, "Just tell me how you feel. I can preach it either way." OIDAP wants to make sure they are creating the OIS in such a way that it will say whatever Social Security management at any given time wants it to say -- and remember that the term of Social Security Commissioner Michael Astrue expires in January 2013. OIDAP wants to please not just the current Commissioner but his successor as well. No one knows how that person will want it preached so OIDAP is building in the capacity to preach it any way the new Commissioner wants it preached.
     Recently I asked what I think are the key questions about Social Security's OIS plans:

• Let us suppose that you collect data on a job, such as office assistant, and this data shows it to have been performed at the sedentary level at 20% of employers, at the light level at 70% of employers and at the medium level at 10% of employers. How would this job be categorized in the occupational information system you are developing -- as sedentary, as light or as medium -- or would it be categorized as all three -- or would it be broken down into three new job titles of Office Assistant I, II and III? A similar question could also be asked about the length of time it takes to learn the job of office assistant or many other criteria.
• When will the decision be made on how to categorize jobs? Before or after the data is collected?

     OIDAP has released its Occupational Information System Fiscal Year 2011 Research And Development Plan which gives us a good idea of how these questions will be answered. I have uploaded this using Scribd, a helpful service that allows anyone to get documents on the public record for free. This is a roadmap for Social Security's OIS development. It definitely answers the last question I asked above and sets forth the process for how the rest of the questions will be answered, although everything is expressed in OIDAP's typically opaque prose.
     One early stage in the development of the OIS to is create a prototype data analysis plan. The plan identifies the following as a "key question": "What are SSA's analytic objectives (e.g., aggregation of position-level data to job-and occupation-level categories and calculation of occupational prevalence) How do they align with SSA OIS requirements." page 36. Exactly. They are not asking the abstract question of how this data should be classified but how does Social Security want the data classified so that it meets the agency's "requirements." They are going to be making the decision about whether the hypothetical job I was asking about will be aggregated simply as Office Assistant or broken down as three separate jobs of Office Assistant I, II and III or how the information about this job or jobs is presented based upon what the Social Security Administration wants. But you may object that the language I quoted above only shows that OIDAP is legitimately concerned with Social Security's needs and isn't really going to tailor the OIS to what Social Security management wants at any given time. Read on.

     The next stage is to conduct a pilot using the prototype. Here the report identifies a key question as being "What can SSA learn for OIS design from the occupational information collected as part of the data collection analysis pilot? (For example, does the information help SSA determine what level of within-title heterogeneity is acceptable given the agency's program needs.)" page 37. That term "within-title heterogeneity" tends to throw one, doesn't it? The titles they are talking about are job titles and by heterogeneity they mean how wide or narrow to make the job titles. Again, do we just have the job title of Office Assistant or do we divide it into Office Assistant I, II and III. They say explicitly that this decision will be made based upon "the agency's program needs" after data collection has started Again, the decision is not being made on the basis of any science or the basis of any independent judgment but on the basis of what Social Security wants. Are you starting to get the picture?

     Next we go to a national pilot and again there is the same key question about "within-title heterogeneity." pages 39 and 41.

   Beyond that, OIDAP must formalize an "occupational title taxonomy" and again the same key question of "heterogeneity" pops up. page 43.

    At the next step, OIDAP intends to evaluate what will happen when the OIS is implemented in the real world by comparing results using the OIS with results using the DOT "to assess relative utility of new OIS data and potential operational and programmatic gaps." page 44. They want to make sure that Social Security is achieving whatever goal it has in approving and denying claimants. Of course, as part of that they will again address that pesky issue of "within-title heterogeneity." page 48.

     If you read the whole thing, it is clear that this plan explicitly allows Social Security management to control the process each step of the way to achieve whatever goals that agency management has at any given point. OIDAP is ready to "preach" it however Social Security management wants it preached.

     Science is being employed here, to be sure, but science will not determine the results. The results will be determined by the agency's policy desires. That is the whole point. Social Security management doesn't want to spend hundreds of millions of dollars to produce an OIS that it cannot stand and the Social Security employees working on the OIS don't want to worry that they will be fired by a new Commissioner because they were unable to predict what that future Commissioner would want. What we have here, to quote Dilbert, is "the way of the weasel."