From a recent report by Social Security's Office of Inspector General (OIG):
A claimant can appeal an ALJ’s [Administrative Law Judge's] decision to deny or dismiss a disability case. Claimants file these appeals through a request for review to SSA’s AC [Appeals Council] in the Office of Disability Adjudication and Review (ODAR). If the AC grants a review of the case, it will issue a fully favorable, partially favorable, or unfavorable decision; or it may remand the case to an ALJ. If the AC does not grant a case review, the earlier decision remains unchanged.
ODAR tracks the AC’s decision on every appealed case and calculates a quality performance measure for each ALJ. The decision agree rate represents the extent to which the AC concludes the ALJ decisions were supported by substantial evidence and contained no error of law or abuse of discretion justifying a remand or reversal. At the time of our review, the national agree rate goal for ALJ decisions was 85 percent. The national dismissal agree rate goal for ALJ dismissals was 72 percent, but less than 6 percent of the AC workload related to dismissals. ...
Note that there is no similar system for tracking remands and reversals by the federal courts. I think some interesting patterns would be seen if that were studied.
... Regarding the 11 ALJs whose decision agree rates were 65 percent or lower for 3 consecutive years, we found ODAR managers had taken action to improve the quality of their decisions. Five ALJs had undergone a focused quality review, three were scheduled for this type of review, and one had been nominated to undergo a review. Of the remaining two ALJs, one had undergone a regional quality review, and the other was scheduled for a one-on-one counseling session with the Regional Chief ALJ. A focused quality review involves an in-depth review of a sample of an ALJ’s decisions to ensure the decisions are policy compliant and legally sufficient. If managers find a pattern of error-prone cases, they offer advice on how to correct errors and determine whether the ALJ should take additional targeted training. Three of the five ALJs who underwent focused quality reviews had completed targeted training.