Following their individual assessments of what is ostensibly the same object, they compare notes―and are puzzled to find their conclusions about the object’s appearance to be in complete disagreement.
A recent EBRI Notes article¹ examined retirement plan participation through the prism of data from the Survey of Income and Program Participation (SIPP), which is conducted by the U.S. Census Bureau to examine Americans’ participation in various government and private-sector programs that relate to their income and well-being.
Now, as the EBRI analysis notes, the SIPP data have the advantage of providing relatively detailed information on workers’ retirement plans, but SIPP is fielded only once every three to five years. By comparison, the Current Population Survey (CPS), which is also conducted by the U.S. Census Bureau, provides overall participation levels of workers on an annual basis, but the CPS does not provide information on the specific types of plans in which the workers are participating. Another data source is the Bureau of Labor Statistics’ (BLS) National Compensation Survey, which annually surveys establishments’ offerings of employee benefit programs, including retirement plans―but at an employee level it includes information only on occupation, union status, and part-time/full-time work, and no information on age, gender, or race/ethnicity. Consider also that the CPS collects information about anyone who worked at any point in a previous year, while SIPP and BLS ask only about current workers in the month of interest.
As you can see, each of these national (and widely cited) surveys collects data in a different manner, at different times, and has different questions that can lead to different conclusions. Consider the chart below. The top line shows retirement plan participation rates for all workers from SIPP, and the lower line graphs retirement plan participation, also for all workers, from the CPS. While the trend lines generally move in the same direction, the more frequent CPS data allows us to see the more incremental movements―and to see a drop in participation rates in 2000–2002 following the burst of the tech bubble that would be completely missed looking only at the SIPP results. Moreover, relying strictly on SIPP data, one might well be inclined to see an increase in participation rates, rather than the leveling off that we see based on CPS data.²
That said, each survey provides important data that can’t be found elsewhere: CPS has the annual participation data with a complete set of worker demographics, while SIPP has the complete set of worker demographics plus retirement plan types, and BLS has detailed data on establishment characteristics, along with retirement plan type, although with limited worker demographics.
There was nothing inherently wrong in the individual observations made by the three blind men in the story―other than their examinations focused on one particular attribute, rather than appreciating the reality that a truly accurate assessment required looking at ALL the pieces, rather than each in isolation.
Similarly, and as the EBRI analysis concludes, those who would draw conclusions from individual surveys or datasets are well-advised to benchmark those results against other data, lest they conclude that the elephant(s) in the room are different than they truly are.
Nevin E. Adams, JD
¹ See “Retirement Plan Participation: Survey of Income and Program Participation (SIPP) Data, 2012,” online here.
² Another potential limitation of these surveys is that they are based on self-reported information, which is to say they rely on respondents’ recollections, rather than actual administrative plan data or IRS tax records. For SIPP specific results, see, for example, Irena Dushi, Howard M. Iams, and Jules Lichtenstein, “Assessment of Retirement Plan Coverage by Firm Size, Using W-2 Tax Records,” Social Security Bulletin, Vol. 71, No. 2, 2011, pp. 53–65, online here.
No comments:
Post a Comment