Lies, Damned Lies, and Statistics

I am fortunate enough to have access to a vast array of studies, research, and surveys about this business. Even more fortunate to have access to a PLANSPONSOR research arm that provides an opportunity not only to gather and analyze, but to pose our own questions to a remarkably diverse audience. Still, as Mark Twain once famously wrote, “There are three kinds of lies: lies, damned lies, and statistics.” (1)

We recently ran coverage of a survey that spoke to trends among defined benefit plans. That engendered the following response from a reader:

Isn't it interesting how perspective can rule the most simple things? The article you refer to that shows defined benefit plans decreasing in number is such a case. Mercer deals with the larger corporate plan sponsors. Their plans are underwater for numerous reasons and are being terminated in wholesale lots. On the other hand, small companies are making defined benefit plans the plan du jour. There are several reasons for this; larger tax deductions, demographics, insecurity with Social Security, the investment experiences of 1999-2002, the desire for a "guaranteed" benefit, etc. Our company is the largest pension administration firm in Florida, and we have set up five defined benefit plans to every three defined contribution for four consecutive years. Our experience has been shared by virtually all TPA's in the small-company market all over the country.

Now, I have no independent validation of that reader’s claims, but I heard from a number of advisers and providers that support smaller employers that the rumors of the defined benefit plan’s demise is, to draw on another quote from Mr. Twain, “greatly exaggerated”.

In our business, we must constantly guard against the favorable positioning of some survey results vis-à-vis the interests of a sponsoring organization; or the extrapolation of too much conclusion from too small, or unscientific, a sampling. Generally speaking, I favor sharing as much information/insights as possible—with as much full disclosure about the size of the sampling and/or the interest(s) of the sponsoring organization as possible. In our coverage, we try very hard to position—as high in the story as possible—the size of the sampling, the sponsoring organization, and where it seems applicable (and not obvious), some indication of possible motivation in putting out the information. It’s not that they necessarily would be pushing a specific result—sometimes it just influences their perspective on the proper conclusion to be drawn from the data.

Surveys Say

The challenge, of course, is that some data are simply more available. Larger providers tend to have larger client bases, and client bases of larger clients, to draw on, and/or larger budgets to pay other organizations to gather that information (generally from plans that fit their targeted plan demographics). Moreover, the use of “averages” frequently, if unintentionally, obscures the reality in small samplings. And when averages are averaged—well, it’s Katie, bar the door!

In my experience, human beings are inclined to focus on surveys that reinforce their own version of reality, rather than one that challenges it. Still, market trends are a significant motivator of plan sponsor behavior. That’s not illogical, IMHO—in an environment where complex financial decisions fraught with personal and professional risk are the order of the day, “what everyone else is doing” can offer a compelling reality check. But only if the reality is real—and not just “lies, damned lies, and statistics.”

- Nevin E. Adams, JD

==================================================

When considering data that purports to offer insights, it’s worth knowing:

Who sponsored/conducted the survey?
What are they selling?
Is the conclusion supported by the data?
Does that conclusion “fit” with those drawn by similar surveys?
How unbiased is the survey sampling?
How scientific is the survey sampling?
How similar is the survey sampling to your perspective/size?


(1) While the original quote is attributed to Benjamin Disraeli, Twain popularized it in the U.S. in “Chapters from My Autobiography.”