(Just Because) Survey Says?

 We’re often told (via surveys) of any number of retirement plan options that plan participants want. Should it matter?

Most recently, we’re assured that participants are clamoring to have access to private market investments. Before that, it was cryptocurrency — and for what seems like months now it’s all been about retirement income. Apparently vast majorities of participants are eager to have access through their 401(k) to an array of complex financial instruments to which they have, thus far, been largely (or totally) barred — or so surveys say.

Indeed, I’ve always been amazed that organizations are able to find so many ostensibly knowledgeable participants to weigh in on these complex topics — particularly since there is an abundance of (other) surveys that suggest that when it comes to financial matters, participants are, largely, clueless.

Not that that seems to dampen their collective interest in these new options — though when given a chance to put their money where their mouth is, participants seem to be about as cautious as you’d expect (want?) largely financially clueless individuals to be.  Doubtless the questions posed in these survey instruments are less complex than the actual decision points turn out to be.   

So, why bother asking participants what they (think they) want in the first place?

It’s not that there isn’t value to be gleaned in assessing participant sentiment but — as noted above, the actual take-up rate on those products, even when offered by a plan, has traditionally been…disappointing, certainly from the standpoint of the manufacturers of those offerings.

Manufacturers that, it bears noting, inevitably turn out to be the sponsors of these surveys claiming that participants want what they rarely seem to take. That doesn’t make them irrelevant, of course, nor does it mean that the results are hopelessly biased — but it is, I would argue, worth some caution in blindly embracing (or sharing) the results, even when a reputable canvassing body is involved. At a minimum, a visible footnote acknowledging that specific “interest” in the outcome seems warranted.

The reality is that these types of surveys are rarely designed to actually gauge participant interest — savvy product manufacturers have likely already done some of that alongside the inevitable market and profit margin projections. Rather, they are a means to create and/or fuel public awareness, but perhaps more specifically to encourage plan sponsor/fiduciary consideration on the path to the ultimate acceptance of options they might otherwise be disinclined to entertain.

Prudent plan fiduciaries should, of course, take note of such things — but arguably with a grain of salt, certainly as it pertains to their specific workforce. However fervent the interest of a group of unrelated individuals might be, that doesn’t make it relevant for those under your care.

Indeed, ERISA’s standard of care is a high threshold, one that requires that the services engaged not only be reasonable in terms of fees and applicability, but solely in the best interests of plan participants and their beneficiaries.

And, as any parent can attest, sometimes those in our care want things that don’t meet that standard, whatever their opinion may be at the time.

Regardless, plan fiduciaries shouldn’t feel coerced into incorporating options just because a sponsored survey says (some) participants want it — participants who, of course, might not know what they’re asking for. Because, after all, prudent plan fiduciaries are expected to.

- Nevin E. Adams, JD

Survey Says—Or Does It?

When you see a headline that confirms your sense of the world, you’re naturally predisposed to embrace, remember (and these days “share”) it as a validation of what you already perceive reality to be.

Indeed, as human beings, we’re drawn to perspectives, surveys, and studies that validate our sense of the world. This “confirmation bias,” as it’s called, is the tendency to search for, interpret, favor, and recall information in a way that confirms our preexisting beliefs or hypotheses. It also tends to make us discount or dismiss findings that run afoul of our existing beliefs—even if the grounds supporting that premise are shaky, sketchy, or (shudder) downright scurrilous.

Here are some things to look for—likely in the fine print or footnotes—as you evaluate those findings.

There can be a difference between what people say they will (or might) do and what they actually will.

No matter how well targeted they are, surveys (and studies that incorporate the outcome of surveys) must rely on what individuals tell us they will do in specific circumstances, particularly in circumstances where the decision is hypothetical. When you’re dealing with something that hasn’t actually occurred, or doesn’t actually exist, there’s not much help for that, but there’s plenty of evidence to suggest that, once given an opportunity to act on the actual choice(s), people do, in fact, act differently than their response to a survey might suggest.

Let’s face it, people tend to be less prone to action in reality than they indicate they will be—inertia being one of the most powerful forces in human nature. Also, sometimes survey respondents indicate a preference for what they think is the “right” answer, or what they think the individual conducting the survey expects, rather than what they might actually think (particularly if it’s something they haven’t previously thought about). That, of course, is why the positioning and framing of the question can be so important (as a side note, whenever possible, it helps to see the actual questions asked, and the responses available).

Now, survey takers will inevitably champion the higher accuracy rate of in-person surveys (or at least phone calls) versus online surveys, though the latter are ever more common (and less expensive to conduct).  

The bottom line is that when what people tell you they will do, and if you later find that they don’t—just remember that there may be more “powerful” forces at work.

There can be a difference between what people think they have, what they say they have, and reality.

Since, particularly with retirement plans, there are so few good sources of data at the participant level, much of what gets picked up in academic research is based on information that is “self-reported,” which is to say, it’s what people tell the people taking the survey. The most prevalent is, perhaps, the Survey of Consumer Finance (SCF), conducted by the Federal Reserve every three years.

The source is certainly credible, but it’s based on phone interviews with individuals about a variety of aspects of their financial status, including a few questions on their retirement savings, expectations about pensions, etc. In that sense, it tells you what the individuals surveyed have (or perhaps wish they had), but not necessarily what they actually have.

Perhaps more significantly, the SCF surveys different people every three years, so it pays to be wary of trendlines that are drawn from its findings—such as increases or decreases in retirement savings. Those who do are comparing apples and oranges—more precisely the savings of one group of individuals to a completely different group of people… three years later.

The survey sample size and composition matter.

Especially when people position their findings as representative of a particular group, you want to make sure that that group is, in fact, adequately represented. Perhaps needless to say, the smaller the sampling size—or the larger the statistical error—the less reliable the results.

Case in point: Several months ago, I stumbled across a survey that purported to capture a big shift in advisors’ response to the Labor Department’s fiduciary regulation. Except that between the two points in time when they assessed the shift in sentiment, they wound up talking to two completely different types of advisors. So, while the surveying firm—and the instrument—were ostensibly the same, the conclusions drawn as a shift in sentiment could have been nothing more than a difference in perspective between two completely different groups of people—at two completely different points in time.

When you ask may matter as much as what is asked.

Objective surveys can be complicated instruments to create, and identifying and garnering responses from the “right” audiences can be an even more challenging undertaking. That said, people’s perspectives on certain issues are often influenced by events around them—and a question asked in January can generate an entirely different response even a month later, much less a year after the fact.

For example, a 2020 survey of plan sponsor sentiment on a topic like ESG litigation is unlikely to produce identical results to one conducted in the past 30 days, any more than an advisor survey about the potential impact of the fiduciary regulation prior to its publication would likely match that of advisors dealing with those realities six months after publication. Down in those footnotes about sample size/composition, you’ll likely find an indication as to when the survey was conducted. There’s nothing wrong with recycling survey results, properly disclosed. But things do change, and you need to be careful about any conclusions drawn from old data.

Consider the source(s).

Human beings have certain biases—and so do the organizations that conduct and pay to conduct surveys and studies conducted. And sometimes the organizations paid to conduct such surveys are aware of those biases, and—consciously or unconsciously—that filters in to the way questions are posed, or in the way results are evaluated.

Not that sponsored research can’t provide valuable insights. But approach with caution the conclusions drawn by those who tell you that everybody wants to buy the type of product(s) offered by the firm(s) that have underwritten the survey.   

Be wary of sentiment ‘aggregation.’

It’s rare that the authors of a particular survey don’t have a preferred/expected outcome in mind—but legitimate surveys, objectively worded, sometimes receive a more tepid response than those authors might prefer. Typical are those that claim a “majority” are in favor of a certain outcome—a majority that requires combining what is generally a small minority who are strongly in favor with a (much?) larger number who are (only) somewhat in favor (for example, 16% strongly in favor, 35% somewhat favor turns into “A Majority Favor…”). 

It’s not exactly exaggerating to say that the combined result is at least somewhat supportive—but it can produce a result that is positioned far more enthusiastically in favor of a particular outcome than a discerning look at actual adoption/take-up later reveals.

Compound ‘Interests’

One of the more obvious ways to get people’s attention is to publish a survey/study that purports to find a dramatic impact of some kind. Basically, the authors will state an assortment of assumptions (and they’ll make no bones about THAT), and then take those assumptions, multiply them and…voila a gigantic impact that warrants attention (or at least clicks, likes and shares). 

The math checks out, so next thing you know it’s a headline where, as Mark Twain once noted, a “lie” travels around the world while the truth is still getting its boots on. It does so by being picked up, uncritically, by news media outlets which (apparently) draw comfort from the academic credentials of the authors—and their ability to lay the veracity of the claims at THEIR feet. 

When, in fact, all they’re doing is compounding the problem(s).     

- Nevin E. Adams, JD