Survey Says—Or Does It?

When you see a headline that confirms your sense of the world, you’re naturally predisposed to embrace, remember (and these days “share”) it as a validation of what you already perceive reality to be.

Indeed, as human beings, we’re drawn to perspectives, surveys, and studies that validate our sense of the world. This “confirmation bias,” as it’s called, is the tendency to search for, interpret, favor, and recall information in a way that confirms our preexisting beliefs or hypotheses. It also tends to make us discount or dismiss findings that run afoul of our existing beliefs—even if the grounds supporting that premise are shaky, sketchy, or (shudder) downright scurrilous.

Here are some things to look for—likely in the fine print or footnotes—as you evaluate those findings.

There can be a difference between what people say they will (or might) do and what they actually will.

No matter how well targeted they are, surveys (and studies that incorporate the outcome of surveys) must rely on what individuals tell us they will do in specific circumstances, particularly in circumstances where the decision is hypothetical. When you’re dealing with something that hasn’t actually occurred, or doesn’t actually exist, there’s not much help for that, but there’s plenty of evidence to suggest that, once given an opportunity to act on the actual choice(s), people do, in fact, act differently than their response to a survey might suggest.

Let’s face it, people tend to be less prone to action in reality than they indicate they will be—inertia being one of the most powerful forces in human nature. Also, sometimes survey respondents indicate a preference for what they think is the “right” answer, or what they think the individual conducting the survey expects, rather than what they might actually think (particularly if it’s something they haven’t previously thought about). That, of course, is why the positioning and framing of the question can be so important (as a side note, whenever possible, it helps to see the actual questions asked, and the responses available).

Now, survey takers will inevitably champion the higher accuracy rate of in-person surveys (or at least phone calls) versus online surveys, though the latter are ever more common (and less expensive to conduct).  

The bottom line is that when what people tell you they will do, and if you later find that they don’t—just remember that there may be more “powerful” forces at work.

There can be a difference between what people think they have, what they say they have, and reality.

Since, particularly with retirement plans, there are so few good sources of data at the participant level, much of what gets picked up in academic research is based on information that is “self-reported,” which is to say, it’s what people tell the people taking the survey. The most prevalent is, perhaps, the Survey of Consumer Finance (SCF), conducted by the Federal Reserve every three years.

The source is certainly credible, but it’s based on phone interviews with individuals about a variety of aspects of their financial status, including a few questions on their retirement savings, expectations about pensions, etc. In that sense, it tells you what the individuals surveyed have (or perhaps wish they had), but not necessarily what they actually have.

Perhaps more significantly, the SCF surveys different people every three years, so it pays to be wary of trendlines that are drawn from its findings—such as increases or decreases in retirement savings. Those who do are comparing apples and oranges—more precisely the savings of one group of individuals to a completely different group of people… three years later.

The survey sample size and composition matter.

Especially when people position their findings as representative of a particular group, you want to make sure that that group is, in fact, adequately represented. Perhaps needless to say, the smaller the sampling size—or the larger the statistical error—the less reliable the results.

Case in point: Several months ago, I stumbled across a survey that purported to capture a big shift in advisors’ response to the Labor Department’s fiduciary regulation. Except that between the two points in time when they assessed the shift in sentiment, they wound up talking to two completely different types of advisors. So, while the surveying firm—and the instrument—were ostensibly the same, the conclusions drawn as a shift in sentiment could have been nothing more than a difference in perspective between two completely different groups of people—at two completely different points in time.

When you ask may matter as much as what is asked.

Objective surveys can be complicated instruments to create, and identifying and garnering responses from the “right” audiences can be an even more challenging undertaking. That said, people’s perspectives on certain issues are often influenced by events around them—and a question asked in January can generate an entirely different response even a month later, much less a year after the fact.

For example, a 2020 survey of plan sponsor sentiment on a topic like ESG litigation is unlikely to produce identical results to one conducted in the past 30 days, any more than an advisor survey about the potential impact of the fiduciary regulation prior to its publication would likely match that of advisors dealing with those realities six months after publication. Down in those footnotes about sample size/composition, you’ll likely find an indication as to when the survey was conducted. There’s nothing wrong with recycling survey results, properly disclosed. But things do change, and you need to be careful about any conclusions drawn from old data.

Consider the source(s).

Human beings have certain biases—and so do the organizations that conduct and pay to conduct surveys and studies conducted. And sometimes the organizations paid to conduct such surveys are aware of those biases, and—consciously or unconsciously—that filters in to the way questions are posed, or in the way results are evaluated.

Not that sponsored research can’t provide valuable insights. But approach with caution the conclusions drawn by those who tell you that everybody wants to buy the type of product(s) offered by the firm(s) that have underwritten the survey.   

Be wary of sentiment ‘aggregation.’

It’s rare that the authors of a particular survey don’t have a preferred/expected outcome in mind—but legitimate surveys, objectively worded, sometimes receive a more tepid response than those authors might prefer. Typical are those that claim a “majority” are in favor of a certain outcome—a majority that requires combining what is generally a small minority who are strongly in favor with a (much?) larger number who are (only) somewhat in favor (for example, 16% strongly in favor, 35% somewhat favor turns into “A Majority Favor…”). 

It’s not exactly exaggerating to say that the combined result is at least somewhat supportive—but it can produce a result that is positioned far more enthusiastically in favor of a particular outcome than a discerning look at actual adoption/take-up later reveals.

Compound ‘Interests’

One of the more obvious ways to get people’s attention is to publish a survey/study that purports to find a dramatic impact of some kind. Basically, the authors will state an assortment of assumptions (and they’ll make no bones about THAT), and then take those assumptions, multiply them and…voila a gigantic impact that warrants attention (or at least clicks, likes and shares). 

The math checks out, so next thing you know it’s a headline where, as Mark Twain once noted, a “lie” travels around the world while the truth is still getting its boots on. It does so by being picked up, uncritically, by news media outlets which (apparently) draw comfort from the academic credentials of the authors—and their ability to lay the veracity of the claims at THEIR feet. 

When, in fact, all they’re doing is compounding the problem(s).     

- Nevin E. Adams, JD

Commencement "Address"

This is the time of year when the nation’s graduates line up for accolades (and their diplomas). It is, for them, a beginning—a commencement of a new phase in their life. 

But ahead of that, most are given the “opportunity” to hear some words of wisdom and inspiration from an individual that they have likely never heard of (though their parents may have). In that spirit, I’d like to offer the graduates of 2023 some lessons I’ve picked up along the way:

Your first job can be like your first love—it will either bring a smile for years to come—or it can break your heart. And sometimes both. 

Just because you’re young(er), people are going to assume you know things you don’t—and assume you don’t know things you do.

Everything you’ve heard about your elders isn’t true. But some of it is.

There actually ARE stupid questions.

If your current boss doesn’t want to hear the truth, it may be time to look for a new one.

There can be a “bad” time even for good ideas.

Your work attitude often affects your career altitude.

When you don’t have an opinion, “what do you think?” is a good response. And sometimes even when you do.

People who ask for something ASAP probably want it sooner than you think is possible.

Emails (generally) don’t have to be answered right this minute.

Don’t be afraid to pick up the phone—BEFORE it rings.

If the only time your boss hears from you is when there’s trouble, don’t be surprised if they don’t look forward to your visits.

Book some quiet time in your day.

Most meetings really COULD be replaced with an email.

The world is made up of introverts and extroverts—learn and respect the difference(s).

A picture may be worth a thousand words, but it pays to read the fine print.

Never say you’ll never…

Always sleep on big decisions.

There is an inverse relationship between the number of people in a meeting and its productive output.

Never let your schooling get in the way of your education.

Sometimes the questions are complicated, but the answer isn’t.

And most of all, don’t forget that you’ll want to plan for your future now—because retirement, like graduation, seems a long way off—until it isn’t.

Congratulations to all the graduates out there. We’re proud of you!

- Nevin E. Adams, JD

A New Fiduciary Standard?

Resistance to retirement plan innovations (like automatic enrollment) have long been excused as being “too paternalistic” – but there might be a better standard.

We’ve all heard it – concerns that imposing certain default choices on participants (and sometimes plan sponsors) are, however well-intentioned, intrusive and demeaning. Generally speaking, such concerns aren’t challenged – we “get it,” after all – most of “us” are do-it-for-myself types.

Of course, most participants aren’t – and there’s plenty of anecdotal evidence that workers, and particularly younger workers, WANT that kind of proactive support from their employer.

All of which calls to mind a new standard – one first (to my ears, anyway) articulated in the Nevin & Fred podcast by none other than Fred Reish. See, Fred was talking about explaining to his daughter what a fiduciary was – and she quickly grasped the concept, applying it to her mother and her support for her kids in looking out for them, and their best interests. It’s something I suspect just about every mother (or everyone who has had a mother) can relate – the notion that you’d do anything for your kids. No matter how old they (or you) are. A maternal standard of care, if you will.

How might that manifest itself in plan design? Well, immediate participation and automatic enrollment, for sure – though the latter likely at a rate higher than the 3% threshold that’s been established (first by tradition, then by law) as a minimum. And, depending on that starting rate, contribution acceleration – but one that follows automatically, not dependent on a separate affirmative election. These are not big stretches from where things stand at present of course – but it took the Pension Protection Act of 2006 to bring these structures to the fore – and years longer to lift those initial thresholds – years that higher levels of participation and savings could have been accumulating. 

Now, there were – and in some cases still are – legitimate reasons for plan fiduciaries to hold back on such things. For automatic enrollment, there were concerns that it imposed a financial decision that participants don’t need or can’t afford. There were (and are) administrative costs and burdens attendant with them all – and if there’s anecdotal evidence to suggest participant support, the concerns regarding negative reactions are just as real. And yet, how many have been left on the savings sidelines by those rationalizations?

Of late, I’ve been thinking about another plan design “hesitation” – retirement income. There’s little argument that those solutions are a need – but no real consensus that providing it is, or should be, a plan sponsor’s responsibility. As with the PPA, the SECURE Act provided encouragement; some much-needed (1) legislative structure and guidelines to provide fiduciary comfort with the selection and review of potential provider(s), (2) a safe harbor for the portability of benefits – and even (3) presentation on the participant statement of an amount designed to remind them of what their accumulated balance could produce in retirement income. In fact, those elements were specifically crafted to overcome the traditional objections to considering these options.

To date, the adoption rate – by plan sponsors AND participants – has not been what proponents would hope. Of course, those guidelines and provisions became law just ahead of COVID-19, and there have been a lot of other employment/benefit concerns that arguably, and even rightfully, took precedence. Participants are not, in fact, asking for these features (at least not to their employers), and there remain real fiduciary and operational concerns remaining, even with the guardrails. 

That said, I wonder if it’s not time for plan sponsors to take a more “maternal” approach to plan design – to consider anew – but still prudently and thoughtfully – plan designs like retirement income – to do more than just what the law requires, but what those whose interests they are charged with considering – need. 

I suspect it’s what Mom would do.

- Nevin E. Adams, JD