We have over the past couple of years devoted some of this space to highlighting some of the most egregious instances, but as I close this chapter of my professional career, I wanted to share with you a “top 10” list of things I have learned in my search to find reliable, objective, actionable research:
- There are always assumptions in research; find out what they are. Garbage in, garbage out, after all (the harder you have to look, the more suspicious you should be).
- Just because research validates your sense of reality doesn’t make it “right.” But just because it invalidates your sense of reality doesn’t necessarily make it right, either.
- Take the conclusions of sponsored research with a grain of salt.
- Self-reported data can tell you what the individual thinks they have, but not necessarily what they actually have.
- Sample size matters. A lot.
- “Averages” (e.g., balances/income/savings) don’t generally tell you much.
- There’s a certain irony that those who propose massive changes in plan design, policy, or tax treatment, frequently assume no behavioral changes in response.
- When it comes to research findings, “directionally accurate” is an oxymoron.
- In assessing conclusions or recommendations, it’s important to know the difference between partisan, bipartisan and nonpartisan.
- In an employment-based benefits system, the ability to accurately gauge employee response to benefits change is dependent on the reaction of the employers who provide access to those benefits.
That said, I’ll close by commending to your attention one of my favorite “lessons” – a quote attributed to Mark Twain, and one worth keeping in mind along with the 10 “lessons” above:
“It’s easier to fool people than to convince them they have been fooled.”
Here’s to not being fooled.
- Nevin E. Adams, JD