Abused Stats and Figures: Maintain a Healthy Degree of Skepticism 100% of the Time

Stats and figures help people make decisions or convince others to make a choice. Whether you’re a sales rep or a consumer, these numbers can be beneficial, but they are also easily misunderstood, misrepresented, or abused.

Uri Simonsohn, a research psychologist at the University of Pennsylvania’s Wharton School, sensed that something was amiss with several sets of research findings published in his field. Upon investigating, he discovered that the studies’ authors had taken liberties with the data and were forced to back away from their published articles. For his efforts, he was labeled a “data vigilante,” which paints a portrait (either white hat or black hat, depending on your views), but more importantly, presents us all with cautionary advice: be careful how you use and interpret data and statistics. (See the full article “The Data Vigilante” by Christopher Shea in The Atlantic from November 28, 2012.)

The article in The Atlantic offers a somber comparison between massaging data to suit your study’s needs and doping by professional athletes: “Outright fraud is probably rare. Data manipulation is undoubtedly more common—and surely extends to other subjects dependent on statistical study… Worse, sloppy statistics are ‘like steroids in baseball’: Throughout the affected fields, researchers who are too intellectually honest to use these tricks will publish less, and may perish. Meanwhile, the less fastidious flourish.” In essence, cheaters with more sensational findings will fare better than by-the-book do-gooders.

Train Yourself to Be Skeptical of Stats and Figures

After seeing the article referenced above, it got me thinking about sales stats and figures dropped into pitches, presentations, and marketing literature. How much of it is true? How much of it is used out of context?

In their book Taming the Terrible Too's of Training: How To Improve Workplace Performance In The Digital Age, authors Daniel and Ken Cooper illustrate why maintaining a healthy degree of skepticism is essential in order to avoid being duped:

“In doing research on the effectiveness of various e-learning media, we ran across some useful research that is commonly quoted across the Internet:

• According to Albert Mehrabian, 55% of what we communicate is through body language, 38% is through tone of voice, and 7% is through words.
• As illustrated in The Learning Pyramid, after two weeks people tend to remember 10% of what they read, 20% of what they hear, 30% of what they see, 50% of what they hear and see, 70% of what they say, and 90% of what they say and do.
• Researchers at Simon Fraser University found that the average continuous attention span for literate humans is 8 seconds with a maximum of 30 seconds, and the average general attention span is from 10 to 12 minutes.”

After presenting these three critical sets of data regarding e-learning, the authors reveal that none of it is true! These are instead myths lurking on the web for anyone to stumble upon and consume or abuse. They provide three explanations for this dilemma:

1. Too many people automatically believe everything they read, especially if it’s online.
2. People don’t have the skills or don’t take the time to evaluate what they find online to ensure that it is true and in context.
3. Once people find the information or data that satisfies what they’re looking for, they seldom continue their search to find contrasting opinions.

Figures (especially ones from reputable sources) can be presented in a very compelling way and can be used to move buyers toward a decision. But how are these figures arrived at, by whom, how long ago, and under what circumstances? What is opinion, and what is fact?

7 Suggestions for Vetting Sales Stats and Figures

As suggested by The Data Vigilante, if data looks too good to be true, it probably is.

Few of us have the resources, time, budget, or inclination to conduct our own scientific, peer-reviewed research on a given topic. Therefore, we accept or carefully scrutinize the stats put forth by others in an effort to make better-informed decisions.

You can’t personally vet each set of figures, though. So, you must approach each with balanced trust and skepticism. This goes both for consumers (BtoB or BtoC) receiving the data as well as sales and marketers to either conduct their own original research or collect and package stats to bolster their pitches.

1. Source —Is it reputable? Have you ever heard of it? Anyone can whip up a quick poll on SurveyMonkey these days. You don’t always have to cite a McKinsey or Harvard Medical School study, but be careful not to bet your sale on a random blogger’s unsubstantiated poll.

Is the source the company providing them? Have they conducted their own study among their clients or users? That can be okay, as long as they don’t mask their involvement.

2. Objectivity — Not just questioning whether the source is reputable, but considering what the source has to gain by conducting and promoting the study. Is there any risk or interpretation of bias involved? Who gains from the outcome? Even third parties can benefit.

3. Sample Size —It’s often said that you don’t need to ask more than 100 people the same question to get a meaningful data set. That may be true, and there certainly is a point after which gathering more data becomes meaningless because it fails to influence the outcome. but it is important to have quality respondents in your data set that represent their demographic.

For example, it’s understandable that a survey of Fortune 100 CEOs may only include 15 responses because of how busy they are and difficult to reach. But if you’re trying to survey heads of sales, HR, or accounting from the same Fortune 100 companies, one would expect you to have greater participation rates for the data to be of value.

4. Age —How long ago was the study conducted? Is it still meaningful? Some data sets can stand up for many years between studies, while others go stale in much less time. This is particularly true of anything driven by technology devices and habits.

5. Relevance and Context —Do the numbers and findings make sense for how you are using them? Can someone question the connection between the numbers you’ve cited and the message you’re attempting to bolster or undercut? Don’t stretch the truth to suit your needs — if discovered, you’ll lose credibility quickly.

6. Common or Unique Data Set —Has this study been conducted often, or is it a one-of-a-kind analysis? Do the findings align with previous studies or are they worlds apart? If the latter, can you satisfactorily justify or explain the difference?

7. Defensible — Be able to defend your stats or don’t use them. Whether they’re your numbers or someone else’s, if you cannot easily defend them or justify their relationship to your message, then they should not be used. Find better stats to cite or else risk that these questionable figures will serve as a distraction or undermine your credibility.

As a sales rep, when using data in a pitch or presentation, be conscious of avoiding manipulating data for fear-mongering or for leading your audience toward a mirage of false promises or hopes. If you aren’t, you’re just as likely to lose them as you are to impress them. And so, in addition to the points mentioned above, I’d also suggest these final two points:

• Beware the Data Barrage — Don’t overwhelm or numb your audience with too many numbers and stats. Your audience may feel bullied if you throw too much at them at once.
• Sell with Substance —Don’t expect to hang your entire presentation or pitch on a string of impressive stats. Even if they’re rock solid, you still need to be able to show them what you can do to make their company better or simpler.