Junk science in life coaching: rationalizing views through questionable research

“Without data, you’re just another guy with an opinion.”

That’s what my boss told me many years ago, when one of his peers disagreed with a major change I was recommending.

My opinion was well formed. I’d worked in the field for a decade, I was well regarded, and I’d had many successes. I’d developed a keen awareness, insight, and understanding of the work.

None of that mattered. The other guy had more power, so if it was just my opinion against his, then his would win… no matter what the truth was.

Fortunately for me, I had the data.

Not just any data. Clean data. Clear data. Data that answered the questions on the table.

And that made all the difference because my opponent was a rational guy who knew how to see through junk research. He revised his opinion and agreed with me when he saw the solid evidence.

Don’t create evidence to support the outcome you want

It’s reassuring to know that something you believe is backed up by research.

I believe, for example, that life coaching can be profoundly life-changing for a client. I believe this because I have experienced it, I have witnessed it, and I make it happen. Lots of stories and lived experience have led to this belief.

But wouldn’t it also be great to be able to share clear, clean data that proves it?

If you’ve read my blog for any time at all, you may have noticed that I rarely cite such research. There’s an important reason for that.

Most people can’t tell the difference between science and rationalization.

Science seeks truth through observation. Rationalization seeks observation to support belief.

So much of the “research” I see out there looks a lot more like rationalization than science.

A popular book I have come to hate

I’ve been reading a book recently about human behavior. This book has been widely praised, and the release of its latest edition generated much coverage.

I am not naming the book in this post because (a) I don’t want to seem like I’m capitalizing on its popularity for SEO purposes, and (b) my post is about critical thinking, not about that book.

I already despise this book, even though I agree with most of the author’s conclusions. I’m only about 25% into it. It might get better. But I have little hope.

One reason I hate this book is that it’s ten times as long as it needs to be. The author consistently uses 10,000 words when 1,000 words would suffice. And that’s by design, as you’ll see later.

But I’m even more mad about the author’s cavalier use of rationalization disguised as science.

Three ways this book makes rationalization look like science

Most of what I’ve read in this book goes like this:

1. Someone did an experiment, and here is what happened.
2. That thing happened because of X.
3. X is therefore a universal truth.

That seems okay, right? Well… it depends.

If you’re not coming to this book in a skeptical mindset, then you may just roll straight through from step 1 to step 2 without thinking too deeply, and then follow on to step 3.

I would guess that most readers are not treating the research with skepticism. People who buy this book are looking for answers, and the author is giving them the answers they’re looking for.

It turns out, however, that there are problems in the way the author moves from step 1 to step 2 to step 3.

Huge problems. Three in particular I want to highlight.

1. Legitimizing junk data by mixing it with scientific research

First, the author mixes legitimate research with anecdote as if they carry equivalent weight.

This happens in Step 1 (“someone did an experiment”).

In one chapter he describes a good experiment done decades ago. The researchers observed statistically relevant results which illustrated a clear trend in people’s behavior under very tightly controlled conditions. Pretty good science, actually.

In the next chapter, he cites an informal poll another scientist did at their own small dinner party. This other scientist is well regarded, and the result of their poll was both interesting and amusing. A fun story to read, but not at all science.

If the reader is not paying particularly close attention—which they’re not, because they’re trying to slog through 10,000 words when 1,000 would have sufficed—then they might later remember the dinner party as having the same legitimacy as the scientific experiment.

In fact, the author is counting on it. Because he’s mixing junk data with real data to rationalize his opinion in the reader’s mind.

2. Declaring “it’s because of X” without any proof

There’s a difference between saying “we saw this result from the experiment” and saying “this result from the experiment is because of reason.”

This occurs in the transition point from Step 1 (“here’s what happened”) to Step 2 (“it happened because of X”).

It a bit of sleight-of-hand that happens frequently in sales meetings. The salesman presents you some trustable data. Then they make a claim for why.

Since the data is good, you don’t stop to question the subsequent claim.

“Ninety-eight percent of surveyed users were highly satisfied with their user experience. That’s because the product is the easiest to use on the market.”

That’s a real statement (paraphrased) I once heard from a vendor. The data point was true and accurate—they had the survey results to back it up. Their conclusion, however, was dead wrong. They were asking users only after a successful transaction, so anyone who had a bad experience never even saw the survey.

Because I knew I was in a sales situation with a vendor trying to get a big contract, I was already in a very skeptical frame of mind and looking for ways they might be trying to influence my decision.

People don’t buy a highly publicized book by a known author with that kind of skeptical mindset. They come to this book looking for answers from a trusted source. Readers are ready to accept what they’re being told.

In the book, the author frequently uses this sleight-of-hand approach. He cites a reliable result from a legitimate experiment, then makes an unsupported logical leap to “this is why.”

Because we’re still fighting our way through 10,000 words where 1,000 would suffice, we don’t slow down to question the conclusion. It feels right, after all, since we’re looking for answers and the answer either (a) fits our predisposition or (b) feels surprisingly insightful.

3. Oversimplifying astoundingly complex things and ignoring inconvenient variables

Third, the author draws some very short, very straight lines between a small bit of useful data and some very sweeping conclusions about human nature.

This is Step 3 (“X is therefore a universal truth”).

Again, because you’re trying to swim through 10,000 words when 1,000 would suffice, you may just roll right along with the author and not notice the complexity he conveniently ignores.

One of the experiments he cites provides a very good example of this. This experiment took place in two different stores of the same retail chain. In one store, guests were greeted one way, and in the other another way. The results showed a statistically relevant difference in the shopping behaviors of the guests at the two different stores.

This certainly is very valuable and potentially actionable data for the owner of the retail chain. If the author’s point had been that focused research can lead to actionable data, I’d be all in.

But if you’re going to claim that you’ve discovered some fundamental truth of human nature, then I have some questions.

Where, physically, were these stores located? Did people have to drive to get there? Was the weather the same in both places during data collection? Were they located in a rural or urban area? What level of disposable income, debt, and wealth did the typical shopper have? What cultural or faith biases and influences did shoppers bring with them into the store? What type of thing were shoppers coming in for, typically? Wrenches? Milk? Socks? Automobiles? Greeting cards? Were they young or old? Did they have children? Grandchildren? Chronic illness or good health?

The author implied that he’d uncovered a universal truth about all humans, everywhere, in every situation. And, if you were predisposed to agree with that truth, then you might now think you had data to back it up.

After all, it’s honest data. It’s just disingenuous analysis.

Telling the difference between research and rationalization

I’m a big fan of research. I love to look into data and find the patterns that emerge. I feel delight and joy when I see patterns that surprise me, when the data shakes up some idea I had held faith in.

After all, without data, I’m just another guy with an opinion. And I prefer my opinions backed up by truth.

But problems occur when people start with an opinion and go looking for data to support it.

They lose rigor. They lose perspective. They lose skepticism. They become susceptible to charlatans eager to separate them from their money. (I regret I bought that book, which was not cheap.)

That’s not science. That’s not research. That’s rationalization.

Science is about forming beliefs based on observation. Rationalization is about selecting observations that support belief.

Intellectual integrity is important to me

At the top of this post I pointed out that I rarely cite research on my blog.

Read all my posts, and you’ll see that I often make sweeping statements and then illustrate my points with story, anecdote, and lived experience.

I sometimes cite research. I often refer to cognitive biases. (One of my favorites is the Dunning-Kruger Effect.) But it’s rare for me to claim anything is a universal truth in human behavior. We’re just too complex for that kind of generalization.

I’m in the coaching space. That means I’m here to help people reach their highest goals.

I believe deeply in the power of coaching to change people’s lives. I believe in it because I see it all the time. I’ve experienced it. I make it happen. My clients attest to it.

Despite the observable, anecdotal evidence that coaching has real value, we as a profession crave scientific proof. We are desperate for studies that prove our beliefs beyond any argument.

There is some legitimate, valuable research being done that gives us a better understanding of how people work. More will be done, I have no doubt.

But this desire for validation leads to two unfortunate results: (a) the creation of junk science or even outright falsehood, and (b) gurus and authors who make claims of scientific proof that simply aren’t backed up by actual research.

Yes, I understand that I just made a pretty broad, sweeping claim about human nature in that last paragraph.

And I understand that without the data to back it up, I’m just another guy with an opinion. So evaluate it on its merits, apply your own skepticism and rational thinking, and decide for yourself.

It does not have to be universal to be true

And that’s the whole point.

I won’t try to convince people that I have scientific proof of my beliefs if that proof isn’t clear, clean, and unassailable.

One of my beliefs is that life coaching can be profoundly life-changing. I believe this because I have experienced it, I have witnessed it, and I make it happen for my clients. Lots of stories and lived experience have led to that belief.

It sure would be great to have clear, clean, unassailable data that proves it’s true for all humans, everywhere, in every situation.

But it doesn’t need to be true for all people, everywhere, in every situation in order for it to be true for my clients.

If I help them clarify their lives and achieve their goals, then that’s all the evidence I need to feel good about my belief.

I can help.

I work with top executives and middle managers to improve their leadership skills, their workplace culture, and the effectiveness of their teams. Also, I help individuals identify and achieve their personal goals. Would you like to become more aware, be more effective, be more empowered, and feel fully prepared for your next steps?

Let’s talk.

You can help.

Think of one person who would benefit from reading this post. Sharing is caring! Forward it to them right now. They will think you’re super smart and well informed.

Stay informed.

Be sure to join my email list! Get notified of new posts here as well as new courses, books, and events from me both here and at Gray Bear Publications.