back home

You can’t split test human experience

Written by Justin on January 25, 2016

You can't split-test human experience

The problem with advice is that it’s not testable.

When you ask someone for their secret to success they’ll say things like: “This class changed my life,” or “I get up and meditate every morning at 3am.”

Attributing an individual’s success to one or two factors is a fool’s bet (especially if you try to replicate it). No one can say emphatically “this is why I’m successful” because there’s no way to test that claim.

Jen might think her success in business was a result of going to Harvard. That could be true. But because there’s only one Jen, and she can only live one life, we’ll never know.

In the software world we can run A/B tests to check our assumptions. You can say: “I think orange buttons convert better than green ones” and then you can run a test and see if it’s true.

To test Jen’s statement we’d need to duplicate Jen and have one version go to Harvard, and another go to a different school. In fact, we’d probably want multiple versions of Jen all going to different colleges at the same time.

So what should we do with advice?

Listen, try it out, and iterate to make it your own.

There’s nothing wrong with saying: “I’m going to test this out.” You might find that what worked for someone else, also works for you. You might find it only gives you 50% of the results you need, but when you combine it with something else, it succeeds. Or, you might find that in your context, it doesn’t work.

How should we give advice?

My friend Paul Jarvis gives this disclaimer when doling out words of wisdom:

This might not work for you.

Let’s keep that in mind; whether we’re giving or receiving advice.

Product & Traction

Get product marketing tactics every week.
Designed for SaaS, software, and digital product businesses.

No spam. Unsubscribe anytime. Powered by ConvertKit
Awww yeaaah! Looks like you're already on my mailing list! Have you subscribed to my YouTube channel?