The following is my critique of a proposed lean test, as well as some ideas on when to kill a lean test off if it isn’t working.
The proposed lean test
The entrepreneur wants to convert their company’s home page banner into a newsletter capture opportunity instead of a ‘Get a Free Session’ opportunity.. this will allow them to capture emails at a faster clip
Parents need to sign up for the newsletter and receive valuable content before signing their teen up for a free session with one of our agents. We believe that after collecting 100 parent emails from our homepage banner we can get 10 parents/teens to sign up for a free session.
Gather 100 parent emails through various channels and provide an email with value (our free best selling book, webinar opportunities, our favorite blog posts). Once parent receives email we will follow up and ask if he/she would like a free session.
Metrics for pass/fail:
Pass = 10 or more parents sign up for a free session
Fail = Less than 10 sign up for a free session
First, the test is proposed perfectly. There is the hypothesis and what the authors suppose to be true. There is the experiment, or how to test the the validity of the hypothesis. And finally there are metrics, which determine whether the test passes or fails.
Second, my response.
This test is pretty close to being good. I think you are trying to test whether changing the home page banner from a quick kill (get a free session) to an info ask (email capture) will improve bottom line conversions (parents signing up for a free lesson). The test is essentially whether your clients need to be nurtured/given information before being closed. That’s a good thing to find out and has direct implications on your sales channel. (Next time, can you include screenshots too please?)
Response from the entrepreneur – aka “When do we move on from the idea”
The #2 point is what trips me up.. it is quite arbitrary, but honestly anything is better than our current home page banner. This is a test of a business model as well, and there is not a guarantee that the market actually needs it (although we believe so). My question through all of this is ‘when do we know this idea needs to be crumbled up and move on to the next one?’
Moving on – consider the opportunity cost
My advice is to approach this from an opportunity cost perspective. I don’t think an idea can ever be really “killed”, but it can be lowered down the backlog. What that means is other ideas have a better chance of success, so we’ll downgrade the current idea until there is another chance/more resources/change in the marketplace.
I suggest using a pre-commitment mechanism to decide when to move on. That means drawing an emotional line in the sand before the test. Whatever “feels” right before the test is what you should commit to. I think the challenge you have right now is that you don’t have enough data, so you don’t know what really looks like a pass or a fail. In that case, you need to start running more experiments! After you have a few months under your belt, and much more exposure to the marketplace, you’ll have more data and be in a stronger position.
Here’s another way to think about it: Most entrepreneurs look for any glimmer of hope they can find. But don’t look for glimmers of hope. Look for a smack in the face!! Don’t decide whether a 3% or 5% click through is sufficient. Keep going until you get a 25-50% click through! If people really want something, that’ll be the sign you’re looking for.