2014 was the year of A/B testing.

2015 was the year of North Star Metrics.

2016 introduced us to conversion rate optimization.

2017 meditated on user churn through customer success.

I’d like to tease out the good, the bad, and the ugly from each of these themes for the budding SaaS entrepreneur of 2021.

A/B Testing

Delivering slightly varied web page content to prospects and measuring difference in impact on completed actions.

Good:

Minimum 10,000 unique visitors per month, no more than 1–2 tests running at a time, split A/B variants (vs A-B-C-D), two-tailed reporting, and examination of large themes like “require credit card to sign up vs don’t.”

Bad:

Single-tailed tests that mischaracterize experiment results, < 10k /month traffic that leads to 1+ month execution to measure statistically significant results.

Ugly:

Demonstrably different traffic sources or traffic qualities over short spans of time. For example, a Facebook ad campaign that runs for 48 hours, then a press hit on TechCrunch. Check key metrics inside Google Analytics (average session length, signup conversion rate, etc.) and look for similar numbers on a last week, last month, last 2 months basis. Testing small themes like “green vs red buttons” or font size.

In summary

Many early stage startups are wasting time running A/B tests. This is the part of company growth where your gut matters. How did thousands of successful companies start-up before Optimizely? Here’s how: in-house market expertise, customer development data, and ability to iterate quickly. Charts that tell you if 1 tagline worked 7% better than another will not get you to product-market fit.

North Star Metrics

A singular KPI, around which the entire organization rallies its product and marketing efforts.

Good:

Easy to measure, motivates your team, and keeps investors happy (if applicable). Improves decision-making speed, because there’s less to analyze.

Bad:

Difficult or time-consuming to measure, such as a complicated “JOIN” analysis between multiple sources of unstructured data. “Vanity” metric that is easy to grow, but exponential by nature vs linear, thus skewing overall perception of business.

Ugly:

Metric that is not correlated with business success, i.e. Free Trials vs Free Trial Conversions. If multiple data points need to be compared to form a North Star Metric, the yielded insight is not a North Star Metric. This is only an OK reality.

In summary

Some businesses can get away with tracing 1–2 metrics, and nothing else. For the rest of us, spending a few extra minutes every week to analyze how multiple metrics tell the full story of your product’s health is plain smart.

Conversion Rate Optimization

Ongoing improvements to user discovery, on-boarding, and retention that increase overall customer lifetime value.

Good:

Data driven best practices like fast page loads, social proof, live chat, and hand-crafted first time user experiences.

Bad:

Popups that aggressively ask for emails when someone hovers to the top-left of their screen, creepy marketing automation that tries to guess exactly who someone is before information is requested.

Ugly:

10,000 word landing pages that require a phone call appointment to learn about pricing or product. Pushy sales automations or alarmist cold email subject lines.

In summary

If at least 3–5% of your traffic completes an import behavior, like starting a free trial — great. Spending multiple weeks on campaigns to improve conversions 0.25%, when you have other problems, is sub-optimal.

Churn

Users lost during a period of time, often expressed in monthly intervals.

Good:

Measuring churn by user context, i.e. “On-boarding” vs “Retention.” Monthly task list of 3–5 tactics that aim to reduce churn in a given category. If on-boarding, an extended (or protracted) free trial could help. If retention, an annual discount offer at the 60 day mark would qualify.

Bad:

Brand-damaging discounts, like “50% off if you upgrade TODAY” offers that would upset current paying customers if they found out about it. Making users click more than once to cancel.

Ugly:

Phone call requirement to cancel an account, annual commitment requirements, cancellation fees, hostile customer service.

In summary

Not only does retaining customers afford you more budget for future customer acquisition, but it also starves your competitors of prospects. Every customer you have is a customer your competitors can’t touch, making the “fixed pie” that much smaller and increasing your defensibility.

Picking your SaaS Battles in 2021

I firmly believe — and have demonstrated at Fomo — that someone can build a multi-million dollar SaaS company with just 200–300 unique visitors per day.

To prove it, here’s the traffic at Fomo over a 12 month span:

‍