Experiments and Hypotheses: How to Systematically Grow Your Business

The core goal of literally any business is constant, long-term, and predictable growth, but achieving that is no easy task.

Regardless of the scale and size of a company, one particular approach is often successfully applied: growing by experimenting.

What does experimenting mean in business? Why and how does it work? Let’s dive in, step by step.

The Price of Uncertainty

Whether you're starting a new business from scratch or growing your existing one, there's always a scope of uncertainty.

You may not know how to promote your new product, how to price it, how different regions or countries may perceive your USP, or how the UX will perform. You may be in the process of finding new ways to improve the profits (increase ROI and retention, decrease CAC, etc.) of your existing business.

In every case, the set of unknown parameters will be unique, but it's always significant.

Every time you step into a new field with a new product, service, or marketing challenge, uncertainty comes on stage.

The only thing you can rely on is the actual practice-backed data:

• Which price points converted better
• Which variant of a landing page or paywall worked better
• Conversion rates
• How the LTV or CAC changed
• Feedback, and so on

The more experiments you run, the more actual data you receive, the more control you gain over your business, and the more you may rely on your success.

Conversely, the more you rely on your conjectures, the less accuracy you get, the more money you lose, and the higher your chances of failure.

Now let's see how it works.

How to Formulate Your Hypotheses

Every time you make a suggestion, it is a hypothesis.

"I think this is their core issue we may solve."

"I think these pricing packages may work well."

"I think this UX will be easy to understand."

Basically, a hypothesis is any suggestion not backed by data that is about to be tried and tested.

But it's important to formulate your thoughts in the right way so that a suggestion will turn into a real hypothesis to test.

How to formulate a hypothesis

Well-formed business hypotheses are:

• precise,
• discrete,
• and testable.

This means that in every single case you should be aiming to test one specific KPI, rather than an abstract mass of related metrics. For example:

In the formulation on the right, you can find a target audience and a specified KPI, which makes two out of three. Of course, you should understand exactly how you are going to test that.

What Is an Experiment?

An experiment is the actual process of testing your hypothesis and getting a real-life market response.

Before you start, it is important to understand a few factors:

• Test only one thing at a time. Of course, you may run a series of simultaneous experiments, but they should not interfere with each other (e.g., don't test a CTA and a title on the same page.)
• Not every experiment is supposed to be successful. It's not a win versus a fail; it's a process of checking whether every particular idea works, on a smaller scale. Most of your experiments may be unsuccessful, and that’s absolutely okay. But some of them will bring you results, and these will form the basis of your future marketing or improvements.
• Tests need time. Don't make any conclusions until you get enough impressions, because on a smaller scale, the data may be irrelevant, and preliminary conclusions may change several times.

Now let's see how it actually works.

Types of experiments

There are two main ways to run an experiment:

• A/B testing (split testing). You compare two different versions of what you test (e.g., one CTA versus another, or one color scheme versus another). These may be two new untested things or an improvement versus an old option.
• Multivariate testing. This type of test compares multiple versions within the same principle.

A/B tests are applicable to any kind of business, while multivariate tests are usually used by large companies with huge volumes of organic traffic in order to get trustworthy results.

How to run an experiment

At the very first step, you will formulate the experimentation process.

There are different approaches to that, but the overall principle is always the same:

• Write down the hypothesis you are going to test.
• Define the KPIs you are going to measure. There may be one core KPI and several additional ones.

For each KPI, write down:

• Current level (if applicable)
• Target level of improvement
• Define exactly how the test will be run.
• Specify the duration of the experiment, required resources, and further actions.

For example:

Experimenting should be a continuous cycle. This means you should always be generating new hypotheses and testing not only new ideas but also what has already been tested before.

There is no limit to perfection, which means every result can be improved.

How to Prioritize Your Hypotheses

In the beginning, you might struggle to ideate new things to test. But once you start, more and more ideas will come to your mind, as well as to the minds of your teammates. So, you may come up with a long list of ideas very quickly.

But how do you choose the right one to run first?

There's a universal formula used by many companies to prioritize hypotheses called ICE, which consists of:

Impact. What is the potential upside of a successful experiment? Assign scores according to the expected value of the impact, for example:

• 1 - 2 — very low (You likely won’t get much additional revenue.)
• 3 - 4 — low
• 5 - 6 — average
• 7 - 8 — high
• 9 - 10 — very high (Such hypotheses may bring you a lot of profit and impact your whole business or its valuable parts.)

Confidence. How likely is this experiment to succeed? Score it following the same principle. To help you in that, a very useful framework is the Gilad Confidence Meter. It can be easily adapted to your scale.

Ease. How easy is it to run? If it requires many people or a lot of time and other resources, your "Ease" will be low, and vice versa. So, here it will be:

• 1 - 2 — very difficult (You will probably need to spend a lot of time and attract many people to run this as an experiment.)
• 3 - 4 — difficult
• 5 - 6 — average
• 7 - 8 — easy
• 9 - 10 — very easy (You can run this experiment at almost no cost with little effort.)

Each parameter may be scored from 1 to 5, from 1 to 10, from 1 to 100, or by any other scale that is convenient for you and your team.

Want to read full article? Subscribe to my blog (it's free) 👇

To count your ICE for a specific hypothesis, use this simple formula:

For example, if your Impact is very high (5), your Confidence is low (2), and your Ease is average (3), your ICE score will be:

Based on the ICE score, choose hypotheses with a higher potential impact. The higher the ICE, the higher the priority.

Here’s a template you can use to score your hypotheses easier:

In this example, the first hypothesis has the highest priority and is worth trying out in the first turn.

Analyzing Experiments

Now let's see how exactly you can measure the impact of an experiment and decide whether it was successful or not.

Based on that, you will be able to consider your next actions in improving your business and running experiments.

How to collect data

Depending on the exact product or service (and also on the essence of your experiment), there might be different ways and tools to collect data.

For example:

• Google Optimize to run and analyze web-based experiments
• Google Analytics to analyze your website traffic
• Google Search Console and Ahrefs for SEO analytics
• Adapty to analyze your mobile app monetization data
• Mailchimp or another similar tool to run and analyze email marketing experiments

Using a relevant tool, you will gather raw data and be ready to make conclusions.

How to analyze your experiment’s data

It's important to run an experiment until a representative amount of data has been gathered; otherwise, there's a high chance you’ll get a false result:

• False Positive: when data seems to prove your suggestion but changes over time
• False Negative: when data seems to refute your hypothesis but also changes once more participants (visitors, users, buyers, etc.) join

Analyzing the results of your experiment is pretty straightforward:

• You take your data, extracted from an analytics tool;
• You take the initial formulation of the experiment;
• You compare the actual data to the expected result;
• You make a conclusion.

It's useful to not only define whether an experiment was successful or not, but also to write down a broader conclusion on how it went, what worked and what didn't, and what actions should come next.

Applying & communicating experiment outcomes

When analyzing an experiment, you get to learn about your growth model and your users' psychology. There are three levels of analysis:

• Success/Failure: It's the most superficial level of understanding where you define whether the outcome matched your hypothesis. Based on that, you will decide whether to continue this new type of action or return to the old one.
• Accuracy: It's how close your calculated results are to the experiment's actual value, including the margin of error. If you use reliable methods to gather and analyze data and make sure nothing interferes with the results during the experiment execution, you increase accuracy. Analyzing and improving accuracy will not only make your tests reliable, but it also gives you an idea about what factors are worth testing in the future.
• The Why: Analyzing why exactly an experiment was successful or not is probably the most important part of your analysis. It allows you to take what you've learned and apply it in new and creative ways. This is where you come up with new hypotheses for your next experiment.

After this cycle of building hypotheses, experimenting, and analyzing outcomes, it's time for implementation and running new experiments.

Common Mistakes to Avoid

Here are some practice-based insights and issues to avoid that will help you achieve better, more accurate, and useful results when running your experiments.

• Wrong Measurements: The most common mistake is setting up an experiment with the wrong measurement metric that is not 100% relevant or critical in this particular case. This can lead to a false positive or negative result and affect your decision-making process in the future.
• Statistical Significance: Another common mistake is not understanding or misinterpreting statistical significance. This can lead you to make decisions based on insignificant data and affect your ability to accurately measure results.
• Contextual Data: Not including all relevant contextual data in an experiment can lead to incorrect results. Contextual data is information that helps explain why something happened or what might have happened if the experimental environment had been different.
• Collecting Too Much Data: Collecting too much data can lead to a loss of focus and affect your ability to accurately interpret the results. This is often caused by people wanting to be sure that their experiment is 100% precise, but in reality, they get too much information that blurs their vision.
• Oversimplifying the Process: This happens when people try to make their experiments as simple as possible, without taking into account all of the potential factors that could affect the outcome. This may also affect the result.

Conclusion

As your business grows, you will want to experiment more and come up with hypotheses about what is working (and what isn’t) so you can make the most of your efforts. By understanding how experiments and hypotheses work, you can streamline this process and continue growing your business in a deliberate way.

Experimentation will help you improve your product by understanding what features or changes your customers want and need. Additionally, it can also help increase engagement and drive referrals by helping you understand how customers interact with your product.