Geo Holdout Testing: What It Is and How to Use It in Your Marketing

The wealth of data available to modern marketers makes it easier than ever to optimize advertising campaigns based on real-life information.

To find out which version of an ad creative is most effective, often marketers will turn to A/B testing—essentially using a scientific approach of showing a test group one version of an ad and a control group another then measuring the difference in performance between the two groups.

One specific kind of A/B testing that can yield insights about the incrementality of your marketing is geo holdout testing.

In this blog, we’ll cover the basics of geo holdout testing, why you would want to run geo holdout tests, and how to work with Rockerbox to measure their impact.

What is geo holdout testing?

Geo holdout testing is similar to A/B testing, but it differs in that instead of comparing an arbitrary control and treatment group, it compares two different geographies. One geographic area is exposed to the advertising being tested and the “holdout group” is not. The results of these tests essentially show whether the advertising campaign in question had any effect on the selected KPIs.

What can a geo holdout test tell you?

The results of a geo holdout test can give you important insights into whether your advertising is contributing to revenue or whether you’d see the same results without the additional spending. Some examples of the specific things you can discover through geo holdout testing include:

• Overall: What is this channel's impact on overall revenue? Conversely, if we stopped spending on this channel, what would happen to revenue?
• Channel-specific: How much revenue is directly tied to this channel? Am I spending too much, or not enough?
• Halo impact: How does cutting spend in one channel impact all other channel's revenue and performance?

Best practices for setting up a geo holdout test

Rockerbox clients use the geo holdout approach to evaluate the impact of channels like linear TV, OTT, YouTube, OOH, and radio, and they work with Rockerbox CSMs to measure the results. Based on our expertise, we recommend several best practices when preparing to run a geo holdout test.

1. First, identify the test and control markets. When selecting which markets to use, you don’t have to pick same-size areas, but they should behave similarly in terms of:

• Customer composition
• Marketing channel performance
• Overall conversion rate
• Overall growth rate
• Index against total addressable market in each geo

2. Make sure that the geographic areas you’re testing are substantial enough and have enough conversions and ad spend to make the test statistically significant.

3. Consider the timeframe you’re using and make sure there are no seasonal events, macro events, or significant marketing strategy changes (e.g. adding in a new channel during the testing period) that might impact the results.

4. During the test, compare the two groups, test and holdout, across multiple time periods. First, compare the two before any changes to the marketing mix in order to ensure they behave similarly. Then look at activity when the change to spend/marketing mix is implemented, and also after testing to measure latent impact.

Measuring the results of a geo holdout test with Rockerbox

Geo-holdout tests can be executed in most ad platforms, but looking at a channel in isolation can overstate the impact of the channel you tested and cause you to lose visibility into the halo impact of any changes you made.

When you work with a Rockerbox CSM to analyze the results of a geo holdout test, we can leverage Rockerbox data to dig into the impact of the test—in the context of all your other marketing channels—and answer multiple questions, such as:

• What was the impact of the test on demand-driven channels, such as branded search and organic search?
• Did the heavy up in spend result in reaching net new users who were not previously being reached at lower spend levels?
• Did the test result in diminishing returns to ROAS?

In the end Rockerbox can deliver detailed insights on the test’s performance that can tell you whether your ad spend is truly making the impact you think it is.

Choosing Rockerbox as a measurement platform means you have an always-on solution that gathers data on your marketing activities across channels and geographies. There’s an opportunity to uncover incredible insights through Rockerbox data, and CSMs are here to help you make sense of it.