Hi, everyone, my name is Stephanie Menz, and I'm on the customer marketing team at Toast. I’m so excited to share with you what I’ve learned about using A/B testing to help define your target audience.
Here's the agenda: we're going to do a quick intro, followed by how to define your upsell use case and upsell funnel, and then we'll dive into creative testing and audience testing.
I’ve structured this article as a tool that I wish I’d had a few years ago when I started working on customer upsell, so I hope you’ll find it helpful. We'll look at how to create A/B tests, how to create segment-specific campaigns, how to refine your target audience, how to use data to inform your upsell targeting strategy, and ways you can personalize segment-specific campaigns.
Top takeaways from the article:
- How to create A/B tests
- How to create segment-specific campaigns
- How to refine target audience
- How to use data to inform upsell targeting strategy
- How to personalize content
What is Toast?
Before getting started, I want to give a quick overview of Toast. Toast provides a powerful online platform built for restaurants by restaurant people – it's better for restauranteurs, their guests, and their staff. We help restauranteurs to do everything from running the restaurant efficiently to managing a productive team, growing their business, online attracting and retaining guests, and more.
Defining your upsell use case
Different types of upsells serve different purposes and ways to sell solutions to your customer base. For example, there are new product launches; there’s also product-led growth as a channel; then there’s customer success, which customer marketers can use for upsells.
Customer Marketing Upsell
- New product launch - Goal to build new upsell funnel, define key audiences, messaging, and channels.
- Product-led growth
- Customer Success team
- And more.
I'm going to dive into the use case around the launch of a new product that can serve all customers and show how I use A/B testing to find that target audience. Our main goal is to generate higher quality leads from our customer base, so we can have higher conversion throughout our upsell funnel.
Defining the upsell funnel
To get started, we first need to define that upsell funnel. Here's an example of what your upsell funnel might look like.
It may look different in your organization or various add-on products, but it's important to go through this exercise for consensus across key stakeholders and to make sure everyone's speaking the same language.
At the top of your funnel, you have your total addressable market, which is your customer base that is fit for the product. You have your hand-raisers – the customers who show their interest in the product. If your customers need a demo before purchasing, that’s the next stage. The final stage is the win.
Creative testing
Now that we’ve defined our funnel, it's time to get started with creative testing, but where to begin? When I went through this exercise, I wanted to start with the basics and make sure I had the positioning nailed down for this product; then I wanted to move on to the best audience segmentation that I could develop.
I like to think of refining the target audience as another funnel. You want to start broad and then refine to the point where you end up with customer personas for this solution.
My first step was to start testing messaging our customer base. Using the messaging framework from product marketing, you can come up with an email to send to your customer base and then start testing the subject line.
Email is a great channel for testing messaging as it's very scalable and the results are usually pretty easy to digest. I'm starting with the subject line because, ultimately, I want customers to open the email and I want to know how I can get them to open it, so they buy the product and become a hand-raiser.
Testing email subject lines
When creating your subject line A/B test, you want to make sure you have a control version of your subject line. Let’s say for test one I want to test personalization in the subject line – subject line A will have the customer's name, and subject line B will not have the customer's name; apart from that, they’re the same. This way, we can get an easy read on which subject line is the winner.
Maybe we're hosting a webinar around the product – we can start testing “Stephanie's Bakery, are you free next week?” (variation A) versus “Hi there, are you next week?” (variation B). The winning subject line in this example is variation A, and that will become the control variation for test two.
For test two, we can experiment with including the date and time of the webinar along with the personalization token. This time, variation A is “Stephanie's Bakery, are you free on October 5th?” and variation B is “Stephanie's Bakery, are you free next week?” From here, we can see which one's the winner and keep testing our customer base.
Types of subject line tests
Here are some of my favorite subject line tests. This is by no means an exhaustive list – there are so many different tests you can do with subject lines – but I want to highlight a couple.
Examples of different types of subject lines to test:
- Personalization tokens: Adding company name or person name
- Value proposition: Short, actionable subject line that includes value prop
- Seasonal: Back to school, holiday messaging, new years goal setting
- Date/time: Webinar details
One is personalization tokens – adding the company name or the person's name to the subject line to see if that leads to an increase in open rates.
I also like to experiment with value propositions within the subject line, making sure that it stays short and actionable.
Seasonality is another great one. Maybe your solution helps with customer pain points around going back to school or the holidays – you could test messaging there versus more generic messaging to see if seasonality influences open rates.
Think about date and time as well. As we saw a moment ago, you could use the subject line to tell your customer when a webinar or event is happening.
Testing calls to action
After identifying which subject line drives customers to open their email, we have a good opportunity to start testing which calls to action drive customers to take the behavior we want them to take – becoming a hand raiser and ultimately purchasing the solution. At this point in our testing journey, we're still targeting a broad audience with broad messaging; we're at the top of that audience funnel.
We can also start testing other channels at this point, such as in-product messages. You can use the winning subject line from your previous test as a headline for your in-product message.
Types of CTA tests
Identifying the best CTA is great because you can then test it across different channels within your company. Let’s look at a couple of CTA tests you may want to try within your organization.
One of my favorites is personalization, using terms like “my” and “your.” Here are some examples:
- Schedule my demo
- Chat with my customer success manager
- View my offer
Another CTA to test is one that creates a sense of urgency. You can say things like “apply now” or “get started now.”
Special promotions are another CTA you can test to see if they help drive customer behavior. For example, you might want to offer your customers 50% off.
Outside of the CTA copy, you can also test the placement and imagery of the CTA within the email. You might want to try having the CTA as a button, and look at whether it’s more effective to put it at the beginning or the end of the email.
Every company is different, and everyone's audience has different needs, so different messaging and positioning will resonate best between companies.
Testing messaging
To recap, we now have a good sense of what subject line will drive someone to open our emails and which call to action will drive someone to click a link in our email. Now's a good opportunity to test different messaging in our email.
We're still at the top of the funnel with our target audience. Our messaging is still pretty generic, and we have the same goal of the customer opening the email and taking an action.
Types of messaging to test
Here are a few of my favorite types of messaging to experiment with:
- A bulleted list of product benefits: This is just a simple list explaining the product, which you can test against displaying the benefits in paragraph format.
- A bulleted list of the acquisition process: This is a great way to set expectations with the customer.
- An overview video product of the product experience: Your video could include the product benefits, and you can A/B test it against the bulleted list of product benefits.
- Seasonal examples of problems the product can solve: If, for example, the holidays are coming up, you can do some sort of holiday play to see if you can increase the use of a gift card module among your customer base.
Audience testing
Now that we have a good feeling about our messaging and positioning, we can move into audience testing and refine our target audience. We're starting to move down the audience funnel, where we'll be digging into our data to create some hypotheses around which audiences we think will convert into customers.
As I went through this exercise at Toast, I prepped for the different attributes that I could test. To begin, I went through the current attachment data set for our product. That helped me understand which segments were converting in which regions.
I was then able to collaborate with our business intelligence team, who went deep into our customer data set to help me understand if additional products, time of year, or any other factors influenced customer attachment to this particular product. Together, we came up with hypotheses that we could test around target audiences.
How to create an audience A/B test
Let’s lay out some steps to think about as you go into your audience testing.
Step one: Dig into the data you have readily available to you. Are you able to see if some segments are converting at a higher rate than others? Maybe you're at a B2B company that sells across verticals – in that case, you can look into the conversion rates in healthcare versus IT. Compare regions and look at time as a customer and time in business too. Also, see what effect NPS has on attachment.
Step two: Define hypotheses. Say you’re focusing on New York City in a regional test. Your hypothesis could be that if you send customers content related to NYC (think case studies, bugs, or a local event that your product will help with) then you can expect a higher attachment from that cohort. You can then test that theory by sending New York-related content to your NYC cohort, leaving a subset out as the control group.
Step three: create content with the tools from your toolbox. As you go through this exercise, think of all the different tools in your toolbox that you can use. Do you have any relevant case studies? How about blog posts? Is there any messaging related to this segment that you can test in an email? The list goes on.
From the data analysis and looking at all these different attributes and segments, we uncovered high-intent segments for our testing needs. We defined them by restaurant type and region. Your segments might look a little different; the key is to define the verticals that you want to test.
A/B testing a segment-specific campaign
Now that we've decided which customer segments we want to hone in on and test, let's begin the testing process. Here's an example of an A/B testing schedule for a segment-specific campaign. The aim is to uncover the best mix of messaging, creative, and channels for this segment that will ultimately drive customers to convert.
While we're testing, we want to have a randomized control group from this segment; that way, we can measure the lift in conversion among the cohort receiving marketing campaigns versus those who aren’t.
We might have cohort A, which is the segment we're testing into that gets segment-specific messaging. Then we have cohort B, the segment we're testing into that will receive general messaging. Lastly, we have cohort C, a randomized group of customers that we're also going to test with the generic messaging. We want to measure attachment across those cohorts to understand if sending segment-specific content leads to higher conversion.
Let’s look at our quarterly testing schedule. For month one, we want to test messaging, positioning, and creative via email. After we have our messaging and positioning nailed down, we can start testing different channels, so in month two we test an in-product message channel. In month three, maybe we want to start testing a sales motion with our customer success team, so we can partner with them and enable them to help.
Continue refining the target audience
Going back to our audience test funnel, our audience test and our messaging are refined, and we're continuing to move forward with testing our hypotheses on these segments. We can also start expanding channels and doing segmentation tests within these channels.
As we further refine our audience tests, we can think about testing different creative treatments, including segment-related photos. We can look at different types of segment-related messaging, including use cases in emails or whichever channel we're using. We can also experiment with supporting content related to segment-specific needs, such as case studies, blog posts, and whatnot.
We also want to think about regional tests. Maybe there's a local event like a marathon or a festival that your solution helps customers with; we want to see if we can use that as a regional play. We could include a case study from that region and other supporting content related to that region's needs.
Depending on how large your audience is within these two segments, there may be an intersection where you can get even more specific and continue to create hypotheses across these two high-intent high-conversion audiences.
Test your personas
As you continue to tailor your messaging, your cycle starts all over again with creative testing. Once you hit that persona stage, you keep testing which subject line best increases conversion and which messaging and positioning works best for this audience.
The cycle just continues and continues, which I think is one of the really exciting things about being a marketer – there's so much you can test and refine within a subset of your customer base.
Top takeaways
As a quick reminder, let’s look at the top learnings I hope you take away from this article:
- How to create a simple A/B test
- How to create segment-specific campaigns, thinking through tests for emails, messaging and positioning, and different channels
- How to refine your target audience by looking into your data and creating hypotheses
- How to use your data and hypotheses to inform your upsell targeting strategy
- How to use personalized content within segment-related tests to help increase attachment from your customer base.
One of the biggest things I learned from setting up a new funnel is not to over-engineer the marketing. A/B tests are a simple but effective tool to help refine your messaging and positioning, and they’re a great way to define your target audience. This is a framework I wish I’d had when I first started upsell testing, so I hope you found it helpful.
