Talk Abouts

An Interview with EMG's Brandon Reiser

By Amanda Vande Brake

Part two in this Talk About continues the conversation on online performance testing and landing page optimization and goes into the specific benefits of multivariate testing as well as the unexpected nuances considered in testing. Landing page and website optimization is discussed further, too, in context with the potentially ongoing cycle of testing and optimizing.

EMG's Lead Content Developer Amanda Vande Brake sat down with EMG's SEM Specialist Brandon Reiser to learn more about what's behind the testing tactics that drive landing page optimization for EMG clients.

EMG: Let's start with what you think are the top two benefits of multivariate testing.

Reiser: I would say the first thing is that multivariate testing allows you to find the individual variable that is actually causing the lift or decrease in page or site performance. For example, let's say a marketing director starts with an A/B test and for version B, she changes the background from white to black, changes the calls to action, adds a new photograph to the layout and changes the text on the submit button. The result is a 35% increase in conversions for version B with version A as the control. But what actually caused the lift? Was it the black background or different imagery? With A/B testing, you just don't have the ability to answer those questions. A multivariate test provides the clarity to determine the most significant factor and the confidence to make decisions based on that information.

The second benefit of multivariate testing is being able to test out multiple variations of many different values. In multivariate testing, a marketing director can provide 3 different images, 4 different call to actions and 5 different background colors, and all of these variables can be tested to determine the most significant variable as well as the best performing combination of variables.

Testing is important because without it, you just don't know which version works out the best. On a basic level, testing gives you information about what your users like and don't like. There are also other benefits in which you can apply what you learned online across other channels. If an unconventional call to action works on your website, then you might want to try that out on your direct mail campaign. And, I'm finding that in a lot of the case studies that I've been going through lately, the most unexpected and quirkiest calls to action seem to be the most effective these days for certain audiences.

EMG: Is it possible that a test returns negative results, and what if the test variations end up performing worse than the control?

Reiser: This actually happened on one of our recent tests when the control beat out all of the test variations. From my standpoint, I think we definitely have winners and losers in all of our tests, and in this case here, we definitely learned what didn't work. Now we know that when we do a future test, we need to try something different and go in a different direction.

For this specific test, there was one test value that was dragging down the performance of every multivariate version we were testing, so we stopped the test before it reached a 99% confidence level. Then we removed that value and moved forward and started a new test. If you have overwhelming quantitative evidence that a variable is performing so poorly that it is killing all your test variations, you are definitely free to stop the test, remove a value and then restart another test.

EMG: Sounds like running a multivariate test requires a lot of time and attention, then, too.

Reiser: Definitely. We monitor our tests on a daily basis. I do think it's entirely possible for a company to handle testing and optimization themselves, but the level of analysis needed to make business decisions based on the information gathered from testing usually has companies turn to agencies. If you are a traditional marketer who has been focusing on print campaigns and hasn't really looked at analytics, then I would recommend doing some substantial reading or partnering with an in-house analyst to get up to speed. If you have an existing team on-site, I would say it would be a natural progression to move on to testing based upon them having that knowledge.

I think it's difficult for some people to analyze results. We talk so much about levels of confidence, but I think a lot of people by nature are reactive. If they're doing a simple A/B test, and they get one million conversions over a weekend from version A, the reactive response it to go with A. But in actuality, the traffic sample may be relatively small and may just be random fluctuations of luck involved in the performance during the beginning of the test.

Somebody may see that A is beating B by 34% and immediately thinks that they have the information they need and just pull the plug. Some of our tests may have been running for 3 whole days and have generated only 3 conversions with version A receiving 2 conversions and version B receiving 1 conversion. Just because version A is beating version B by 100% doesn't mean we have the data we need. We need the confidence level to be over 95% that this is actually true, and the more traffic to a site, the more conversions we actually get, the faster we can get to the confidence level we need. You have to let the test run and have the mindset to see it through all the way to the end.

EMG: What changes would you make to a test already in progress and why?

Reiser: If one of our smaller campaigns dips for a bit, it's not a big deal. But if we're testing for a huge ecommerce site, then even a little dip could be tens or thousands of dollars. So we would go into testing just segments of traffic, rather than just testing every single person that visits the website.

So if you have 1000 visitors, instead of having all 1000 participate, we could say that we only want 5% of traffic to participate in the test. Most large corporations will test a small segment like this, and then if they have a clear winner, then they'll do a bigger test based on the initial information they gather. For larger, revenue-generating sites, it's a make or break deal to have values that underperform by 50-60%. That's a killer for them, so they would go with a smaller segment to test the waters first.

EMG: Again, it all leads back to the end result of testing being an increase in the ROI of the site being tested. So, if a site gets relatively strong traffic, there's really no reason not to engage in testing and optimizing is there?

Reiser: I feel like you're missing out when you're not doing testing. I don't really know of any case studies where people have done testing over a long period of time and the end result is negative. You really stand to gain by testing.

From the data we collect, we're trying to come up with a hypothesis with what's going on with the site because without testing, we don't necessarily know anything for sure. We may get a couple of visitors that send us a survey that says "Hey, this is the greatest site I've seen in my life," but you know, we make some changes to it, and it turns out to be better than what it was in the beginning.

EMG: How does multivariate testing take into consideration factors outside of on-page user interface elements like color, text and layout? Are social factors considered during these tests?

Reiser: Oh sure! When Patrick Swayze announced he had brain cancer, we had a huge spike in traffic to our campaigns that include brain cancer-related terms, and this was noted in the tests we performed. We also could've tested those people specifically coming in for the keyword brain cancer or anything brain cancer related versus another segment. So you can get as granular as you want to and create any persona level that you want. You just have to have the traffic to support it.

EMG: How far can you take this cycle of testing, then optimizing, then testing again, etc.?

Reiser: From my perspective, your audience is constantly shifting and their behavior is constantly changing. So, testing should really be an ongoing process. No matter what the project or the goal is, we can always improve depending on the priorities and the goal of what the client is trying to achieve.

If you're in a space that's highly competitive, then you have this continual cycle to build an online product, market it and analyze and optimize. If you're in a space where you're the only player in town, the need for an aggressive analyze / optimize cycle is less. No matter what, though, continually analyzing and optimizing leads to innovation which is key to staying ahead in any market.