Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

Appendix A:
Broadening Your Optimization Program

Google Analytics Breakthrough has covered the strategy, implementation, reporting, and analysis phases of the optimization pyramid that we saw all the way back in Figure 3.1. Throughout our discussions, we have suggested specific optimization steps that you can take based on the data that Google Analytics provides, but there are aspects of optimization that we have not yet discussed directly.

As essential as quantitative Web and mobile analytics data is for achieving insight and action, it’s important to remember that your overall optimization program should also comprise qualitative inputs, overlay visualizations, and testing strategies. As the next level of optimization, personalization goes beyond analytics to present different experiences to specific user segments.

In this appendix, experts share key insights and strategies for these components of your optimization program.

Qualitative Inputs

In our optimization context, quantitative refers to numbers-based Web and mobile app analytics data that users generate but do not provide directly and qualitative refers to any inputs or evaluations that end users or testers do provide directly.

Qualitative inputs can sometimes identify conversion issues that analytics could not. As an example, what if your users are expecting to see a money-back guarantee on your payment page? If you have set up a goal funnel or Enhanced Ecommerce in Google Analytics for your checkout process, you’ll be able to identify a high drop-off on the payment page, but a single comment in a customer evaluation or a usability test could readily reveal the specific problem.

In this section, Duff Anderson of iPerceptions discusses the power of surveys (as well as iPerceptions integration with GA), and Hannah Alvarez of UserTesting provides key insights on usability testing. Note also that Google Consumer Surveys is available as a tag template in GTM for deploying basic survey functionality.

As you consider qualitative inputs, remember to actively seek out opportunities to speak with your organization’s prospects and current customers, as well as the people in your organization who interface most frequently with your customer base, such as the sales team, call center representatives, and customer/technical support specialists. Never underestimate the importance of these conversations.

GUEST SPOT Enhancing Google Analytics with Visitor Surveys

Duff Anderson

Duff Anderson is senior vice president and co-founder at iPerceptions.

You can’t get into your Visitors Heads with Google Analytics Alone

The power of clickstream data in understanding what visitors experience is colossal. But for all the insight that can be gleaned from Google Analytics such as “What are your visitors doing?”; “When are they are on your site?”; and “Where are they coming from?” the WHY remains unanswered. Avinash Kaushik, digital marketing evangelist for Google and bestselling Web analytics author, stated that, “It [Web analytics] cannot, no matter how much you torture the data, tell you WHY something happened.”¹

When you’re wondering why a landing page has only a 0.5% conversion rate or if 3.34 pages/session indicates a good or bad visitor experience, you need to gather and interpret qualitative inputs in addition to your quantitative data. It is only by getting inside your visitors’ heads with qualitative inputs that you can start to complete the visitor experience picture.

How do You Get Into the Heads of Your Visitors?

The only way to understand your visitor behavior is by asking them and collecting qualitative feedback data or what is commonly referred to as Voice of the Customer (VoC) data. There is a variety of ways to engage visitors and collect qualitative data ranging from a passive persistent feedback button to random active solicitation. Figure A.1 Voice of the Customer data.

Figure A.1 Voice of the Customer data.

Passive Persistent Engagement

Passive feedback or comment cards are a persistent button that is available for visitors to click on to leave their feedback. iPerceptions research shows that users who use passive engagement methods are more likely to leave negative feedback. Therefore, passive persistent approaches are more of a remediation tool that can help fix broken links and give visitors an outlet to vent.

Targeted Engagement

Targeted engagement collects qualitative data by showing a survey to a specific audience based on target criteria such as number of pages viewed or time on site. It is typically used to optimize sections of a website or a particular site feature. As an example, by triggering a survey to collect feedback from visitors who visited only the support section of your site, you can understand how effectively your online support tools are meeting visitors’ needs and how you can optimize those resources.

Pre/Post Random Active Engagement

The other type of qualitative data is representative and leverages a random pre/post sampling methodology. This method collects qualitative data by inviting a portion of your website visitors early in their visit to take a survey at the end of their visit. This provides information that is representative of the population being sampled creating internal alignment and helps drive strategic decisions.

It is important to understand that not all qualitative data is created equal. The method you choose will ultimately impact the type of data you collect (representative to individual) and how it can be used (strategic to tactical).

How to Ask for Feedback?

The way you ask will also have a big impact on the quality of the results you collect. Below are some fundamentals that you should always follow:

Asking your visitors for feedback is an extension of your brand. This means it is essential to customize your research to match your brand; from survey invitation to thank you page, ensure your brand is front and center.
In today’s multiscreen world, it’s important to collect customer feedback across devices. Whether it is on a desktop, tablet, mobile, or in-app, ensure the survey or comment card is responsive to your customer’s device.
Keep your research objectives focused. One of the biggest mistakes is to try and answer everything in one survey. Running multiple studies with clear objectives—from improving site usability to increasing marketing effectiveness to increasing conversion—will help increase your responses and the quality of the data you collect.

What Should You Ask?

It’s important to collect a variety of qualitative metrics to contextualize your Google Analytics data and get a holistic picture of how well you’re meeting visitor needs. As mentioned, it is essential to keep your research focused on your objectives, but there are few metrics that should form the backbone of any qualitative research.

Visitor Satisfaction

Based on today’s visit, how would you rate your overall experience on the website today?

Understanding and measuring the overall experience is critical to gauge the end-to-end experience of your visitors. With visitors providing a score of your website’s experience on a scale of 0–10 (where 0 represents a very bad experience and 10 is an outstanding experience) you can quickly evaluate your website’s performance and then see how your site improves over time.

Visitor Intent

Which of the following best describes the primary purpose of your visit?

Why are people visiting your website? What are they trying to do during their visit? Visitor intent, also known as purpose of visit, is essential to understand the needs of your visitors. Once you know why people are coming to your site, you can better align your content to what visitors are trying to achieve.

Task Completion

Were you able to accomplish everything you wanted to do on the site today?

Task completion tells you if visitors are able to successfully complete the reason for their visit. If visitors are not able to complete the purpose of their visit, this will impact their satisfaction with your site, their likelihood to return and their likelihood to recommend your site.

The Powerful Combination of Google Analytics and Qualitative Data

With the right data and taking a focused approach to collecting qualitative data you can fill the Web analytics void and better understand your customers to ultimately optimize the visitor experience.

The integration process, depending on the provider, should be a straightforward affair. Typically the data collected is sent to Google Analytics using events, so you can create a segment as shown in Figures A.2 and A.3. (Segments are discussed in detail in Chapter 11.)

Figure A.2 If your qualitative data integrates into Google Analytics as events, you can create a Google Analytics segment based on event data for visitors who responded to a qualitative survey with very low overall satisfaction (0 – Very Bad OR 1 – Fair).

Figure A.3 As an example of segmentation based on survey responses that appear in Google Analytics as events. The Pages report above has three segments applied based on job role: Senior Management, Marketing, and Research.

Once you have set up Google Analytics segments based on the qualitative event data, you can:

Better evaluate landing pages, exit pages, keywords, traffic sources, and campaigns by qualitative Visitor Satisfaction rates.
Compare Google Analytics goal conversion rates against your qualitative Task Completion rates (Did you accomplish your task?) to gain a better understanding of the conversion cycle.
Compare Visitor Satisfaction rates by time on site, pages visited, sections visited, and geographic region.
Examine time on site by Task Completion rates to distinguish between visitors struggling to find information and those positively engaged on the site.

In order to understand your customers, web analytics is not enough, you need to know what they are thinking. Make sure your optimization strategies focus on understanding the “why” behind the “what.” It’s only then that your objectives and strategies will align to your visitor needs and expectations. Using the techniques outlined above, you can begin understanding the why behind the what and aligning your design, content, and marketing strategies with visitor needs and expectations.

GUEST SPOT User Research and Qualitative Optimization

Hannah Alvarez

Hannah Alvarez leads the content marketing team at UserTesting.

As you’re digging deeper into your analytics, you may find yourself asking a lot of questions.

Why does one product page have a much higher bounce rate than the others? Why has average time on page dropped over the last two months? Why does traffic from Twitter convert at a higher rate than traffic from LinkedIn?

That’s where user research—specifically user testing—comes in.

User testing is watching over the shoulder of real people interacting with your site and speaking their thoughts aloud as they go. Listening to users think aloud is great for finding out where and why users become stuck, confused, or frustrated on your site. It’s your opportunity to see your site through the fresh eyes of a first-time visitor.

By user testing, you’ll uncover the usability issues on your site that cause high bounce rates, low conversion rates, and unusual session durations. Some of the most common usability issues include confusing navigation, unclear forms, and unhelpful search results. These issues can be very difficult to uncover through analytics alone, but analytics are a great place to start. When you’ve pinpointed specific pages or behavior flows that look problematic in your analytics, you can run user tests on those pages and flows to find out why users are giving up. In your user test, you will choose which tasks you’d like to watch users complete and which questions you want them to answer about their experience.

User testing isn’t limited to websites, either! It’s equally important to gather user feedback on your mobile apps, if your company has them. You can test mobile apps using the same process outlined below. Make sure to test both your iOS version and Android version, if you have both.

In-Person vs. Remote User Testing

User testing can be in-person or remote, and both styles have their advantages and disadvantages.

In-person user testing usually requires a usability lab and a moderator (though it can happen in an office, a conference room, or even a coffee shop). You can gain very rich and deep insights from in-house research. Researchers can observe the test participants’ body language and facial expressions as well as their on-screen actions and verbal feedback—and the moderator can help guide the participants if they become stuck at any point or need to ask a question. In-person sessions tend to run longer than remote sessions, so they can be better for activities that require a very long time to complete (such as applying for health insurance).

On the other hand, it is often expensive and time-consuming to set up for in-person user testing sessions, and it can be challenging to recruit test participants who accurately represent your target market. Plus, first-time test participants may feel nervous or overly eager to please, so it’s up to your moderator to press them for open and honest feedback.

Remote user testing tends to be faster and easier than in-person testing. You can recruit participants from anywhere in the world via an online panel of users, or you can recruit your own users and invite them to participate. Because remote user tests don’t require participants to be in a certain place at a certain time, it’s often much easier to quickly find people to take your test, and you can run larger-scale studies more easily.

In a remote user testing session, test participants record their screen and their voice as they interact with your website, and you watch the video recording when it’s complete. Depending on which platform you use and the complexity of your study, you can get results in hours, days, or weeks. Most remote user testing does not involve a moderator, so it’s essential to write a clear test plan that will guide participants through the tasks you need them to complete. (However, some remote user testing platforms do allow for moderated testing.) Since you usually do not see the facial expression and body language of the participants, it’s critical that test participants continue speaking their thoughts aloud the whole time.

How to Run a Successful User Testing Study

You don’t have to be a professional researcher to conduct a study. All you need to do is follow five steps.

1. Identify your testing objective.

What are you trying to learn? If you’re simply trying to find the most obvious usability issues on your site, your test plan will be different than it would if you were trying to find out why the bounce rate is high on a particular landing page. This is your opportunity to establish what you’re aiming to optimize.

2. Think about the audience for your test.

You will likely want to run your user test with your target market. If you have a niche audience, the recruiting process may be more time-consuming. For in-person testing, you’ll need to find those users and make arrangements for them to come in to your lab. For remote testing, you will designate your demographic requirements when you set up your study in the platform.

Make a note of which devices you’ll want to include in your test. The user experience on your mobile site is just as important as your desktop site, so don’t forget to test with iPhone, Android, and tablet users.

If you’re aiming to uncover usability problems, you don’t need very many test participants. Five to eight users will reveal most serious issues.

On the other hand, if you’re trying to find trends in users’ opinions (such as how likely they would be to trust your company), then you’ll want a larger dataset. Aim for at least 20 to 30 participants, and make sure to include users from each of your different audience segments.

3. Write the test plan.

Focus on writing tasks and questions that will uncover the insights you need. Make sure to keep your questions neutral so you don’t bias your test participants’ answers.

When you write tasks, make sure you don’t handhold participants too much or lead them to the “right” way of accomplishing the task. For example, if you say, “Find a pair of men’s dress shoes in size 10,” you’ll uncover more insights about how participants would naturally go about finding shoes—and any struggles they encounter—than if you said, “Click on the tab labeled Men’s, and then scroll down until you see the Shoes category. . . .” It’s always a good idea to run your test with one participant first to make sure there aren’t any confusing instructions that could derail your study.

4. Run the study!

If you are testing in-house, you’ll want to make sure your moderator is ready and you have your audio and video recording tools set up beforehand. Your moderator will lead the participant through the test task by task, asking clarifying questions along the way. If you are using a remote testing tool instead, you’ll order your tests online and wait for your videos to be completed.

5. Analyze the results.

As you watch the users interact with the website, take note of any places where they become stuck or confused. These insights will help you answer the questions about your data. Using either your remote testing platform or traditional video editing software, you can create clips of interesting moments from your tests to focus on the most important findings.

What to Do with Your Results

Now that you’ve gotten feedback from your users, you can organize and prioritize the issues you’ve found. You’ve likely created a list of problems your users ran into. These are your optimization opportunities! Any one of them could become your next A/B test.

Start by assigning a priority to each issue you’ve found. How many customers does this affect? How big of an impact could it make if you improved it? How many resources will you need to allocate in order to make a change?

If you find it’s always a challenge to get buy-in to make changes, try showing your stakeholders video clips from your user testing sessions. This is a great way to convince your team to make improvements.

The qualitative insights you’ll gain from user testing are the perfect way to humanize the data and complete the story of what’s happening on your website. A combination of analytics and qualitative feedback can be your secret weapon of optimization. As an added bonus, these real user stories can help you build a customer-centric culture at your company.

How StubHub Used Qualitative Optimization to Boost Revenue by Millions

The team at the online ticket marketplace StubHub noticed that a lot of customers were entering their purchase funnel but never completing it. The analytics showed that there was a significant drop-off on the ticket search results page, which prompted users to “See details” to go to the purchasing page for each available ticket. They ran user tests to find out why this drop-off was happening, and they discovered that users thought “See details” would take them to the fine print, not the purchasing page.

With this insight, the team changed the “See details” link to an orange button with the word “Go.” After making the change, they saw a 2.6% increase in conversions, leading to millions of dollars in increased revenue. (See Figure A.4.)

Figure A.4 Following user research, a change to the button text and styling on StubHub’s ticket search results page drove a significant conversion increase.

Overlay Reporting

As mentioned in Chapter 12, the Enhanced Link Attribution setting in your GA pageview tags can help the GA In-Page Analytics report to display more meaningful data in overlay format, that is, numeric data or data visualizations superimposed onto the page itself. In-Page Analytics, however, is currently limited in functionality (and sometimes does not display at all).

Several tools offer fuller-featured overlay reporting (and session recording), such as CrazyEgg, SessionCam, or HotJar, which Alex Harris uses in conjunction with the GA Landing Pages report to generate quick wins, as he demonstrates next.

GUEST SPOT Quick Wins for Big Results

Alex Harris

Alex Harris is a conversion rate optimization manager with over 15 years experience with Ecommerce, lead generation, and UX.

Numerous years as a Web designer focused on conversion optimization helped me discover that the biggest opportunities start with understanding how to find quick wins, including identifying your most important landing pages, interactions on those pages, and why or why not visitors are completing the intended actions.

Let’s say you’re optimizing your site to improve your conversion rates. Here’s how you can start finding those quick wins:

Review your top landing pages.

Locate the Google Analytics “Landing Pages” report under Behavior, then Site Content. Review the top 5–10 landing pages generating the most traffic.
Understand how visitors interact with your landing page.

Use tools like HotJar, we can create heat maps, scroll maps, form completion tracking, and visitor recordings. These data collection actions give you the ability to fine-tune the user experience. Watching your visitor’s sessions, as they go from page to page, will give you valuable insight to form a hypothesis.
Determine why (or why not) visitors are completing a conversion.

To collect this qualitative insight, we can also use Qualaroo polls and surveys. Say, if people are on your landing pages, and they move their mouse to the back button, you may want to pop open a survey that asks them, “What is missing from this page?”

Conclusion

Through the years, I have applied this same process with all my new projects to effectively help my clients increase their conversion rates, understand their visitors and the interactions on their sites, and make more money off of their websites.

Testing

Your lead generation form for the Machu Picchu tour on your travel site is showing a 70% exit rate in GA, so you change the main graphic on the form from mountains to hikers, and the exit rate goes down to 50% the next month. The hikers photo made the difference, right?

In a usability evaluation for one of your main landing pages on your health care website, the tester suggests that you include insurance information at the top of the page. In the next week, the conversion rate for your contact-us goal rises from 0.9% to 1.5%. Can we be confident that the insurance information drove the conversion increase?

In both cases, the changes likely had a positive impact, the increase could have been due to any number of external factors that changed at the same time as your page update.

As a more data-driven approach, you can instead test two or more concurrent variations against each other relative to the same performance metric such as bounce, goal conversions, or Ecommerce revenue, thereby ensuring a cleaner comparison.

Following, Chris Goward walks us through hypothesis generation for your testing, Bobby Hewitt shares best practices for test documentation and analysis of test results, and Tara Dunn offers tips for successful testing as well as framework for mobile app testing through Optimizely.

GUEST SPOT Creating Powerful Experiment Hypotheses Using the LIFT Model

Chris Goward Chris Goward is the founder of optimization agency WiderFunnel and author of You Should Test That! An A/B testing program can produce some of the greatest returns on investment of any marketing activity. Many companies are lifting their revenue from the insights their optimization experiments deliver. But other companies aren’t so fortunate.

What makes the difference between successful and failed programs? Our experience building these programs at WiderFunnel since 2007 shows remarkable consistency. One of the characteristics that separates the best is the company’s commitment to using frameworks to answer business questions.

Finding a list of conversion optimization tips is a simple Google search away, but will the tips and tricks you find work for you? More likely, they’ll be guesses that ignore your situation. They’re nearly useless.

Using frameworks to help you answer your question is more robust. It gives you an answer that doesn’t expire when one of the variables changes.

The LIFT Model² is just such a framework, which you can use to create powerful experiment hypotheses. First, we’ll look at how the LIFT Model works, then show how to use it within a proper hypothesis structure.

Using the LIFT Model

When I developed the LIFT Model as a framework for creating A/B testing hypotheses, I didn’t realize it would become as popular as it has. It’s now used in hundreds of companies and universities globally. And you too can use the model to improve your marketing.

The LIFT Model shows the six factors that affect your conversion rates right now. By testing hypotheses to improve each of these factors, you can improve the results of all of your marketing touchpoints.

The WiderFunnel LIFT Model

Value Proposition

The vehicle that provides the potential for the conversion rate is your Value Proposition, making it the most important of the six conversion factors. You can think of it as an equation that goes on in the mind of your visitors when they’re subconsciously considering whether to take action on your page (i.e., to convert).

If the perceived benefits outweigh the perceived costs of taking action, they’ll have motivation. If costs outweigh the benefits, they’ll bounce immediately.

All the other factors simply enhance or detract from the value proposition.

Relevance

Does the landing page relate to what the visitor thought they were going to see?

The Relevance of the value proposition and context of the source media is critical. Your page must use terms your visitor relates to and be consistent with the incoming link or your visitor will be disoriented and leave the page.

Clarity

Does the landing page clearly articulate the value proposition and call to action?

Clarity is the most common of the six that we find marketers struggling with. The two aspects of Clarity that must be analyzed are Design and Content. Designing for Clarity creates an unimpeded “eye-flow.” Content clarity ensures the images and text combine to minimize comprehension time.

Urgency

Is there a motivation for the action to be taken now?

Urgency has two components: Internal (or how the visitor is feeling upon arrival) and External (or influences the marketer can introduce to the visitor). While Internal Urgency is generally preexisting when the visitor arrives on the page, the tone of the presentation, offers, and deadlines can all influence External Urgency.

Anxiety

What are potential misgivings the visitor could have about undertaking the conversion action?

Anxiety is any uncertainty in your prospect’s mind about taking action. Are there things on the page or missing from the page that could create that feeling? Anxiety is inversely proportional to the Credibility you have built with the visitor and the Trust you are asking them to have.

Distraction

Are there items on the page that could divert the visitor away from the goal?

The more visual inputs and action options your visitors have to process, the less likely they are to make a conversion decision. Minimizing distractions like unnecessary product options, links, and extraneous information will increase the conversion rate.

Within each of these six factors are the many tips and subfactors that are used by the skilled conversion rate expert to develop hypotheses to be tested.

By evaluating your website, landing pages and conversion funnel from the perspective of your prospective customer, you can identify conversion problems that can be tested and improved.

While there are other fine conversion frameworks available as well, LIFT has gained popularity, I believe, because of its ease-of-use and apparent simplicity that belie its powerful results. The framework itself is a demonstration of how maximizing Clarity and minimizing Distraction can gain acceptance!

A LIFT Analysis Example

The optimization experts at WiderFunnel use the LIFT Model every day to analyze user journeys and create experiment hypotheses. For example, when International Rescue Committee needed to increase donations from their main paid search landing page, they brought in WiderFunnel to run a series of experiments. Our optimization strategists identified dozens of copy and design elements that could increase relevance, clarity and urgency while decreasing anxiety and distraction—and reinforcing the IRC’s compelling work.

Landing Page LIFT Analysis: this page contained several conversion barriers.

Among other issues, the two-column layout of the form created anxiety, appearing more difficult than it actually was. The form also lay below the page’s “fold” and the Call-to-Action (CTA) button didn’t include a hover state. On the copy side, there was a lack of clarity around the programs the donation would support, and no positive messaging supporting the decision to donate.

Once the roadblocks to conversion were identified, we were able to create an optimization plan designed to generate the greatest amount of lift and learning for IRC. Early tests showed, surprisingly, that much longer persuasive copywriting and content increased donations.

WiderFunnel’s series of experiments using their LIFT Analysis resulted in a 30% conversion rate lift. And the best part? Not only did the number of donors jump, but they were also more generous: Revenue shot up by 94.6%–extending the work the IRC could do around the world!

Landing Page Winner: testing the LIFT-identified hypotheses generated a 30% conversion improvement and a 94.6% donation increase.

Creating a Hypothesis

Once you’ve identified problems on your touchpoints using the LIFT Model, you can create hypotheses out of them. This is a process of changing your weaknesses into strengths that you can test.

Creating a strong hypothesis is an important part of any scientific experiment. The strength of your hypotheses determines the outcome of your testing.

A hypothesis follows the following structure:

Changing [the thing you want to change] into [what you would change it into] will lift the conversion rate for [your conversion goal].

For example, you may believe your headline doesn’t fulfill the internal urgency your shoppers are feeling. Your hypothesis may be:

A strong hypothesis will clarify the purpose of your experiment.

You can create hypotheses about any of the six conversion factors, but some of the most insightful tests you can run will aim at the core of the LIFT Model—your value proposition.

Testing Your Value Proposition

You need to discover your best value proposition to have the best chance to close the most sales. Emphasizing the most important parts of your value proposition will maximize your conversion rate.

When you run experiments to discover your best value proposition, you gain more than just a conversion rate lift on a single page. You gain a valuable insight that can be leveraged across your other marketing activities.

So, how do you know which features are most important to your customers?

Traditionally, marketers have relied solely on their own thinking to come up with the right features to create and emphasize. They might do some customer research, focus groups, or a survey.

Today, there’s a better way.

By testing different value proposition approaches, you can find out what works best with statistical certainty before committing to one assumed positioning.

I like to use a framework adapted from Joel E. Urbany and James H. Davis (https: //hbr.org/2007/11/strategic-insight-in-three-circles). The following Venn diagram encourages you to think about your Points of Parity (POPs), Points of Difference (PODs) and Points of Irrelevance (POIs):

Identifying Points of Difference in your value proposition can give you powerful hypotheses to test.

Points of Parity (POPs)

These are the features you offer that are important to your prospects that you also share with your competitors. Think of them as the basic entry requirements to the game. Your prospects need to know that you offer the POPs, but emphasizing them won’t impress anyone.
Points of Difference (PODs)

Here’s where you can win the game. These are the features that are important to your prospects and not available from your competitors.
Points of Irrelevance (POIs)

All the other features that you offer, but aren’t interesting to your prospects are Points of Irrelevance (POIs).

Your PODs are where you want to focus. These are the features you can emphasize that will move your prospects to action. They’re your differentiators.

Once you’ve discovered what you believe to be your best PODs, your work is not quite done. You have some good ideas. Now, you should test them.

Your landing pages are some of the best areas to test value proposition approaches. Create separate variations of your page, each emphasizing a different POD and run controlled A/B tests to find your winners. Then, use the PODs you discover to destroy your competition.

GUEST SPOT Get More Out Of Your Split Testing with Documentation and Post-Test Analysis

Bobby Hewitt

Bobby Hewitt is the president and founder of Creative Thirst, the conversion rate optimization agency that focuses exclusively on increasing revenue and average order value for companies selling direct-to-consumer health products and natural health supplements.

If we don’t understand where we’ve been, it’s very hard to figure out where we’re going. Where do we start? With systematic documentation.

Keeping track of your split test data is no easy task, which is why documentation is often overlooked. Yet, it’s vital to a successful optimization program. Done correctly, it acts as a resource of past performance and provides a knowledge base of insights across time. Plus you can develop many new hypotheses for future tests from past inspiration.

As a fictitious example, suppose we wanted to A/B test which treatment of testimonials had a bigger impact on sales. For this test, we’re going to take our existing customer testimonials that are on the control page and restyle them to have more of a Facebook type of look.

Control and Treatment 1 test examples showing the differences in the visual treatment of the testimonials being tested.

Naming Convention for Your Tests

First we’ll give our test a descriptive name that includes what we’re testing, the product or page category, as well as a test ID number, as shown in the following figure.

Test Type:

A/B, Multivariate, etc.

Goal:

State the goal in a sentence. For example: Increase the sales of skin cream xyz.

Primary Metric:

Metric you will use to measure success: Conversion Rate, Average Order Value, etc. If we have two metrics, we’ll also document the second metric.

Background:

Any specific information that provides context. Is this a follow-up test? Is this test being launched for a particular reason? Is this test running during a seasonal slump or high period? Include any information that we might want to know when we’re looking back on this test a year from now.

For this test it might be why we’re testing this particular treatment. For example: Our audience is somewhat active on social media, but we’re not sure how influential it is to purchase intent.

Primary Question:

States what you’re trying to accomplish with this test in the form of a question starting with the word “which” (e.g., Which testimonial treatment will get more visitors to buy our skin cream product?).

Hypothesis:

The hypothesis section simply presents what you think will happen: If social media plays a factor in social proof, then more visitors will buy our skin cream as measured by conversion rate.

Treatments:

The specific combination of test variables and values.

Variables:

Describe the variables we’re testing. A variable is a general element that you intend to test (e.g., Headline, Image, Price, etc.).

The variable in this test is the testimonials section.

Values:

Describe the values we’re testing. A value is the specific variation you are testing. If your variable is the headline then your values would be the different versions of that headline.

The values of this test are what the different testimonials sections look like: The control is all text. Variation 1 is presented to mimic the look of a Facebook conversation.

URLs:

List the URLs of the test pages if this is an A/B Test (e.g., Control URL, Variation 1 URL, etc.), including the primary goal URL and any secondary goal URLs if you have them.

Traffic Source:

Because different traffic types convert differently, we want to include the primary traffic type that will be driven to the test. For example, cold traffic, email promo traffic, affiliate traffic, etc.

Note: Targeting Specific Traffic Sources Targeting specific traffic sources is available with some split testing vendors. However, you can also segment a traffic source by driving specific traffic to a landing page that you’ve targeted for a test.

Test Period

Log the date the test started as well as the end date once the test is concluded.

Final Results

Record the number of visits and conversions each treatment received as well as the final conversion rates of each. Also, record the statistical confidence level reached and the percentage of improvement.

Table A.1 Final comparison of key metrics for control and treatment.

	Control	Treatment 1
Conversions	235	294
Visits	5,276	5,283
Conversion Rate	4.45%	5.56%

Statistical Confidence: 95%

Total Results: 25% Increase in Conversion Rate

Treatment 1 is the winner!

Insights

The Insights section is reserved for after you’ve finished your post-test analysis, which we’ll unpack in just a minute. You’ll use this area to write out what you learned from the test, and about your visitors.

Appendix

Finally, the appendix section is a collection of screenshots of each test treatment, so that you can refer to what the pages actually looked like during the test period.

Once your split test has conclusive results, you’ll know if you had a lift or not, but your work is not over, yet. Maximizing the impact of conversion optimization to the bottom line requires more than knowing which treatment won.

Using the Decision Insight Post-Test Analysis Framework you’ll be able to maximize your optimization program by answering the question: “What do the results of this test tell me about my visitors?”

The decision insight post-test analysis framework (registered trademark of Creative Thirst, LLC).

When you understand the why behind what happened, you can get to the root of customer insight, and compound your optimization efforts across your entire organization.

Test Outcome

The first step in the framework is to examine the outcome of your test. Did one of the treatments win or lose?

Hypothesis

Go back to your initial hypothesis and look for evidence that supports that it was proven.

In the case where your hypothesis has been disproven, is there evidence that suggests a different hypothesis?

To find such evidence, the framework takes a multi-step approach.

Evidence

In the first step of analysis we look at click map and scroll reach data with Crazy Egg to see behavior at a broad level.

The click map reveals that visitors are clicking on the Facebook style testimonial treatments, perhaps they are expecting it to link out to Facebook or they expect to comment and or like certain testimonials.

The average scroll reach of the page supports that visitors are scrolling to the area where our testimonial treatments have been presented.

Behavior

In the second step, we go deeper, by viewing individual screen recordings gathered during the test, using a tool like HotJar or Inspectlet. Observing the behaviors of visitors, and noting the mouse movements, pauses, clicks, and scrolls.

Pro Tip: Segment Recordings Segment recordings into two groups: those that converted and those that did not. You can even further segment based on desktop versus mobile.

Insight

From watching visitor screen replays, we observe that visitors who purchased our skin cream interact more with the Facebook-style testimonials. In most cases the behavior on the top section of the page that has the product offer is virtually identical to both treatments. However, we do see mouse movement pausing on scrolling through both the Facebook-style testimonials.

In each step of analysis we write down the observed behavior for each test treatment in a series of bullet points.

For example our refined list might look like this:

Pauses on scrolling to testimonial section.
Scrolls slower over Facebook treatments.
Moves mouse horizontally over Facebook style testimonials as if reading with their mouse.
Scrolls back up to purchase after reviewing testimonials.

Once we have a refined list, we ask the question, what does this behavior suggest about the action the visitor took? And create yet another list, adding what each behavior suggests.

For example:

Behavior:

This Suggests:

Pauses on scrolling to testimonial section.

Scrolls slower over Facebook treatment.

Moves mouse horizontally over Facebook-style testimonials as if reading with their mouse.

Scrolls back up to purchase after reviewing testimonials.

Testimonials are acting as a speed bump slowing them down to read.

They are consuming the content.

More evidence point to them reading the testimonials.

The testimonial treatment is influencing purchase.

By analyzing each behavior this way, we can see inside the prospects mental decision process.

Connecting the dots uncovers the why of what happened and we can glean insight by mapping our test results to the specific behavior visitors performed during our test. In other words, visitors from the winning converting group behaved this way. And visitors from the losing group behaved this other way. This helps us answer our ultimate insight question: What does this tell us about the visitor?

The insights for this test might be something like: If visitors are more engaged with the Facebook style testimonials than the control, then perhaps this means that they are more influenced by social proof. We would then add this to the insights section of our test document.

What this may tell us about our visitors is that Facebook is influential on purchase decisions.

If this is the case, then some key opportunities that we should pursue as a business might be:

Increase our Facebook sponsored post marketing.
Create a Facebook specific-campaign funnel.

You never know what gold nuggets you’re going to uncover from conducting posttest analysis. But in any case, a deeper insight into the visitors behavior can be used to develop new hypotheses to test, create new product lines, change how you manage customer service, etc. The possibilities are endless and true optimization across your entire organization then becomes possible.

Downloadable Resource

Download the Creative Thirst Test Document template to use on your next split test at www.e-nor.com/gabook.

GUEST SPOT Tips for Success with A/B Testing

Tara Dunn

Tara Dunn is a Digital Analytics Consultant at E-Nor.

In the ever enriching world of Web optimization, we now have the luxury of choosing between several visual editor/WYSIWYG testing tools. These tools make it easier than ever to plan, implement, and monitor a test. The challenge that marketers are now discovering is that with such a user-friendly interface for testing, it can be easy to get overexcited and quickly create tests that are not optimized for success.

In addition to the strategies for hypothesis generation, test documentation, and test analysis offered by Chris Goward and Bobby Hewitt, these are my tips for you to use, to avoid creating ineffective tests and take advantage of some of the deeper features of most visual editor-based testing tools.

Involve at Least One Designer, One Developer, and One Stakeholder in Your Testing

Often the problem with visual editor based testing tools is hand-in-hand with their key selling point–they enable anyone to get involved in testing. This can mean that marketers are responsible for optimizing their own landing pages, or that data analysts are suddenly directly involved in web development.

It’s important to remember that when you’re testing, you are making changes to the website. Although testing can (and should!) iterate at a separate schedule than normal Web releases, it still needs to be treated as part of the web development process. Involving a designer helps to ensure the test creative remains on point within your brand, and involving the developer helps to ensure that the test can and will work within the confines of your CMS. The stakeholder is there to make sure that the test is in line with overall business objectives.

When Creating and Editing Your Test, Make Sure to Select Exactly the Element You Want to Change

When you have many elements sharing the same space it can be tricky to select exactly the element you want to edit with your cursor. A fairly common scenario may be that you want to select a link, and you have a div container wrapped around a link wrapped around an image. Your cursor may select the image itself, rather than the link.

Most visual editor based tools offer a way to review and edit the code that is generated by the visual editor. This often requires a basic knowledge of jQuery and/or CSS.

Check Compatibility across All Browsers and Devices

While most tools attempt to prevent cross browser compatibility issues, you should always preview a test in a variety of browsers and devices. A call to action image that you want to test could look great on desktop, but move around the page or disappear completely with a responsive mobile design.

If you do not like the way your test variation appears in mobile, you can either adjust your variation to be responsive with some clever CSS, or you can often opt to exclude the test from mobile devices using your audience targeting settings.

Additionally, most tools offer a way to generate some form of preview link to help you to discuss the A/B test and QA internally before pushing live. This is a great way to involve your teams in testing and get stakeholder sign-off as well.

Always Test at Least Two Variations

I am often tempted to ignore my own rule with this one, but I promise it’s a valuable one! Testing one variation will only answer “Does this change have an impact?” So you will only get a yes or no answer. Testing two variations enables you to answer a wider range of questions and develop new hypotheses. “How much of an impact does this change have compared to alternatives? Why does one variation outperform the other? Which of these options is the best way to improve conversion rate?”

Thankfully most tools help you make additional variations quickly, by allowing you to copy existing variations. This way, you are not starting from scratch each time. Once you’ve copied a variation, you can simply adjust the copy variation as desired.

Communicate, Communicate, Communicate

Consider who may be impacted by this test. I’ve had it happen several times where an exciting test was completely derailed because of some external factor that could have been avoided with better communication. Say your design team has launched a new test, but the marketing team was not informed. Turns out, an email with a promotional offer was being sent out at that time, but the landing page doesn’t reflect that promotion in the test variations. With all of this happening live, either the marketing promotion or the test must come to a screeching halt.

Document and share with your teams:

The objective of the test. Why are we doing this?
Who this test impacts. What audiences will see the test and why?
The run dates of the test. When do we anticipate results? When will we start and stop testing?
What the test looks like so they will know what to expect. Include screenshots or send preview links.

You can utilize some features of your analytics tools and often the test reporting tools to add a little more control to this, as well. You can set up a scheduled email with a custom report and a message to remind internal teams of the test and share progress effortlessly.

When targeting test audiences in your testing tool, you usually have the option to include or exclude certain marketing efforts. For example, you can exclude users who come from email campaigns by excluding the URL query parameters that indicate they are from email. You can also usually block the test from certain IP addresses, such as your internal teams. Consider that it may be beneficial for your internal teams to have visibility to the test, however.

If you follow these tips, your tests may still fail, but you will not fail to get actionable insights from them.

GUEST SPOT Testing Mobile Apps with Optimizely

Tara Dunn

Creating a mobile app is a massive undertaking in and of itself. For each new version release, your team must plan, document, design, code, QA, and finally submit to the Apple or Android store for review. If someone wants to introduce a change at any point in the future, the whole process starts over again. The idea of introducing yet another element, testing, to mobile app development can seem like a step in the way of progress.

So Why Do A/B Testing on a Mobile App?

One of the biggest reasons actually has to do with the way mobile apps have to be released. Unlike a website, where changes can be pushed live within minutes or even seconds, a mobile app is actually downloaded by the user onto their device. Think of it as being similar to installing Microsoft Word onto a Mac or a PC. So rolling out new features or bug fixes can only be done by submitting a new version of your app to the app store for review, and hopefully publication.

Testing allows you to try out variations to your app without going through the entire mobile app development process each time. If a test fails, you can roll it back without your developers ever having to build it into a finished release. You can also conduct a phased rollout, with only a small percentage of users seeing the variation when it is first pushed live in your experiment. Most importantly, testing enables you to make data driven decisions. Knowing not only whether a new feature had an impact or not, but also the magnitude of the impact, will help you to make key design and development changes to your app as time goes on.

Optimizely offers A/B testing for both iOS and Android devices. Note that Optimizely does not currently offer support for Unity—we would recommend checking out Splitforce or Leanplum for Unity app testing. Optimizely is well known for their huge testing community, great documentation, and stellar customer support.

I’ll use Optimizely for the specific examples below, but note that many of these tips and philosophies work for other mobile app testing tools as well.

Plan Ahead

Although you’ll find that tools like Optimizely will likely save some time in the long run, you will need to invest some time towards your initial planning and implementation. Consider these questions:

What are the key aspects of my app’s functionality? Examples: add to cart, create an account, subscribe to users, post, share, list products, updates in push notifications.
What are the distinguishing features of my app that separate me from my competitors? Examples: Frequent promotions, simple navigation, easy return policy.
Who are the different types of users I want to target my tests to? Examples: new adopters, frequent users, high revenue, business specific categories.
What are the different stages in the lifecycle of my app?
Will I conduct staged rollouts of my tests? If so, what is my rollout plan?

The answers to these questions will help you to plan out the inclusion of your testing tool in your next app release. For example, you might identify that your percent discount is something you will regularly want to experiment with.

Testing Ideas

A great place to start is the user’s first impression. By improving your “forgot my password” prompts, you may be able to increase your successful login rate. You may also be able to drive influence into the app store by experimenting with the notification in your app asking users to submit a review.

For existing users, the goal is to keep them coming back to your app regularly. Testing the frequency and nature of push notifications is a great way to impact your retention rate. You can also test different motivators to create an account, such as access to new content, so that you can reengage from other platforms as well.

To test your conversion process, it is recommended to evaluate any form fill processes to ensure they are easy to understand and short. You can experiment with your discounts and promotions as well, to find a good motivation/profit margin balance.

Create Your Experiments

Optimizely offers three features to help you implement tests on mobile apps. The first one is the visual editor, which is easy to use and intuitive to understand. You can click on elements and select options to modify their position, color, size, and much more.

The second feature is Optimizely’s Live Variables. With this feature, you would identify in your app all the elements related to the percent discount. You could then use Optimizely as many times as desired in the future to experiment with different discount levels.

The last feature is Optimizely’s Code Blocks. This feature enables you to identify which code to run in your experiment variation. This feature works very well for a variety of uses, such as testing entire new forms or releasing an experiment to your users slowly by using a staged approach.

Note that you can use any of these features in combination. For example, you could use Code Blocks to serve different variations of a form, but then use the visual editor to tweak the form styles.

QA Your Work

There is no escaping QA! Thankfully Optimizely helps to simplify QA by offering a preview mode, which you can share with other developers if needed. Optimizely also offers some developer debugging tools, such as Optimizely’s data object.

A nice feature of Optimizely is that the Optimizely datafile is downloaded by the SDK every two minutes by default. So if the user accesses the app and is put into an experiment but then travels somewhere where their phone is out of network, they will continue to see the variation because the data for that variation is now stored on their device.

Launch & Analyze

This is usually my favorite part of any experiment. It’s addictive to monitor the data as it pours in, and figure out how your test idea actually performed! If it doesn’t happen to perform so well, you can very quickly shift the experiment in Optimizely to show the original to 100% of users.

Optimizely offers a clean and friendly reporting interface, with data backed by their impressive stats engine. Their stats engine is meticulously fine-tuned to provide users performance data that is as accurate and reliable as possible, so that you know if your variation really won or lost in the experiment.

Although I have seen the stats engine evolve to be more and more reliable over the years, I will continue to consistently encourage a culture of questioning data and doing deeper analysis to better understand the depth and breadth of the experiment’s impact. Integration with other analytics tools can really help with this deeper dive. Optimizely’s mobile app testing solution integrates with Universal Analytics and Mixpanel, as well as Amplitude and Localytics, which are mobile specific analytics platforms. Integration with Universal Analytics offers a wide array of additional dimensions for analysis, such as landing screen and custom events.

Optimizely will likely continue to expand their integration offering in the future, to be more on par with the much more extensive list of integrations offered to Optimizely website experiments.

In Summary

One final benefit of testing that wasn’t covered earlier is that it lays a foundation to help you personalize your app. An example of this would be if you found that targeting notifications to new product shoppers boosted your conversion rate, you could use that information or possibly even the testing tool itself to implement that feature for that specific audience in your mobile app.

If you have a mobile app and you aren’t testing it, it’s time to get started!

Marketing Automation and Personalization

One of the primary themes of the book has been segmentation: analyzing your data based on audience subsets rather than aggregates to amplify trends and more quickly gain insight. We also learned how we can use segments, aka audiences, to remarket based on specific visitor characteristics and behaviors.

If we analyze and market to specific audience segments, it’s a logical next step to actually offer a customized experience to different types of visitors. Below, Mike Telem introduces an RTP (real-time personalization) platform that allows us to present different images and messaging based on previous behaviors and (anonymous) CRM data. He also discusses the integration of company “firmographics” data for both personalization and analysis.

GUEST SPOT Marketing Automation and Google Analytics: Integration and Personalization

Mike Telem

Mike Telem is vice president of product marketing, real-time personalization at Marketo, the leader in digital marketing software and solutions.

As two of the main pillars in your marketing toolbox, Marketing Automation (MA) and Web analytics can work together to provide valuable and actionable information about the interactions of your prospects and customers in ways that neither platform can do as well on its own. An MA platform, such as Marketo, benefits from the rich interaction reporting and flexible segmentation that Web analytics provides, while a Web analytics platform such as Google Analytics benefits from the broader offsite or offline data that MA can provide, including anonymous, non-PII visitor firmographics, email campaigns, social media campaigns and interactions, trade show and convention attendance, and, to some extent, face-to-face interactions via CRM data.

However, the integration between these two tech assets isn’t fully utilized until you have all of the relevant data flowing between them. With MA data available in GA, you can segment your GA reports based on this richer set of data that includes offline interactions and firmographics, which in turn provides tightly focused optimization opportunities. If, for example, you create a segment for the financial services industry and note a high drop-off at a certain point in the Behavior Flow or Goal Flow reports, you can take advantage of the personalization capabilities in the MA platform to display customized content to that segment at the drop-off point instead of the more generic default content, as displayed in Figure A.7.

Real-time Web personalization—the capability to dynamically change the user experience based on his or her attributes—furthers the potential of this integration by not only reporting on interactions but also optimizing them on the fly—not just reporting based on segments, but presenting different messaging and imagery based on these segments.

Modern marketers are leveraging the capabilities and data that exists in MA, especially if it includes real-time personalization capabilities, and GA to achieve the following objectives:

B2B Data enrichment and advanced analytics.
Personalized Web experiences and performance data.
Personalized remarketing ads.

For those MA platforms that provide out-of-the box integration with GA (yes, Marketo does, I must admit) it is literally just flipping on a switch: the same JavaScript tag that sends data to GA is then used to enrich Google Analytics through the Google Analytics API with relevant info that your MAP has on both anonymous and known visitors.

B2B Data Enrichment and Advanced Analytics

B2B marketers care a lot about which companies people work at, how big the companies are, and what industry they belong to. Marketers would rather focus on specific groups of companies in targeted account lists where they can be more successful by driving more revenue or closing deals faster (an approach typically referred to as ABM [Account-Based Marketing]). Marketers also care very much about people’s job title, role, and their stage in the sales cycle or lead score. All of this information exists in select marketing automation platforms but not in GA. Enriching GA with anonymous firmographics data such as the visitor’s company, industry, size, and revenue (yes, even if that visitor is still anonymous) to provide more precise analytics for the B2B marketer and leveraging GA’s extensive reporting capabilities together allow you to understand which ads drove not just the most clicks but which one drove the most clicks and conversions from target accounts or which referring channels were more effective in driving C-level exec’s to your site.

By adding lead and firmographics data as a secondary dimension to your Acquisition reports, you could also correlate which channels and campaigns drove more opportunities and deals within different audience segments and place more budget on those traffic channels that eventually impacted more revenue. (See Figures A.5, A.6, and A.7 below.)

Figure A.5 This custom report is showing performance by industry, one of the several types of firmographic data that Marketo can provide to Google Analytics as custom dimensions.

Figure A.6 You can apply segments based on lead status and firmographics data to your flow reports to identify where those audience segments are dropping off.

Figure A.7 This Google Analytics dashboard displays lead and firmographic data.

Personalized Web Campaign Performance Data

Personalizing the Web experience based on various attributes such as location, behavior and known lead data is becoming common practice. No need to explain why providing the right message to the right person at the right time works to your benefit, whether you’re selling cars, watches, servers or software. Imagine someone visiting your site from an insurance company: shouldn’t they see relevant case studies and videos on your home page and across their visit to your site? How about returning visitors to your site or, moreover, people who have already purchased from you before: should they have the same experience and see the same offers as first timers? Of course not.

The reasoning for real-time Web personalization is clear. However, tying the performance of these targeted page variations to your main web analytics platform is essential to see the overall impact on engagement levels (# of visits, pages viewed, time on site, etc.), goals (clickthroughs, lead conversions, purchased, etc.) and bottom-line revenue. With the personalization variations appearing in GA as events, we can create segments based on these variations and measure goal and Ecommerce outcomes for visitors who saw the variation vs. visitors who saw the default, as shown in Figure A.8. (The Behavioral Segments section in Chapter 10 discusses segmentation based on events.)

Figure A.8 An MA platform such as Marketo allow you to personalize a landing page by industry, as in this example, or other firmographics data, and report the performance of personalized variation in Google Analytics.

Personalized Remarketing Ads

Remarketing ads have been with us for a few years now and have proven to be very effective compared to regular display ads and other advertising methods. However, these ads are still not too personal as they are based mainly on pages or products viewed on the site, geo-location, and other.

By sharing valuable MA information with GA, you can define your Google remarketing ads based on firmographic and lead/contact data in addition to the other built-in GA dimensions. (Defining a remarketing audience is described in detail in “AdWords Remarketing with Google Analytics Audiences” in Chapter 14.) Imagine being able to show ads only to the audiences you care about the most such as prospects using your competitor’s software or individuals from the finance industry or shoppers that previously bought a specific product and are marked as VIPs in your database. Unleashing the power of this data with your ads means much better ROI on a budget that is typically quite large.

Benefits of Integration

As part of the ongoing evolution towards a more user-centric approach to marketing, sales, and Web analytics, the integration of Marketing Automation and Google Analytics provides a big leap forward. For more information, see Marketo’s “Definitive Guide to Web Personalization” at https://www.marketo.com/software/web-personalization/.

Notes

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for Appendix A: Broadening Your Optimization Program

Create new playlist

Sign In

Sign Up

Qualitative Inputs

Overlay Reporting

Testing

Marketing Automation and Personalization

Notes

Table of Contents for
Appendix A: Broadening Your Optimization Program