• About Us
    • New York
  • Work
  • Capabilities
  • Careers
  • Technology
  • Blog
  • Contact Us
  • About Us
    • New York
  • Work
  • Capabilities
  • Careers
  • Technology
  • Blog
  • Contact Us
July 01, 2015

Fractional Factorial Experimental Design (ABCDE Testing)

Posted by George Andrade and Preston Smith

Tired of waiting around to get relevant data for your standard A/B tests? At PMG we recently created and tested a new way to get statistically actionable insights. This type of testing not only cuts the time out of ad testing, but also allows you to test at least 5 different elements at one time so you can know where to go next.

Our goal was to find a way to gain insights for our ads at a faster pace. The usual A/B testing methodology was limiting for us as we would have to wait 2-4 weeks for significant data to come through to make a logical decision. Through fractional factorial experimental design, we were able to cut testing times in half, and provide multiple learnings for various elements within our ads in paid search.

We wanted to start by defining the pieces of a paid search ad that we focused on in our test. Marketers have control over almost everything you see in a search ad. Three of the most influential factors in a search ad are the headline, line 1, and line 2. The headline, highlighted with a green box below, is the most prominent. It is the first thing a user sees when they make a search. Line 1 is highlighted in orange while line 2 is highlighted in blue. These lines have similar character limits, but the positioning of line 1 is much more prominent than line 2.

paid-search-ad

Our initial test included testing the following in our paid search ads:

  1. Use of a branded headline for non-branded searches
  2. Using two different call-to-actions
  3. Having the call-to-action in line 1 or line 2

With the old A/B testing method, we would have had to run three separate tests using 6 ads for the above example. We wanted to implement a fractional factorial experimental design to allow us to run the tests concurrently with only 4 ads. To get the four ads, we built out the factorial table shown below. All eight combinations of ads are shown. To get the fraction we wanted, we simply multiply the three signs and get an “All Factors” sign. We then simply choose all of the positives or all of the negatives from that final column. The negative ads were chosen, as highlighted below.

kpi

The KPI for this type of test was click-through rate, clicks divided by impressions.

We then created Boolean columns for each of our three tests using the “string” R package to detect what type of headline was served in each ad, what type of call-to-action was shown, and the location of the call-to-action.

Finally, we split the rows of lumped up impressions and clicks into a new data frame where each row represented one impression of the ad. The “clicks” column now contained a 0 for each impression that didn’t receive a click and a 1 for each click. This manipulation of the data allowed us to fit a linear model on the click-through rates for each ad. We fit a different model for each market in order to determine if there was any difference between the markets that we tested.

Implementing fractional factorial experimental design is already changing the way many in our agency are going about testing. This methodology has allowed us to gain insights faster than any other technology available today.

a/b testingad testingexperimentspaid search experimentsppcstatistics
Previous
Next

Latest White Papers

  • Shifting Plans for 2020 & Beyond
  • Game On: How Brands Can Log Into A Diverse Multi-Billion Dollar Industry
  • What CCPA Means For Brands
  • How Google is Improving Consumer Data Privacy
  • Ways to Prepare for the Cookieless Future
  • See all White Papers

Featured Posts

  • Ad Age Names PMG #1 Best Place to Work in 2021
  • Hindsight 2020 & Looking Ahead to 2021
  • Preparing for Streaming’s Growth & The Future of TV Buying
  • MediaPost Names PMG Independent Agency of the Year
  • PMG Client Portfolio Trends During Amazon Prime Day 2020

Categories

  • Consumer Insights
  • Content
  • Creative Design
  • Data Analytics
  • Development
  • Digital TV & Video
  • Ecommerce
  • Industry News
  • Local
  • Mobile
  • Paid Search
  • PMG Culture
  • Programmatic & Display
  • SEO
  • Social Media
  • Structured Data
Fort Worth

2845 West 7th Street
Fort Worth, TX 76107

Dallas

3102 Oak Lawn Avenue
Suite 650
Dallas, TX 75219

Austin

823 Congress Avenue
Suite 800
Austin, TX 78701

London

33 Broadwick Street
London
W1F 0DQ

New York

120 East 23rd Street
New York, NY 10010

Get in touch

(817) 420 9970
info@pmg.com

Subscribe to the PMG Newsletter
© 2021 PMG Worldwide, LLC, All Rights Reserved
  • Contact
  • Privacy Policy
 Tweet
 Share
 Tweet
 Share
 Tweet
 Share
 LinkedIn
We and our partners use cookies to personalize content, analyze traffic, and deliver ads. By using our website, you agree to the use of cookies as described in our Cookie Policy.