Github

Spying On Your Competitor's A/B Tests

Seeing what your competitors are testing on their website can be very useful. It can give you an indication of what parts of their business they are focusing on (for example, pricing), or serve as inspiration for what you might want to try on your own site.

Luckily for us, if your competitor is using Optimizely, you can see every single test they are running. The Optimizely client script contains a JSON object that has every single experiment, variation, and audience targeting data, right in your console:

While you could check out a site's tests in the dev tools console, it would be much better to automatically get a list of all running experiments for a given domain. So with Phantomjs and a little trial and error, I present to you...

A Command-Line A/B Test Spy

For the purpose of this article I am only going to focus on OptimizelyX, their newest client & API version. This is also possible to do with Optimizely classic, but their javascript API is slightly different.

Let’s start with some basic Phantomjs setup. We open the site that’s passed in as an argument to the script, and if that page has Optimizely installed, we query for its data object and then send it along to a callback function:

var domain = system.args[1];
page.open('http://' + domain + '/', function(status) {
  if (status == 'success') {
    page.evaluate(function(domain) {
      setTimeout(function() {
        if (window.optimizely) {
          window.callPhantom({optimizely: window.optimizely.get('data')});
        }
      }, 3000)
    }, domain);
  }
});

The window.callPhantom method will send the Optimizely data back to our phantomjs page object. We can receive that callback here:

page.onCallback = function(data) {
  if (data['optimizely'] && data.optimizely['campaigns']) {
    processXData(data['optimizely'])
  }

  phantom.exit(0);
};

This function does some final checks to make sure we’re dealing with OptimizelyX data, and then sends it along to our parsing function:

function processXData(data) {
  var expCount = 0;

  for (var campaignId in data.campaigns) {
    var campaign = data.campaigns[campaignId];
    console.log('Campaign', campaign.name);

    for (var i = 0; i < campaign.experiments.length; i++) {
      var experiment = campaign.experiments[i];
      expCount++;

      if (experiment) {
        console.log('  Experiment', experiment.name);

        for (var j = 0; j < experiment.variations.length; j++) {
          var variation = experiment.variations[j];
          console.log('    Variation', variation.name);
        }
      }
    }

    console.log('===');
  }

  console.log('Found', expCount, 'experiments');
}

Here we simply loop through the campaign, experiment, and variation data, pulling out their names. A more advanced version of this script might even pull the code for each variation to see exactly how each treatment differs. We’ll leave that for next time.

Running this script on various sites will give you a complete list of their A/B and personalization campaigns, regardless if they are running on the homepage or not.

Box.com

$ phantomjs opt.js box.com
Campaign O365 eBook
  Experiment O365 eBook
    Variation Original
    Variation New Form Only
    Variation New Form and Masthead
    Variation New Form and Sneak Peak
===
Campaign Pricing: Redesign
  Experiment Pricing: Redesign
    Variation Original
    Variation New Design
    Variation New Design & Biz Plus
===
Campaign IPP: Button vs Arrow CTA
  Experiment IPP: Button vs Arrow CTA
    Variation Original
    Variation Variation #1
===
Found 3 experiments

Seems like Box.com are testing a new eBook lead magnet, a pricing page redesign, and some different CTAs.

Optimizely.com

As an example of the kind of business intelligence you can do with a tool like this, here is a campaign that shows a different homepage to employees of a certain company, presumably some accounts that Optimizely are trying to close:

Campaign [Takeshi] Homepage Customer Validation
  Experiment Demandbase_eBay
    Variation Variation #1
  Experiment Demandbase_Macy's
    Variation Variation #1
  Experiment Demandbase_Sony
    Variation Variation #1
  Experiment Demandbase_Priceline
    Variation Variation #1
  Experiment Demandbase_Alias_is_Bentley
    Variation Variation #1

Uber.com

$ phantomjs opt.js uber.com
Campaign Chameleon V2 Experiments - Live Chat - France
  Experiment Chameleon V2 Experiments - Live Chat - France
    Variation Original
    Variation Variation #4
===
Campaign A/A Test - Promo
  Experiment A/A Test - Promo
    Variation Original
    Variation Variation #1
===
Campaign UBR-1: Homepage Divided Path on Hero - phase 1
  Experiment UBR-1: Homepage Divided Path on Hero - phase 1
    Variation Original
    Variation Divided
    Variation Divided + Headline
===
Campaign A/A Test - Homepage
  Experiment A/A Test - Homepage
    Variation Original
    Variation Variation #1
===
Campaign Promo Test - Feifei
  Experiment Promo Test - Feifei
    Variation Original
    Variation Variation #1
===
Found 5 experiments

Uber are running some A/A tests. Very important to do!

Never Run Out of Test Ideas

While you can use this to spy on the competition, I find it useful to check on sites that are in completely different industries. See what other kinds of businesses are testing, and watch their site to see what gets implemented permanently. Then add it to your testing backlog!