Using javascript to A/B test landing pages, Math.random() not random?

Question

I have a javascript ad that runs on multiple websites. The ad is for an app, and when clicking the ad the user is sent to a landing page where js on that landing page determines whether to show A) landing page A, B) landing page B) or C) redirect the user to app store/play store.

The ad is choosing landing pages dynamically, by setting the parameter utm_content={landingA, landingB, nolanding}, so there are basically three choices:

Directly to store (50%)
Landing page (50%)
- Landing A (50%)
- Landing B (50%)

The way this is chosen is by pure js and the code is like this:

if(Math.random() < 0.5) nolanding;
else {
 if (Math.random() < 0.5) landingA;
 else landingB; 
}

When the user then clicks the ad, he or she is sent to the landing page, and based on the utm_content parameter's value, the landing page is showing A / B or redirect to store without showing anything.

In Google Analytics I made a pie chart to just test if the traffic is distributed 50%, 25%, 25%, but that does not seem to be the case.

As you ca see from the attached image, it does not have the expected distribution. So, my question is:

Is the problem that the Math.random() is not as random as I expect? I've looked into some other questions discussing that issue, but it cannot be that inaccurate?

Or is it Google Analytics not being able to track the utm_content parameter correctly? Could it be an issue with the redirect to stores going too fast, so GA can't track the page view?

EDIT: Here is the pie chart setup:

It will only show data that also has the utm_campaign= dynamiskNA or dynamiskTF which is exactly what I want to see.

It's more likely that the pie chart doesn't count the things you want it to count (Does it include the initial landing page? Traffic from other sources?) — JJJ
– JJJ, Commented Aug 10, 2015 at 8:05
@hindmost Yep, I actually believe that it is GA that causes the problem. Juhana: See the pie chart setup in my edit. — stianlp
– stianlp, Commented Aug 10, 2015 at 8:16
@Paul Yep, I've read it now. I get that there is a probability that the numbers won't be exactly 50/25/25, but in my case the pie chart is more like 25/34/41, and the weird thing is that it 'nolanding' which is expected to have 50% of the traffic has less than 25%... — stianlp
– stianlp, Commented Aug 10, 2015 at 9:12

stianlp · Accepted Answer · 2015-08-11 09:35:19Z

0

Ok, so the problem was with the ga('send' 'pageview') event. I did not wait for the hitCallback from GA before redirecting, and therefore GA did not track all events.

It's a pretty straightforward solution, and a stupid mistake.

BEFORE:

ga('send' 'pageview');
// Went ahead and redirected

AFTER:

ga('send', 'pageview', {'hitCallback' : function() {
    //Go ahead an do stuff :)
}});

@Paul Thanks for showing me the chi square test. Learned something useful :)

answered Aug 11, 2015 at 9:35

stianlp

1,0191 gold badge10 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using javascript to A/B test landing pages, Math.random() not random?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related