We recently redesigned DSNews.com. We had a tight deadline so some of the polishing elements have been added on over time. Rather than add them in a careless manner, I decided to develop my testing skill set by always introducing new elements with a 2 or 3 variation test.
The first new feature to be developed and deployed with a test was the incorporation of a “Recent Articles” list at the bottom of every article. The recent articles widget filled a large whitespace on some of the shorter articles. In addition to that, the site picked up by the Google news service, which resulted in an uptick in traffic landing on the article page rather than the homepage. We wanted to encourage this traffic view a couple of stories.
Before
This would be my second test with the shiny new to me Google Website Optimizer. I chose to set the test up as a multi-variant test. I liked the multi-variant test because I could easily use a bit of ajax to load the different versions of the “Recent Articles” list. Since the test was going to be on every article page, this was much simpler then trying to develop an A/B test, which would involve having multiple URLs for every article.
The test had 1 variable with 2 variations. The two variations I chose were to use an image with a headline or just a text headline. When we designed the site we wanted to make sure that every article had a picture. I made sure that the list would not show two articles with the same picture, other than that the headlines were identical.
Headlines with Pictures
Text Headlines
I set up the test rather quickly. I had run one other multi-variant ajax test in the past and converting the code for this test was a breeze. Since our goal was to get them to view additional articles, I placed the conversion code on a click of a recent article from the list.
My prediction was that the image version would perform better than the text only version because people like pictures. I let the test run and got some pretty quick feedback. The text only version took the lead out of the gate and never gave it up.
I let the test run for two weeks. We had 24,000 runs of the test, split evenly between the two variations. The text version converted at 9.5-11.7% during the course of the test. The image version converted at 6.8-7.8% during the test. At the end of the day the text version was a 42.9% improvement. This was a substantial win. Since the article page accounts for 55-60% of the sites page views the benefit of this test can be calculated by:
% of page views the article page represents * % of clicks on a recent article
60% * 9.5% = a 5.7% increase in overall page views (Text only)
60% * 6.8% = a 4.1% increase in overall page views (Image and Text)
Bottom line: My intuition was wrong on this test. People seem to find the large text only headlines more clickable than the images. A 5.7% increase in page views is huge for our site. The winning variation added an extra 1.6% to the top line.
The Winning Combo
If you would like to see the winning version in action you can check it out here.
Leave a Reply