FULLBEAUTY Brands

The two of us have been using technology to sell online since the mid-1990s (one of us helped define and standardize the pixel sizes of the very first banner ads) and we have generated many billions of dollars of ecommerce revenue. Throughout our careers, we have encountered a wave of intellectually-titillating shopping technologies. Some, like search engines and social media, changed the way that customers shop and companies advertise. Others, such as virtual worlds inhabited by avatars that tried to recreate the physical experience of shopping in stores, proved to be expensive and distracting dead ends.

At FULLBEAUTY Brands, we test extensively to distinguish technologies that matter from ones that don’t. Because new website features tend to be used by the best customers, nearly all features will show a higher conversion rate or revenue per visit due to selection bias. The best way to truly test a new technology is an A/B test: show half a random sample the new technology, hide it from the other half, then compare the performance of the two groups. We wanted to share the results of two recent technology tests. One validated a vendor’s claims; the other uncovered a big waste of money.

People have a range of body shapes that can resemble an apple, pear, hourglass, or inverted triangle, and matching body shapes with apparel sizes and fits that can vary widely across brands is a hard solve for ecommerce. TrueFit is one of the latest vendors to try and address this problem. TrueFit helps match customers to the best sizes and fits by clothing brand and we have been testing their technology on our websites for the past several months.

We started with an A/B test and found no difference in the dollars generated during a visit. Obviously not a good initial result for the vendor. But in addition to the initial visit, we wanted to know the downstream impact on metrics such as return rates and lifetime value of the customer. We conducted a matched cohort analysis to account for selection bias and tracked the performance of TrueFit users over the next 100 days.

The result, despite a slight increase in return rates from the TrueFitcohort -- which we theorize results from higher expectations for improved fit -- was a 17 percent increase in dollars per customer. TrueFit failed the A/B test and actually increased return rates, but the overall downstream impact of the technology was highly positive. We don’t run across many technologies that generate a 17 percent improvement in business performance. This was a pleasant surprise.

The best way to truly test a new technology is an A/B test: show half a random sample the new technology, hide it from the other half, then compare the performance of the two groups

Another technology we tested was retargeting site visitors with display ads on other websites. This is a popular way to spend digital advertising dollars because the return on investment appears to be extremely high. We wanted to isolate the incremental impact of the advertising to see if we were wasting ad dollars on customers who were already going to return and complete their purchase. The basic rule is to avoid handing out pizza coupons to customers already in the line at the pizza shop.

Our A/B test showed the retargeting ads had literally zero impact on customer behavior. The revenue per visitor from both samples was identical. There is an old adage “half of advertising doesn’t work, we just don’t know which half”. Testing discovered retargeting ads fall into the half that doesn’t work.