What you need to know:

• Experimentation is an emerging science with momentum that is on the cusp of challenging personalisation as a revenue and customer experience "oil field". 
• The concept was pioneered by Optimizely, a company founded by two former Barack Obama campaign staffers who discovered changing pictures and language drove 40 per cent higher donation numbers. But it's now being used in online shopping, customer experience design and in-store.  
• Deloitte Digital’s Nima Yassini says Google is investing in the experimentation area, and is likely now the market leader, ahead of Optimizely and Adobe’s Target software - and it's not necessarily a good thing.
• ‘Personalisation’ has been an expensive buzzword pushed by tech companies that can make them lucrative margins but is hard and expensive to operationalise. Content and Context is great, Yassini reckons, but with no experimentation programs, is likely missing significant new revenues.
• It's about sweating the small things. Coles’ Fallyn Lowe says the supermarket giant discovered people who selected a delivery slot before filling their basket had a higher conversion than those who chose the delivery time at the end of the purchase  – it's an example of how an experimentation program led the supermarket giant to actively change how it designs its online shopping experience and lift revenues. 

The experimentation optimisation conundrum

Brands will waste money on half-finished personalisation strategies sold by major tech companies, or fruitlessly run AB tests with the wrong focus, when they should swing more to ‘experimentation’ programs. Google has clocked the trend and has suddenly emerged as a market leader in experimentation software. 

Instead of AB tests, which have traditionally focused on digital campaigns and customer comms, experimentation takes it a step further to e-commerce, website design, customer experience, call centres and bricks and mortar stores.

Coles, for example, has learned lessons from the online basket behaviours of two customers cohorts and is applying that to its whole customer journey. “It’s about understanding what you’re trying to achieve – what is the customer problem?” Fallyn Lowe, Coles’ Product Manager for Growth and Optimisation, says.

Nima Yassini, a former WPP ad agency network exec who co-founded experience optimisation firm New Republique before it was acquired by Deloitte Digital earlier this year, is now a Deloitte Digital Partner and says the current pursuit of ‘experimentation’ first started with a company called Optimizely.

Optimizely was born out of former US President Barack Obama campaign analytics experts, who realised back in 2010 that a photo of Obama with his family drove more funding than an image with American voters. Likewise, adding ‘Learn More’ instead of ‘Join Us Now’ or ‘Sign Up’ drove 40 per cent higher results – and likely $60 million in additional donations.

“It started as ‘AB testing’, then it went to ‘conversion rate optimisation’, lovingly known as CRO. In between, there was ‘test and learn’ and then it evolved to ‘experimentation’. That's the evolution of the word,” Yassini says.

“It's more than just a sales event occurring. Sometimes it's about loss events not occurring.”

The examples are compelling. One unnamed legal services company spent a million dollars a month with Google to make their phone ring – and had a large call centre team as a result.

“Our job was to come in and help that phone ring,” Yassini says. “I had a strategist who came and said, ‘I want to put a green dot next to the phone number [on the website]’. And I'm like, ‘Why?’ … She's like, ‘I just have this hunch. I want to try it’. I was like, ‘Okay, give it a go’. So she put a green dot next to a phone number, 1300 whatever, green dot.”

A red dot has become virtually synonymous with something being recorded. But a green dot is an implicit sign that something or someone is ‘live’ – that, in the case of a call centre, signalled there were people ready to answer the phone immediately.

“Sales went up 13 per cent – just from a green dot,” Yassini says.

In another case, involving an insurance company’s call centre which was getting smashed in the early days of Covid. Customers were calling in to ask about premiums, whether or not they were covered, and waiting longer and longer for answers as a result.

Yassini says 60 to 70 per cent of callers were on the website as they called, so a chat icon was quickly created and linked to Frequently Asked Questions (FAQs).

“We reduced call rates by about seven per cent, which was massive at the time when they were getting slammed,” he says.

“That's got nothing to do with sales. It's got to do with customer satisfaction, NPS and customer management. It was helping customers to be a little bit more self-directive and self-managing.”

But experimentation and optimising the customer experience isn’t always so clear cut. Sometimes it’s just a step. Yassini says comparison sites can deliberately bury information in sub-headings to find out which areas prospective customers want to compare.

“If there are five accordions, I can see which accordion opened first, second, fourth, third, which then tells me the order in which you want to compare products on,” he says.

“Once I've got that insight, I now will design the experience for comparison.”

But there are no silver bullets – those days are gone, Yassini says.

“Those big techs that deliver these massive outcomes are gone. It's the world of death by 1,000 cuts,” he says.

“This is where I want to try and get marketers' headspace to. Within the assets that you have right there is untapped oil pockets. If you can just untap those, you're looking at millions of dollars of gain or millions of dollars in cost efficiencies that you could find. But it's just understanding how to use this properly.”

Two Coles e-com customers

Coles is figuring this out. Through its experimentation program it realised there were two primary behaviour flows on its e-commerce website. The first, “the most self-explanatory one”, per Lowe, is that customers land on the Coles Online website, shop around, put items in their basket, then go to the checkout.

“That's quite a simplistic journey and that's what a cohort of customers do,” she says.

The second cohort, however, does things in reverse. “They make this decision, an active decision, to go in, select their delivery slot, and then from there, once they’ve selected the slot, they then start building up their basket and go on to checkout,” she says.

Even more interestingly, though, was that customers in the second flow had a notably higher conversion rate.

“There was something happening there that was providing a better experience for them. And what my team then wanted to do, is understand what that meant for those first cohorts.”  

Choosing the delivery slot first sets customer expectations early. It established which store the items would come from, meaning only products in stock were shoppable. 

“You start with the you want to be able to drive an increase in conversion, but you can do that through some really interesting customer experiences and understanding the behaviour of your customer and leaning into what they're probably telling you already - that path of least resistance that's sitting out in front of you,” Lowe says.

“Rather than redesigning everything… we came up with a test that kind of prompted the user, as they started building their basket, to consider selecting their [delivery] slot and use different tactics to see if we were able to do that. And that's definitely an easier way for us to do it than re-doing the entire site.”

Experimentation survey

Deloitte Digital has shared the results of a new survey of 250 marketers about the state of experimentation in Australia. One key finding: Marketers don’t necessarily own experimentation. It's being driven by other parts of the organisation.

“It's quite broken, "Yassini says. "There's no one entity that owns it. You can see that close to 40 per cent is still heavily on the marketing side.

“More and more as product teams start to formulate, there's about 39 per cent of companies where experimentation sits within the product team … There's only a small percentage, about 13 per cent, where the R&D teams are involved in experimentation.”

Yassini says some have focused on ‘personalisation’ – to their detriment.

Personalisation versus experimentation

Brands have piled into personalisation over experimentation because that’s where the money is for tech companies, Yassini says. Tech players have had the incentive to sell complex personalisation products because they can make money, even if the end result isn’t very functional.

“It costs time and costs money… personalisation became this buzzword. Media pushed it, the technology companies pushed it, and so the conversation became very topical about ‘hyper-personalisation’,” Yassini says.

“But if you think about personalisation at a very basic level, personalisation is about a context and content… if you're looking for shoes, I show you shoes. You're looking for sneakers? I show you sneakers. And if I know you like Nike, I show you Nike sneakers. Bang. Personalised, right? Content, context. Where that content sits on a page, you can imagine a web page is a very long thing, where it sits on a page, what it looks like, how it's messaged and presented to me, where it is in my journey, whether it's on the homepage or down on a PDP page. All of these things have nothing to do with personalisation - everything do with experimentation.

“So what happens is you create this wonderful container that contextually connects, and the content is really relevant. But where it sits, what it looks like, does not resonate. And the challenge is, experimentation is not expensive to do or implement. Experimentation is actually quite cheap if you compare it to personalisation – it's quite cheap. And so the money is in personalisation.”

Everyone went to personalisation and forgot about experimentation, in other words. Products are now personalised, but “instead of pissing off five people, you’re now pissing off 500 people” because the funnel is broken.

“I'm not saying personalisation is not right,” Yassini adds. “All I'm saying is you need both of them to work together to drive the business outcomes that everyone's promising you.”

Burnt fingers

Coles will take what it’s learning from its e-commerce experience and apply that across its ecosystem, Lowe says.

“We're predominantly focused on [the customer] front end, so starting to think about how we experiment. server side is where we're shifting from a website perspective as well,” she says.

“For us, omnichannel is a huge one. We're starting to think about how we experiment within the store and across both store and digital is something really exciting and something that I think we want to start dipping our toes into.”

The experimentation economy is taking off – and that means a lot of “cowboys” jumping on the bandwagon, Yassini says. Like Search Engine Optimisation, which was discovered, subsumed by Google, and ultimately taken as a service offered and pitched from overseas companies, experimentation risks falling into the same basket.

“My prediction is there's going to be a lot of burnt fingers in the market. The biggest problem with this space is anyone can say anything and people just seem to believe them because there's no barriers to entry,” he says.

“There's a lot of companies who didn't exist or didn't say they do experimentation and then wham, bam, a year later, all of a sudden, they're experts in experimentation because they hired a guy who read a book. 'Experimentation for Dummies' – right? And the biggest problem is, who do they learn on? They learn on the client's website. And who pays the money and creates all the opportunities? It's the client…

“We're seeing that growth with Google Optimize starting to increase in popularity, what's going to happen is a lot of people who don't know what they're doing, they're going to use the client's budget to experiment and learn. And the only person who's really going to get burnt ultimately is the client. So watch your fingers. Watch your toes. It's scorching season.”