Detective Del Spooner, Will Smith’s character in the 2004 sci-fi hit I, Robot was right to be sceptical about machines turning against humans, but was wrong in the dystopian picture it painted.

Rather than machines taking all of our jobs and murdering us in our sleep, a more likely threat is that AI technology will power a frictionless lifestyle that might not kill us, but cause a more painful existence over time, eroding our agency and other less visible threats, or “living with thousands of paper cuts”.

In other words, it will be less I, Robot and perhaps more Wall-E – the Pixar classic where humans, who long destroyed and fled Earth, degenerated into helpless obesity, largely due to laziness and robots catering to their every whim.

That’s the “catastrophic” assessment of a Gen AI-powered world by Amy Webb, the founder and CEO of the Future Today Institute, who is concerned about the direction of travel, speed and investment behind Generative AI technology.

She said the technology could radically alter how humans interact with machines, creating a world where "aggressive data scraping leads to the death of search", which is replaced by a relentless loop of "aggressive curation and recommendation", ultimately herding us down an ever-narrowing path.

“We talk all the time about bias, which is a real thing, but what we don't talk about is homogeneity,” she told thousands of delegates at a packed conference hall at SxSW Sydney.

“I think a lot of people right now believe that we just have unfettered access to be as creative as ever, and AI systems are going to unlock our brains, we're going to invent all kinds of crazy new things, but I actually think the opposite is going to start happening,” she added. 

“There's going to be less choice, there's going to be more directed decision-making. It's going to feel like life is very frictionless and convenient, when what's actually happening is you're being directed, either obviously, or in a way that's not so obvious to make the same couple of choices.”

Forget FOMO

Living in a world where technology auto-populates human decisions could have dire consequences for brands, because it is likely to reduce choice, warned Webb.

She cited her own experiments with Instacart’s ChatGPT-powered recipe and grocery recommendations, noting the incoming arrivals of plug-ins across commerce sites that will ultimately lead to auto fulfilment as a standard option.

“It auto-populates [the ingredients for the recipe]. When I did this [with Instacart] it was all the same brand,” said Webb.

“For those of you who work in consumer, like CPG or retail, you've probably just had a shiver, because this completely destroys your business. If what I'm doing as a consumer is setting parameters, I just want whatever's cheapest,” she said. “We all know that there are other tricks and tools that happen in marketing to get people to change their mind and convert them to what you want.”

To some degree, this already happens on platforms like Amazon’s marketplace, in which it recommends “Amazon choice” products when users search for items. Google has also been fined in the past for nudging people towards its own comparison shopping service. 

“[Amazon] is such a powerful business because it removed all those points of friction,” Webb said. “So if it's the case that we are continuously creating data all day long in different forms, and there are systems that can mine, refine and commercialise the use of that data in ways to make our lives friction free, that fundamentally changes our relationship to machines and the relationship that data has to third parties. It completely upends the internet.

She also warned against companies jumping into Gen AI blind for fear of missing out.

“Where is that line between human and machine? All of this is becoming invisible to us. My concern here is that we're going to rush forward because it feels very much like our lives will be better and more fulfilled with all of this technology in it. But at some point, we're going to realise that we've lost agency and that the consolidation of power that exists right now, among just a handful of technology companies, becomes even more entrenched.”

A creativity killer or enabler?

Not only will Gen AI remove friction and interrogation, but in the age of assistive computing, people may never have to think on their own again. This tech has long existed – word processors and auto correct have removed the need for spelling accuracy, but Gen AI is likely to accelerate this into other realms, including some areas of the creative industry.

Webb believes Gen AI could take over formulaic creative work, managed by humans.

“For the vast majority of creative work that we all love and enjoy, that's just not something an AI system is going to be able to accomplish in a way that makes sense,” she added. “If we are thinking about these as assistive tools rather than as replacement tools, we're going to be better off.” 

She said that companies that are looking for AI to reduce a permanently smaller workforce will find the going tough in the medium to long-term, because AI tech could create far more jobs than it replaces.

Other terrifying examples of how technology can be used include the use of deepfakes to alter medical records, including the detection and treatment of cancers, and law enforcement using functional MRI scans after issuing search warrants for a suspect brain.

On the plus side, added Webb, AI tech is being used to help people live longer and healthier lives – even if they have to endure more metaphorical paper cuts along the way.