DDB Melbourne, AI consultancy, Pow Wow Solutions and independent Australian news organisation, Crikey, have come together to develop a tool dubbed the 'Bullshit’o’Meter'. The tool is designed to demonstrate how artificial intelligence (AI) can distort news, highlighting the ease with which truth can be manipulated in significant stories, such as the US election.

Crikey has been vocal about its negative stance on AI-powered journalism. The news organisation has argued that large language models (LLMs), often used in AI journalism, frequently tell lies and have in-built biases. "AI powered journalism stinks. Large language models (LLMs) produce ‘bullshit’ - not our word - ‘bullshit’ is the technical, scientific, term - when it frequently and confidently tells lies. Even when AI tells the truth, its answers have in-built biases," Crikey stated.

The Bullshit’o’Meter uses AI to polarise selected news stories, demonstrating to users how easily bad faith news outlets can distort the truth. New stories are routinely added to the Bullshit’o’Meter, allowing users to experiment with breaking news and polarise the election’s biggest stories with their own AI-powered newsroom. All articles created using the Bullshit’o’Meter cannot be copied or pulled off the Crikey platform, ensuring that the polarised AI stories cannot take on a life of their own.

Sophie Black, Crikey Editor-in-Chief, explained the motivation behind the tool: "We think AI generated journalism is one of the most important and rapidly evolving stories going on at the moment. In addition to our coverage of the issue, we wanted to give readers the chance to really understand how the technology works, and how easily it can distort the news. We’ve helped design the Bullshit’o’meter to teach news consumers to recognise the slop that’s being pumped out so they don’t fall victim to it. We’re using the power of AI, in a very small dose, to show readers how corrosive it can be."

The tool is being promoted through an editorial on Crikey.com and a range of activity across the Crikey network. The initiative aims to educate the public about the potential dangers of AI in news reporting. A recent study found that 54% of Australian respondents had no awareness or understanding of the use of Generative AI in news reporting, and 90% were somewhat, very or extremely concerned about it. Among those with strong concerns, 65% called for mandates on AI modified articles, 62% asked for more education to identify AI content, and 61% sought greater awareness of the impact of AI reporting on the news.

"Everyone involved in this project understands the threat to journalism accelerated by GenAI, but we also believe in the technology's capacity, in very small calculated doses, to educate people on how it can be used for the wrong purposes," said Psembi Kinstan, DDB Group Melbourne Executive Creative Director.

Justin Beaconsfield, Pow Wow Solutions Co-Founder, explained the technical process behind the tool: "We trained an LLM to highlight instances of language that are (A) factual, or (B) representing commentary that is subject to bias or emotiveness. We then fine-tuned the LLM to mimic various personalities whilst rewriting this commentary by training models on articles from different media outlets that employ various levels of bias. Once the models had seen enough instances of relevant language, they could consistently mimic this style."