What you need to know

• Deloitte is already finding strong productivity improvements from a company wide rollout of its MyAssist Gen AI agent with 70% of users reporting a meaningful improvement in overall work productivity and 65% of users reporting an improvement in work quality.
• But while engagement with the app has been strong, there is also some evidence of churn in the monthly unique usage.
• With over 10,000 staff having tried their hand, perhaps that's to be expected.
• The accounting and consulting giant has also put a lot of effort into training and building capabilities, with "prompt like a boss" sessions and the creation of a Gen AI Champions group internally.
• Following a pilot, and the wider June rollout across the company, there has already been another significant upgrade in August, with image process capabilities added to enable users to extract information from visual content.
• And the firm's clients are benefiting directly and materially already, with faster delivery of projects and more accurate outcomes says an executive leading the program.

Three months after the launch of its MyAssist GenAI platform, Deloitte Australia says it is seeing significant improvements in workplace productivity among its 13,000 employees. 

Since its rollout on June 17, 2024, over 10,000 users have engaged with the platform, uploading more than 41,000 documents and creating about 650 personal chatbots and generative AI workflows. The tool aims to streamline common tasks, allowing employees to focus on more critical aspects of their roles.

Among the key learnings from the first three months:

• 70% of users reported a meaningful improvement in overall work productivity.
• 65% of users reported an improvement in work quality.
• About 650 personal chatbots and GenAI workflows were created and used.
• On average, 1.2 billion tokens were used per month by 5,900 unique users. 
• In the last month alone, 422 personal apps were created this month and those personal apps are getting very high (approx 80%) usage.
• Users expressed a desire for more role-specific applications and time to experiment with the platform. 

According to Kim Bracke, Deloitte Australia’s GenAI Program Change Lead, the trial found a core set of common tasks can account for up to 50% of the work time of some users, and it is these tasks the MyAssist platform has been optimised to support. During the trial, 70% of users reported improved productivity, while 65% noted enhancements in work quality. 

Deloitte worked with AWS and OpenAI when building MyAssist. According to Bracke, "We've got a dedicated team. It is one of those experiments that had a successful pilot, got lots of attention, and got stronger over time based on user feedback. As it's really designed by our people, for our people, so the features, the design, the fact that we can build apps on it, that's all tailored to what works well for Deloitte."

The most popular uses of MyAssist include researching internal and external sources, analysing documents, creating outputs like draft reports, and summarising meeting actions. Notably, MyAssist is garnering extensive use. However, despite this promising engagement, Deloitte acknowledges churn among users suggests not all employees are consistently engaging with the tool.

"We continuously capture user feedback to make the tool stronger. And obviously, this technology is evolving so fast, so any innovation that comes from OpenAI if it's relevant will also be incorporated onto our platform," says Bracke.

Different teams within Deloitte are starting to identify measureable benefits they can pass on to clients.

"We have some very specific use cases within our Oracle data team. They developed an app and spent weeks and months testing that app, but they're now deploying it on client engagements. They have developed a use case where they have not only reduced the time it took to deliver an app by about two weeks, but they also improved the accuracy and the quality of their work," Bracke continues. "And it freed up time for them to focus on what they enjoyed more in their work, which is spending more time with their clients to really understand their business context."

The platform continues to evolve with a release on August 16 introducing image processing capabilities. This upgrade allows users to extract information from visual content, responding directly to employee feedback. Such changes are part of a broader strategy to enhance user experience and promote adoption.

The implementation of MyAssist is not without its challenges. Bracke cautions that while users were initially impressed at the tool's capabilities, higher engagement also exposed some of the limitations. Some users mistakenly treat MyAssist as a search engine, misunderstanding its function as a pre-trained model designed to protect sensitive information. Additionally, the platform is not immune to the phenomenon known as “hallucination,” where AI models generate incorrect or misleading information.

"So it’s important to teach people how to identify this when it happens. It’s also important to remember that the purpose is to drive efficiency and help users perform better within the scope of their expertise," says Bracke.

Even after the pilot and initial rollout, Bracke stresses these are still early days when it comes to assessing the impact of LLMs on knowledge workers. And she is clear-eyed about the challenges ahead.

"When you look at some of those more transformational use cases, remember this is the worst version of GPT we will ever have. People who are early on now experimenting with it, it is trial and error." 

It is also important to factor in how much time people need to spend fine-tuning the solution, she says.

"But I think its potential in upcoming years is very significant, and that's also why we keep on investing. But that's also why we focus on upskilling our people and giving them the freedom to experiment with it, like giving them the time, the space, and the incentives, because that's important, to be willing to work faster."

Chinese walls

As Deloitte's rival PWC discovered during the controversy over its taxation consulting services to the Commonwealth Government, clients can be very particular about how firms manage their potential conflicts of interest, a point not lost on Bracke.

"One of the guardrails is when you build your own workflow app - for instance in ANA, they've built their own workflow app - we cannot access that from Consulting, because it's a bespoke app. They have uploaded a knowledge base and documents as inputs that are confidential. That's basic access security, of course. So that's definitely one of the guardrails that is in place," she says.

About 1000 people were involved in the pilot project prior to the wider rollout in June. "So it already had quite a fair bit of interest," says Bracke.

"Then, before we launched it in June, we did a Gen AI fluency month, where we created awareness around what is Generative. AI, what is responsible and safe AI use? And when we launched the platform, we also launched ethical training on the safe and responsible use of AI. We had onboarding sessions. We've had 'Prompt like a boss' training with basic prompt training. All these are really foundational."

But for Bracke, the most powerful driver of adoption was an internal community called Gen AI Champions.

"We invested a lot of time with these people to gather feedback on the ground - what's working well, why aren't your colleagues using it, what are the main barriers? What are the psychological barriers? We really sought to understand those barriers to adoption, through focus group workshops, and then we took action to address those barriers."