Shadow AI in healthcare: unauthorized tools pose growing risk to NHS

Full text

Given the pressure that NHS doctors are under, it’s no surprise that many trusts are turning to AI for help. On the whole, this is a good thing.

AI tools can play a valuable role in managing workflows, automating documentation, coordinating care rotas and streamlining communications.

However, the pace of AI development, and generative AI in particular, has accelerated dramatically.

As a result, governance frameworks are failing to keep pace with frontline demand and healthcare practitioners have begun to use tools that have not been authorized by their organisation.

This is causing problems at all levels - particularly when those tools are used to support clinical decision-making.

AI in the shadows

“Shadow AI” is a term commonly used to describe the unauthorized use of AI within organizations. While the phenomenon exists across all industries, it has been growing rapidly in healthcare. One recent US survey revealed that 17% of healthcare workers admitted to using unauthorized AI tools in the workplace.

A another survey found that 39% of frontline healthcare staff say they use generic, free AI tools weekly or more. These are not isolated experiments. AI is already becoming embedded in day-to-day operations, often without formal governance or oversight, despite well-documented risks including hallucinations, inconsistencies and biases.

The reasons for this are understandable. Healthcare professionals are dealing with staff shortages, rising patient demand and overwhelming administrative workloads. Clinicians are expected to document more, communicate faster and process growing volumes of information - all while maintaining high standards of patient care.

It’s not surprising that they are reaching for the nearest AI tool. Particularly when that AI tool is extremely accessible, free or low-cost and easy to use. Shadow AI usage in the healthcare sector currently ranges from frontline clinicians right through to back office functions such as report drafting and recruitment support.

Most health professionals using “shadow” tools are not deliberately flouting rules, therefore. They are simply accessing some much-needed help from a convenient source. And their employer is not providing them with a robust alternative.

Healthcare leaders need to act quickly

Medical organizations, and the NHS in particular, have often been cautious to roll out new technologies, though frequently with very good reason. Clinical systems require extensive testing, strong evidence bases and rigorous governance before deployment and decisions supported by AI can have direct consequences for patient outcomes.

Any clinical decision-making tool must therefore be thoroughly tested, based on solid medical research and exist under constant review and improvement. Yet delays allow shadow AI to fill the gap.

AI tools of the right quality are now reaching the market but the problem remains as to how leaders can rein in the use of shadow AI. It’s a problem that must be addressed quickly because the stakes are high.

The risks are real

The risks of unsanctioned AI usage include security breaches and privacy violations as well as the top concern - patient care.

In terms of data security and privacy , generic AI solutions are effectively “black boxes”. Ask ChatGPT whether sensitive health data will be safe in its hands, for example, and it will advise that it does not comply with health information regulations.

This is not surprising - patient information is “special category data” under GDPR and frequently requires additional protection - but it illustrates the danger of submitting too much information into an unauthorized app.

When it comes to patient care, generic AI solutions used to support clinical decision making carry an inherent risk: even when outputs appear credible, errors may be difficult to detect and assess. They may pull information from a broad range of sources which may be outdated, unaccredited or inaccurate.

A clinician in a hurry, referring to an unauthorized tool, is taking a big chance. If an unauthorized tool provides incorrect guidance relating to symptoms, medications, diagnoses or treatment pathways, the consequences could be severe.

Guidance and leadership, not heavy-handed authority

The answer is not a blanket ban, which is likely to send unauthorized AI further into the shadows. The focus for healthcare leaders must be to establish clear guidelines on the use of AI and to invest in education and training regarding its risks and benefits.

At the same time, organizations need to provide clinically validated, secure and easy to use alternative tools. If approved systems are too slow to deploy or fail to meet operational needs, clinicians and administrators will inevitably continue to turn to unauthorized tools instead.

Governance structures are also an important part of the process. These should include IT teams, security specialists and operational management because AI deployment simultaneously affects patient care, compliance, workforce efficiency and organizational risk.

AI will undoubtedly become an increasingly important part of modern healthcare delivery. The organizations that succeed will not necessarily be those that adopt AI fastest, but which create trusted frameworks on which clinicians can access secure, medically robust and properly governed tools.

We feature the best Electronic Health Records software .

This article was produced as part of TechRadar Pro Perspectives , our channel to feature the best and brightest minds in the technology industry today.

The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/pro/perspectives-how-to-submit