Gender bias concerns raised over GP app

Written by Sam Trendall on 13 September 2019 in Features
Features

Onlookers are asking why the chatbot created by Babylon Health – which provides the GP at Hand service – is offering such different guidance to men and women.  But the company tells PublicTechnology its service is working as intended.

Credit: Adobe Stock

“Where in your chest is the pain?,” the patient is asked.

Having responded that the pain is in the centre of their chest, they go on to answer a number of other questions, revealing that the pain came on suddenly, and is accompanied by a feeling of nausea. 

The patient – who is aged 59, and smokes – is then told that two possible causes of their symptoms are depression or a panic attack. A GP appointment is advised in the former case, while the latter can be treated at home.

At the same time, another patient – also a 59-year-old smoker – presents with exactly the same symptoms.

They are told that, while they could be suffering a panic attack, they could also be suffering from gastritis – for which they should book an urgent appointment with their doctor. 

Worse still, the cause of their symptoms could be one of several heart problems: pericarditis; unstable angina; or even a heart attack. A trip to A&E is recommended in the first case while, for the latter two, an ambulance should be called immediately.

"We have reviewed this since then, and we are confident that the medical evidence supports the outcomes. All the same, [we are conscious of] long-standing concerns about systemic bias in medical research and literature."
Dr Keith Grimes, Babylon Health

These two sets of markedly different diagnostic possibilities come from exactly the same source. 

In fact, the one and only difference between the two cases is that the first patient is a woman, and the second a man.

But the two people are identical in one key aspect: they do not exist. For that matter, nor does the doctor you may have assumed was providing diagnoses.

The exchanges above took place between the symptom-checker chatbot run by Babylon Health and two example users – created by an anonymous NHS consultant, who goes by Dr Murphy.

Responding to videos and pictures posted online by Dr Murphy, many people seem – understandably – deeply concerned by the program’s failure to even raise the possibility of a heart attack or other cardiological condition in the woman’s case.

Their worries are no doubt amplified by the fact that Babylon’s GP at Hand service is serving as an NHS GP for more than 50,000 UK citizens – one of whom, incidentally, is health secretary Matt Hancock. 

Since late 2017, London residents have had the option of switching their registration from their community practice to GP at Hand – which offers remote video consultations via a smartphone app. The service recently launched in Birmingham and Babylon has said it wishes to expand throughout the UK.

The company directly responded to the criticism it faced on Twitter.

“Our system is trained using the best available clinical evidence, supported by subject matter experts. In this case it shows different causes listed for the same symptoms, depending on gender. That’s what the medical evidence suggests should happen,” it said. “The effects of bias can be present in research data, this is something that everyone working in healthcare needs to be aware of. It's great to see your post has generated some thoughtful discussion around the implications of this forAI.”

Speaking to PublicTechnology, Babylon Health’s clinical artificial intelligence and innovation director Dr Keith Grimes – who still practises as an NHS GP one day a week – says that the symptom checker is an optional tool for users, and that patients do not have to use it before booking a consultation.

The AI behind the symptom-checker chatbot operates a probabilistic model, he adds.

Such models are defined as those that can consider the potential effect of unknown variables and suggest likely outcomes based on the available evidence – rather than offering a definitive answer. 

The company also operates a model in which “every decision is recorded and can be explained – and if something has gone wrong, we can fix it”, according to Grimes.

“When people report symptoms or report information about their previous history… the chatbot will collect that information… the symptom checker uses information that has been provided by the user. It will also consider some [other] elements,” he says. 

Such additional considerations include various “risk factors and the strength of link” they have with a range of conditions depending on the surrounding details.

In the case of the male and female users presenting chest pain: “Our app was working as intended at the time – it was providing information and a triage outcome,” Grimes says. “Clearly there are going to be differences in cases and in symptoms between men and women – they are biologically very different.”

He adds: “The cases presented on Twitter were a snapshot of a final outcome. We have reviewed this since then, and we are confident that the medical evidence supports the outcomes. All the same, there are long-standing concerns about systematic bias in medical research and literature – either conscious or unconscious. We scour [our services] to make sure that it does not show any signs of that.”

Although Babylon claims the chatbot was functioning correctly in this instance, further examination of the case by Grimes and the company’s other “expert clinicians” will take place.

“We were very, very careful not only in public testing, but also in being very aware of any feedback going forward,” he says. “We are still trying to understand what might lead it to behave in this way.”

He adds: “This is such a new area – there is always an opportunity to improve things. Our product is really good – and safe – but there is always the opportunity to make it a bit better. We can find out what happened here, and we can improve our processes.”

Bias views
Grimes believes that AI can even play an important role in helping to combat intrinsic bias, by “automatically identifying” its presence in systems.

Judging by the ongoing reaction online, others are less convinced.

But Grimes says the company “welcomes the scrutiny” brought by this case.

This is perhaps just as well, as the issue of recognising and eradicating bias is one that seems certain to remain in the spotlight. The Centre for Data Ethics and Innovation established by the government last year has identified bias – alongside targeting – as the two major issues on which it should focus its initial work.

Eleonora Harwich, director of research at think tank Reform and a member of the AI advisory board at the Kent, Surrey and Sussex Academic Health Science Network, says that bias can germinate in several ways.

Data quality is a significant cause, she says. Anyone using health data should asses it against the common definition of the six dimensions of data quality: completeness; timeliness; consistency; integrity; conformity; and accuracy, she says.

But even that is no guarantee of eradicating bias.

“Are you going to take the view that the data you are using is objective? Obviously, there are laws of science and physics, but I do think that the ways we go about collecting data are a social construct – and there are many different ways of measuring its purity, Harwich says. “If your data is from [NHS] trust A – which does not have a representative population – and you’re then using that data to train the model which is then used by trust B or trust C – it will not work.”

As the use of AI in healthcare becomes increasingly common and high-profile, in respect of Babylon’s technology – and others like it – many will continue to ask whether it is working or not.

 

About the author

Sam Trendall is editor of PublicTechnology

Share this page

Tags

Categories

CONTRIBUTIONS FROM READERS

Please login to post a comment or register for a free account.

Related Articles

NHS staff unhappy at idea of big tech firms handling patient data
27 November 2019

YouGov poll shows little trust that large multinational companies will treat information with adequate confidentiality

Labour manifesto: five tech takeaways
26 November 2019

The free broadband plan has attracted attention, but the party has a number of other proposals for the use and regulation of technology. PublicTechnology rounds up the...

NHSX to appoint citizens to act as ‘patient voice’
12 November 2019

Tech organisation seeks to bring in two part-time advisors to help make services as useful and accessible as possible

GDPR blamed for doubling of Whitehall’s recorded data breaches
8 October 2019

Some departments report vast increases following introduction of new data-protection legislation