Gender bias concerns raised over GP app

Onlookers are asking why the chatbot created by Babylon Health – which provides the GP at Hand service – is offering such different guidance to men and women.  But the company tells PublicTechnology its service is working as intended.

—————————————————————-

“Where in your chest is the pain?,” the patient is asked.

Having responded that the pain is in the centre of their chest, they go on to answer a number of other questions, revealing that the pain came on suddenly, and is accompanied by a feeling of nausea.

The patient – who is aged 59, and smokes – is then told that two possible causes of their symptoms are depression or a panic attack. A GP appointment is advised in the former case, while the latter can be treated at home.

At the same time, another patient – also a 59-year-old smoker – presents with exactly the same symptoms.

They are told that, while they could be suffering a panic attack, they could also be suffering from gastritis – for which they should book an urgent appointment with their doctor.

Worse still, the cause of their symptoms could be one of several heart problems: pericarditis; unstable angina; or even a heart attack. A trip to A&E is recommended in the first case while, for the latter two, an ambulance should be called immediately.

“We have reviewed this since then, and we are confident that the medical evidence supports the outcomes. All the same, [we are conscious of] long-standing concerns about systemic bias in medical research and literature.”
Dr Keith Grimes, Babylon Health

These two sets of markedly different diagnostic possibilities come from exactly the same source.

In fact, the one and only difference between the two cases is that the first patient is a woman, and the second a man.

But the two people are identical in one key aspect: they do not exist. For that matter, nor does the doctor you may have assumed was providing diagnoses.

The exchanges above took place between the symptom-checker chatbot run by Babylon Health and two example users – created by an anonymous NHS consultant, who goes by Dr Murphy.

Responding to videos and pictures posted online by Dr Murphy, many people seem – understandably – deeply concerned by the program’s failure to even raise the possibility of a heart attack or other cardiological condition in the woman’s case.

Their worries are no doubt amplified by the fact that Babylon’s GP at Hand service is serving as an NHS GP for more than 50,000 UK citizens – one of whom, incidentally, is health secretary Matt Hancock.

Since late 2017, London residents have had the option of switching their registration from their community practice to GP at Hand – which offers remote video consultations via a smartphone app. The service recently launched in Birmingham and Babylon has said it wishes to expand throughout the UK.

The company directly responded to the criticism it faced on Twitter.

“Our system is trained using the best available clinical evidence, supported by subject matter experts. In this case it shows different causes listed for the same symptoms, depending on gender. That’s what the medical evidence suggests should happen,” it said. “The effects of bias can be present in research data, this is something that everyone working in healthcare needs to be aware of. It’s great to see your post has generated some thoughtful discussion around the implications of this forAI.”

Speaking to PublicTechnology, Babylon Health’s clinical artificial intelligence and innovation director Dr Keith Grimes – who still practises as an NHS GP one day a week – says that the symptom checker is an optional tool for users, and that patients do not have to use it before booking a consultation.

The AI behind the symptom-checker chatbot operates a probabilistic model, he adds.

Such models are defined as those that can consider the potential effect of unknown variables and suggest likely outcomes based on the available evidence – rather than offering a definitive answer.

The company also operates a model in which “every decision is recorded and can be explained – and if something has gone wrong, we can fix it”, according to Grimes.

“When people report symptoms or report information about their previous history… the chatbot will collect that information… the symptom checker uses information that has been provided by the user. It will also consider some [other] elements,” he says.

Such additional considerations include various “risk factors and the strength of link” they have with a range of conditions depending on the surrounding details.

In the case of the male and female users presenting chest pain: “Our app was working as intended at the time – it was providing information and a triage outcome,” Grimes says. “Clearly there are going to be differences in cases and in symptoms between men and women – they are biologically very different.”

He adds: “The cases presented on Twitter were a snapshot of a final outcome. We have reviewed this since then, and we are confident that the medical evidence supports the outcomes. All the same, there are long-standing concerns about systematic bias in medical research and literature – either conscious or unconscious. We scour [our services] to make sure that it does not show any signs of that.”

Although Babylon claims the chatbot was functioning correctly in this instance, further examination of the case by Grimes and the company’s other “expert clinicians” will take place.

“We were very, very careful not only in public testing, but also in being very aware of any feedback going forward,” he says. “We are still trying to understand what might lead it to behave in this way.”

He adds: “This is such a new area – there is always an opportunity to improve things. Our product is really good – and safe – but there is always the opportunity to make it a bit better. We can find out what happened here, and we can improve our processes.”

Bias views
Grimes believes that AI can even play an important role in helping to combat intrinsic bias, by “automatically identifying” its presence in systems.

Judging by the ongoing reaction online, others are less convinced.

But Grimes says the company “welcomes the scrutiny” brought by this case.

This is perhaps just as well, as the issue of recognising and eradicating bias is one that seems certain to remain in the spotlight. The Centre for Data Ethics and Innovation established by the government last year has identified bias – alongside targeting – as the two major issues on which it should focus its initial work.

Eleonora Harwich, director of research at think tank Reform and a member of the AI advisory board at the Kent, Surrey and Sussex Academic Health Science Network, says that bias can germinate in several ways.

Data quality is a significant cause, she says. Anyone using health data should asses it against the common definition of the six dimensions of data quality: completeness; timeliness; consistency; integrity; conformity; and accuracy, she says.

But even that is no guarantee of eradicating bias.

“Are you going to take the view that the data you are using is objective? Obviously, there are laws of science and physics, but I do think that the ways we go about collecting data are a social construct – and there are many different ways of measuring its purity, Harwich says. “If your data is from [NHS] trust A – which does not have a representative population – and you’re then using that data to train the model which is then used by trust B or trust C – it will not work.”

As the use of AI in healthcare becomes increasingly common and high-profile, in respect of Babylon’s technology – and others like it – many will continue to ask whether it is working or not.

Sam Trendall

Learn More →

73 thoughts on “Gender bias concerns raised over GP app

  1. 고산콜걸출장섹스마사지 November 14, 2024 at 2:51 pm

    https://oragoda.tistory.com/entry/긴급생계지원금-신청방법-및-필요서류

  2. 송도콜걸출장섹스마사지 November 25, 2024 at 1:09 pm

    벼룩시장 신문그대로보기 (구인구직, 부동산) 벼룩시장 신문그대로보기 바로가기 그리고 지역별 벼룩시장 종이신문그대로보기 방법 (구인구직, 부동산) 알아볼게요. 교차로신문 같이 벼룩시장은 지역별 일자리, 구인구직, 부동산 등 다양한 정보를 제공해요. 교차로신문그대로보기 바로가기는 아래에서 확인하고, 오늘은 벼룩시장 신문그대로보기 바로가기 그리고 사용법 섹스카지노사이트

  3. 송도콜걸출장섹스마사지 November 25, 2024 at 1:23 pm

    벼룩시장 신문그대로보기 (구인구직, 부동산) 벼룩시장 신문그대로보기 바로가기 그리고 지역별 벼룩시장 종이신문그대로보기 방법 (구인구직, 부동산) 알아볼게요. 교차로신문 같이 벼룩시장은 지역별 일자리, 구인구직, 부동산 등 다양한 정보를 제공해요. 교차로신문그대로보기 바로가기는 아래에서 확인하고, 오늘은 벼룩시장 신문그대로보기 바로가기 그리고 사용법 섹스카지노사이트

  4. Hairstyles December 3, 2024 at 9:32 am

    I was curious if you ever thought of changing the layout of your site? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having 1 or two images. Maybe you could space it out better?

  5. рейтинг москва December 7, 2024 at 9:53 am

    Attractive component to content. I simply stumbled upon your website and in accession capital to assert that I acquire actually loved account your weblog posts. Anyway I’ll be subscribing in your augment or even I fulfillment you get right of entry to constantly fast.

  6. 신림동콜걸출장섹스마사지 December 8, 2024 at 11:25 am

    이태원게이바

  7. 신림동콜걸출장섹스마사지 December 8, 2024 at 6:37 pm

    청도페이스라인출장

  8. 신림동콜걸출장섹스마사지 December 8, 2024 at 7:58 pm

    아름다운스웨디시업소

  9. 신림동콜걸출장섹스마사지 December 9, 2024 at 9:04 am

    대전호박나이트

  10. 상월동출장섹스마사지 December 12, 2024 at 11:16 am

    https://itgunza.com/195

  11. iziswap January 8, 2025 at 1:45 pm

    A great post without any doubt.

  12. minswap January 8, 2025 at 2:20 pm

    A great post without any doubt.

  13. Hairstyles January 16, 2025 at 8:19 pm

    excellent post, very informative. I wonder why the other specialists of this sector don’t notice this. You must continue your writing. I’m confident, you have a great readers’ base already!

  14. MetaMask Download January 18, 2025 at 12:15 am

    Thank you for joining us and supporting our content. Your attention matters to us!

  15. DreamProxies.com January 18, 2025 at 12:32 am

    Excellent post but I was wanting to know if you could write a litte more on this subject? I’d be very grateful if you could elaborate a little bit more. Bless you!

  16. Buy Best SEO January 18, 2025 at 12:59 am

    Today, I went to the beachfront with my children. I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear. She never wants to go back! LoL I know this is entirely off topic but I had to tell someone!

  17. Buy Best SEO Backlinks January 19, 2025 at 1:30 am

    This is very interesting, You are a very skilled blogger. I’ve joined your rss feed and look forward to seeking more of your magnificent post. Also, I’ve shared your site in my social networks!

  18. Cheap SEO Services January 19, 2025 at 10:57 am

    Hello my family member! I wish to say that this post is amazing, great written and include almost all vital infos. I’d like to peer extra posts like this .

  19. Best SEO Backlink Service January 19, 2025 at 1:00 pm

    Great website. Lots of useful information here. I’m sending it to a few pals ans additionally sharing in delicious. And naturally, thanks to your effort!

  20. Best SEO Services January 20, 2025 at 3:07 am

    My brother suggested I would possibly like this website. He was totally right. This post actually made my day. You can not imagine just how much time I had spent for this info! Thank you!

  21. 섹스노리야동코리아 January 21, 2025 at 3:03 am

    https://mintfin.tistory.com/tag/kbs20편성표

  22. 섹스노리야동코리아 January 22, 2025 at 4:39 am

    https://dnolife.net/software/melon/

  23. Hairstyles VIP January 22, 2025 at 11:38 am

    Keep functioning ,great job!

  24. 섹스노리야동코리아 January 24, 2025 at 10:33 am

    충무로출장업소

  25. sexo con travestis January 24, 2025 at 1:22 pm

    Dialogar con travestis y escort encantadoras es una ocasión única que mejora mi vida.

  26. 섹스노리야동코리아 January 25, 2025 at 4:53 am

    강남콜걸

  27. DreamProxies.com January 28, 2025 at 9:58 am

    I think one of your adverts caused my internet browser to resize, you may well want to put that on your blacklist.

  28. Xerox Premium Transparencies January 31, 2025 at 5:32 pm

    Hi there are using WordPress for your blog platform?I’m new to the blog world but I’m trying to getstarted and set up my own. Do you need any html coding knowledge to make your own blog?Any help would be really appreciated!

  29. Mens Hairstyles February 1, 2025 at 8:20 pm

    I do enjoy the way you have framed this challenge plus it really does present me a lot of fodder for consideration. On the other hand, from everything that I have experienced, I basically hope when other opinions stack on that men and women stay on issue and in no way get started on a tirade involving some other news du jour. Still, thank you for this outstanding piece and whilst I do not go along with it in totality, I value the point of view.

  30. Black Hairstyles February 2, 2025 at 11:58 am

    I used to be more than happy to search out this internet-site.I wanted to thanks for your time for this glorious read!! I positively enjoying every little bit of it and I have you bookmarked to take a look at new stuff you weblog post.

  31. Teodoro Jaggers February 2, 2025 at 4:32 pm

    This is an interesting perspective. It’s impressive how economic policies are impacting the world economy. Let’s see how this unfolds.

  32. Health February 2, 2025 at 9:59 pm

    I do not even know how I ended up here, but I thought this put up was once great. I don’t recognise who you might be but definitely you’re going to a famous blogger if you happen to aren’t already 😉 Cheers!

  33. https://sos-ch-dk-2.exo.io/catering-5/high-quality-catering-solutions.html February 3, 2025 at 5:53 am

    Thank you for sharing indeed great looking !

  34. curvefinance February 3, 2025 at 5:54 am

    A great post without any doubt.

  35. DefiLlama February 3, 2025 at 5:55 am

    The information shared is of top quality which has to get appreciated at all levels. Well done…

  36. Polygon Bridge February 3, 2025 at 5:56 am

    Merida returns to the deserted witch’s cottage, and discovers a message from the witch: she should « mend the bond, torn by delight » by the second sunrise, or the spell will grow to be everlasting.

  37. Fashion February 3, 2025 at 7:13 am

    I used to be suggested this website by my cousin. I am now not certain whether this publish is written by way of him as no one else recognise such special about my difficulty. You’re wonderful! Thanks!

  38. Cheap Proxies February 3, 2025 at 9:49 am

    BEST PRIVATE PROXIES – Elite high quality, Infinite data transfer, 1000 mb/s superspeed, 99,9 uptime, No consecutive IP’s, No practice standards, Multiple subnets, USA as well as European countries proxies – Buy Today – https://DreamProxies.com

  39. sexe toulouse February 3, 2025 at 4:31 pm

    Échanger avec des sexe a toulouse sur internet est une opportunité enrichissante.

  40. MinSwap February 3, 2025 at 7:21 pm

    Sight Care review

  41. Medium Hairstyles February 4, 2025 at 4:00 am

    A lot of thanks for all your labor on this blog. Kim delights in setting aside time for research and it is easy to see why. We all hear all concerning the lively means you produce vital solutions on your web site and even strongly encourage participation from the others on this matter and our child is in fact starting to learn a whole lot. Have fun with the rest of the year. You’re carrying out a great job.

  42. DefiLlama February 4, 2025 at 4:12 am

    A great post without any doubt.

  43. Polygon Bridge February 4, 2025 at 4:27 am

    Your passion and dedication to your craft shine brightly through every article. Your positive energy is contagious, and it’s clear you genuinely care about your readers’ experience. Your blog brightens my day!

  44. MinSwap February 4, 2025 at 4:33 am

    A great post without any doubt.

  45. Hairstyles VIP February 4, 2025 at 1:19 pm

    A lot of thanks for all of the effort on this site. Ellie loves working on investigations and it’s obvious why. I notice all regarding the lively ways you convey efficient guidelines on this blog and in addition recommend contribution from website visitors on this matter then our own simple princess is always understanding a great deal. Take advantage of the rest of the new year. You have been doing a stunning job.

  46. Hairstyles VIP February 4, 2025 at 6:05 pm

    Great post. I used to be checking constantly this blog and I am impressed! Very useful information specifically the ultimate part 🙂 I take care of such information a lot. I used to be looking for this certain information for a long time. Thank you and best of luck.

  47. Hairstyles February 5, 2025 at 12:12 am

    Hello, you used to write wonderful, but the last few posts have been kinda boring?I miss your super writings. Past few posts are just a little out of track! come on!

  48. curvefinance February 5, 2025 at 5:39 pm

    Great website, very user-friendly!

  49. Christoper Antenucci February 5, 2025 at 7:28 pm

    https://linkdirectory.at/de/detail/psychosoziale-beratungsstelle-weiz

  50. Beauty Fashion February 5, 2025 at 10:10 pm

    Woah! I’m really digging the template/theme of this blog. It’s simple, yet effective. A lot of times it’s very difficult to get that “perfect balance” between usability and appearance. I must say you’ve done a fantastic job with this. Also, the blog loads extremely quick for me on Firefox. Excellent Blog!

  51. Polygon Bridge February 6, 2025 at 1:59 am

    Pornstar

  52. Hairstyles VIP February 6, 2025 at 5:31 am

    After examine a couple of of the blog posts on your website now, and I actually like your approach of blogging. I bookmarked it to my bookmark website list and shall be checking back soon. Pls take a look at my web site as nicely and let me know what you think.

  53. iziswap February 6, 2025 at 9:16 am

    https://iziswaps.site/

  54. DefiLlama Swap February 6, 2025 at 1:18 pm

    You can certainly see your enthusiasm within the work you write. The sector hopes for even more passionate writers like you who aren’t afraid to mention how they believe. At all times go after your heart.

  55. Stake Eth February 6, 2025 at 7:08 pm

    https://iziswaps.site/

  56. DefiLlama DefiLlama Swap February 7, 2025 at 9:06 am

    Valuable info. Lucky me I discovered your website accidentally, and I’m shocked why this accident didn’t happened in advance! I bookmarked it.

  57. DefiLlama DefiLlama Swap February 7, 2025 at 10:07 am

    https://iziswaps.site/

  58. DefiLlama DefiLlama Swap February 7, 2025 at 10:29 am

    Thank you, I have recently been searching for information about this subject for ages and yours is the greatest I have discovered so far. But, what about the conclusion? Are you sure about the source?

  59. Beauty Fashion February 7, 2025 at 1:31 pm

    I have been exploring for a little bit for any high-quality articles or blog posts on this sort of space . Exploring in Yahoo I finally stumbled upon this website. Reading this information So i抦 happy to convey that I have an incredibly excellent uncanny feeling I found out exactly what I needed. I so much certainly will make sure to don抰 overlook this web site and provides it a look regularly.

  60. DefiLlama DefiLlama Swap February 7, 2025 at 4:05 pm

    Would love to perpetually get updated outstanding blog! .

  61. DefiLlama DefiLlama Swap February 7, 2025 at 10:41 pm

    https://prednibest.com/# where can i order prednisone 20mg

  62. https://s3.us-east-1.wasabisys.com/catering-4/gourmet-catering-menus.html February 8, 2025 at 8:51 am

    https://s3.us-west-004.backblazeb2.com/catering-5/customized-event-food.html

  63. 섹스노리야동코리아 February 8, 2025 at 3:53 pm

    https://blog.naver.com/combine8589/223743339311

  64. sushi swap app February 8, 2025 at 7:56 pm

    There is definately a lot to find out about this topic. I really like all of the points you made.

  65. Polygon Bridge February 8, 2025 at 8:47 pm

    I visited many websites but the audio quality for audio songs current at this web page is really superb.|

  66. Hairstyles VIP February 8, 2025 at 11:40 pm

    Thanks for your write-up. I also feel that laptop computers are getting to be more and more popular right now, and now will often be the only type of computer utilized in a household. This is due to the fact that at the same time that they are becoming more and more very affordable, their working power is growing to the point where they are as potent as desktop from just a few years ago.

  67. JAVASCRIPT February 9, 2025 at 9:35 am

    http://www.fiverr.com/s/Eg2Y0G9

  68. NODEJS February 9, 2025 at 10:28 am

    http://www.fiverr.com/s/Eg2Y0G9

  69. Ryan Iberra February 9, 2025 at 9:42 pm

    https://zurueckinsleben.co.at/praesentation/

  70. Nidia Fattig February 9, 2025 at 9:43 pm

    https://zurueckinsleben.co.at/potpourri/

Leave a Reply

Your email address will not be published. Required fields are marked *