► The first cancer patient in South Korea to receive Watson's oncology-assisted treatment is browsing information about his disease with an oncologist at Kagawa University's Yoshi Medical Center. Source: Kagawa University Yoshi Medical Center
Recently, Watson, IBM's AI product used to assist doctors in designing cancer treatment plans, has been exposed to numerous problems, including the possibility of prescribing dangerous and incorrect cancer treatment plans. What impact will this have on the healthcare AI industry? Several experts working on medical AI research or industrialization expressed their views.
By | | | | CHIANG HOI YU
Editor Chen Xiaoxue
"This product is very bad. We were buying this product for a hospital promotion in the hope that it would deliver on IBM's stated vision. It turns out that for the most part, it doesn't work at all. "A doctor at Jupiter Medical Center (Florida, USA) said this about their Watson for Oncology in the presence of IBM executives.
Yes, the same Watson, the medically assisted artificial intelligence system that was once favored and stirred up so much controversy. Watson Oncology is very easy to use: simply enter the patient's personal information (e.g. medical history, imaging reports, etc.) into the system, and the system recommends the appropriate treatment plan for the doctor's reference based on a large amount of medical research, medical guidelines, clinical trials, and other information.
On July 25, STAT, a US health care media outlet, exposed internal documents from IBM showing that Watson, an AI product used to assist doctors in designing cancer treatment plans, is notoriously problematic: in addition to the kind of complaints from users above, it can prescribe dangerous and incorrect cancer treatment plans.
Watson Oncology is IBM's key product in the field of medical artificial intelligence, capable of providing treatment plans for a variety of cancers, including breast, lung, rectal, and colon cancers, based on patient medical records and other information, with partners such as Memorial Sloan Kettering Cancer Center, Mayo Clinic, Quest DIAGNOSTICS, and other prominent institutions in the medical field. According to the official website of its Chinese agency, nearly 80 hospitals in more than 20 provinces in China have introduced Watson Oncology Consultation Center.
IBM "family scandal" leaked
STAT received internal documents from Andrew Norden, then IBM Watson Health's deputy chief health officer, in the summer of 2017, on slides used in an internal company presentation.
According to STAT, IBM is unrelenting in its criticism of Watson in this report, with key points including.
The ∆Watson system is trained using data not from real patients, but from hypothetical patients Hypothetical data。
△ Insufficient training data . The slide shows eight types of cancer. As of the day of the report, the highest volume of training data was only 635 cases of lung cancer, while the lowest was only 106 cases of ovarian cancer.
△ When training Watson, the treatment regimen recommended for the hypothetical patient was based on the protocols of specialists at Memorial Sloan-Kettering Cancer Center, the Not medical guidelines or real evidence。
Δ In the test of a hypothetical situation. Watson prescribed an inappropriate and dangerous treatment plan . For example, it recommends concurrent chemotherapy and Avastin for a patient with lung cancer who shows the possibility of being bleeding severely. However, Avastin may cause severe bleeding and should not be recommended for patients who are already bleeding.
△ The experiments used to assess the similarity of protocols between the Watson system and cancer specialists may be biased, making the two protocols easily identical.
In an interview with STAT, Jana Eggers, CEO of artificial intelligence company Nara Logics, noted that the Watson system is clearly not using the big data in the healthcare system, "I can't figure out why they're generating a bunch of fantasy patients out of it when there's real data from real people. "
Some experts believe that， If these Hypothetical data Representative of real patient conditions， It also trains Watson very well.。 merely，“ We also need to see evidence that the data is representative”， Associate Professor, Center for Bioinformatics Research, Stanford UniversityJonathan Chen talk。
In fact, the questioning of Watson never ends. Back last year, in 2017, STAT published a survey of Watson's healthcare AI system that addressed all of these issues. Professor Claudia Perlich, formerly of IBM's Watson Research Center, said in a 2017 interview with Gizmodo that Watson Healthcare is "child's play": "Our perception in the data science community is that anything Watson can do, you should probably find free software to do it, or make one yourself. "
This internal document, more than anything else, exposes IBM's own perception of Watson's limitations.
In May of this year, IBM made cuts to its Watson Healthcare division. IBM spokesman Ian Colley told the Institute of Electrical and Electronics Engineers (IEEE) that only a small number of Watson Healthcare staff were laid off and that the layoffs were only part of a streamlining of the department. However, one of the engineers who was laid off revealed that about 80% of the employees were forced to leave. "The people who left were important technical people and people who dealt face-to-face with customers, not insignificant administrative people." This engineer said.
Recommended treatment is not mainstream
There are many application scenarios for artificial intelligence in healthcare, including medical imaging, genomic information processing, drug development, health data management, guide robots, and more. Applications in clinical care have the most significant impact on existing medical practice and therefore receive the most attention. Applications like Watson Oncology, which attempts to recommend treatment options based on patient information, are just one of many, and not the mainstream.
Currently, in the clinical context, medical AI is mostly related to medical imaging, focusing on providing doctors with various tools or optimizing existing imaging tools to help them diagnose or treat.
For example, AI diagnostic tools can identify lesions from medical images that are invisible to the human eye, or easily missed. In this regard, screening for lung cancer based on CT image identification of lung nodules is one of the most common applications today.