ai

Why Everyone is Hating on IBM Watson - Including the People Who Helped Make It

By Jennings Brown on at

You might have seen an advert for Watson, where what looks like a sentient box interacts with celebrities like Bob Dylan, Carrie Fisher, and Serena Williams; or doctors; or a young cancer survivor. Maybe you even caught the IBM artificial intelligence technology’s appearance in the ad starring Jon Hamm. “It is one of the most powerful tools our species has created. It helps doctors fight disease,” Hamm says. “It can predict global weather patterns. It improves education for children everywhere. And now we unleash it on your taxes.”

In the commercial, which advertises what is essentially a smart tax prep service, Watson is portrayed as a glowing sci-fi cube that holds the key to humankind’s greatest problems. But it’s not a wizard. And lately, several experts from Silicon Valley and Wall Street have spoken up, criticising the people behind the curtain—asking if Watson is a joke or a saviour for IBM. So why is everyone being so tough on Big Blue and its golden child?

The splashy vision of Watson has been integral to IBM’s branding since the groundbreaking question-answering system made its debut in 2011 on Jeopardy!. Now, thanks to billions of dollars of investment and years of aggressive marketing, Watson has come to represent AI in the popular imagination. Siri and Alexa may get more attention these days, but when it comes to big-data computing, Watson was the first to offer up its name, and it has remained a cocksure mascot.

On June 20th, the day after IBM CEO Ginni Rometty met with President Trump, as a member of the American Technology Council (a collection of Silicon Valley top dogs who advise the president on modernising federal technology) she appeared on CNBC’s famously bad show Mad Money for a press release presented as an interview. “You were at The White House yesterday,” Cramer began. “What did you talk about? And what can IBM do for our country?”

Of course, the answer is, a lot, according to Rometty: IBM can improve health care for vets, modernise the nation’s tech infrastructure, create jobs, and train the next generation of “new collar” workers. But it’s the “cognitive computing” (IBM’s buzzword for AI) of Watson that will help many of these visions come true. Peppered throughout the softball exchange were a dozen mentions of Watson. She made deceptively aggrandising claims that by the end of this year Watson “will touch a billion people” and “be able to address, diagnose, and treat 80 percent of what causes 80 percent of the cancer in the world.” (She was referring to their goal of being able to provide treatment suggestions for 12 types of cancer, which IBM says represents 80 percent of the global incidence of cancer. The billion figure is largely thanks to a collaboration between Watson and GM OnStar that theoretically puts Watson in hundreds of millions of cars.)

The interview was representative of IBM’s marketing strategy: promote Watson as a world-changing technology, describe it in quixotic ways, then let reporters run wild with clickbait-y headlines about a robot helping humans do their jobs—for example: “IBM’s Watson Supercomputer May Soon Be The Best Doctor In The World,” “IBM’s Watson Helped Design Karolina Kurkova’s Light-Up Dress for the Met Gala.”

IBM seems to believe the Watson brand can breathe new life into their company. And it sure could use some resuscitation right about now. IBM’s revenue has fallen for 22 consecutive quarters. In May, Warren Buffett dumped about a third of his IBM stock, citing “some pretty tough competitors.” Two weeks later, the Wall Street Journal reported IBM gave its remote employees the option to either move to a regional office or quit. (Since the decision affects more than 40 percent of its 380,000 employees, the article suggests it’s a way of cutting employees without official layoffs.) Then in July, investment bank Jefferies published a report cautioning IBM investors, suggesting the company won’t return value to shareholders because it can’t compete with other tech giants investing in AI.

Amazon, Apple, Facebook, Google, and Microsoft are all retooling their businesses around the belief that AI and machine learning are the future of the tech industry. IBM is in a more vulnerable position than all those companies. Even though IBM was an AI pioneer it has let its lead slip and damaged its reputation with overhyped marketing. There’s a rising sentiment in from tech and finance experts that, for all the idealism, Watson just can’t deliver on its promises.

What is Watson, really?

When Watson debuted in 2011, it was an outstanding achievement that proved the century-old hardware company could still find its way to the bleeding edge. It ushered in the era of big-data computing, or as Rometty put it during her conversation with Cramer: “We are the ones that woke up the AI world here.”

Watson was more than just a supercomputer that answered trivia questions. It could process natural language, including wordplay and unusually phrased questions. When Watson was given a question, dozens of algorithms analyzed the query and produced a list of responses, then ranked the answers. If the confidence score was high enough, Watson hit the buzzer.

Three years later, in 2014, IBM created the Watson business unit to figure out ways to use the technology to actually make money. Now “Watson” represents a whole array of AI technologies built and acquired by IBM—including sentiment analysis, voice recognition, and natural-language processing. UX and UI designers use these tools to create platforms for IBM clients in various industries. Many of these collaborations inspire a wave of buzz, generating headlines that read like the newspaper clippings collected by proud parents of a prodigy: “IBM Watson Gets Its First Real Job,” “After Winning Jeopardy! IBM’s Watson Takes On Cancer, Diabetes,” “The Toronto Raptors Are Using IBM’s Watson to Draft A Winning Team,” “IBM’s Watson has published a cookbook,” “Watson helped make a trailer for a horror movie about AI,” “The Latest Job for IBM’s Watson Is as a Hotel Concierge,” “IBM Watson Helps Create Sculptures Inspired by Gaudi,” and “IBM’s Watson is making music, one step closer to taking over the world.” (Some of those were from Gizmodo.)

Many of the products mentioned in those articles rely on machine learning—algorithms that “learn” how to perform tasks by processing massive amounts of data (hence the buzzword big-data computing) and finding patterns in the information. It’s kind of like how you know where to feel for a light switch in a new dark room thanks to association and repetition. But it’s not always easy to get all that data. One way IBM gets it is by acquisitions of companies with large datasets, like Truven Health Analytics and the Weather Channel’s digital assets. Watson’s natural-language capabilities help the system understand the jargon of that field and process the data better. IBM then works with experts in various fields—oncologist, chefs, accountants—who “teach” Watson how they do their jobs.

But the “cognitive computing” technologies under the Watson umbrella aren’t as unique as they once were. “In the data-science community the sense is that whatever Watson can do, you can probably get as freeware somewhere, or possibly build yourself with your own knowledge,” Claudia Perlich told Gizmodo, a professor and data scientist who worked at IBM Watson Research Center from 2004 to 2010 (at the same time Watson was being built), before becoming the chief scientist at Dstillery, a data-driven marketing firm (a field that IBM is also involved with). She believes a good data-science expert can create Watson-like platforms “with notably less financial commitment.”

Perlich, who knew the team that built the original Watson, said she has a “truly deep respect for the technology” in that Watson computer. “I knew the components in the technology very well,” she said. “The reality however is what Watson as a commercial product represents is very different, even from a technology perspective, from Watson who won Jeopardy!

Ed Harbour, vice president of Implementation at IBM Watson believes Watson is still unique in its field. “Are there other companies out there that offered AI-based systems and machine learning? Yes, there are,” he said. “However...I believe very strongly Watson is ahead of the competition and we’ve got to continue to push [to make Watson better]. No, I don’t think it’s something that anybody can just do.”

But according to Perlich, data scientists who want to create similar platforms as Watson could possibly pull from various offerings from the likes of Microsoft Azure, Amazon Web Services, or Data Ninja. But what those products don’t offer is the Watson branding. “And everybody’s very happy to claim to work with Watson,” Perlich said. “So I think right now Watson is monetising primarily on the brand perception.”

IBM’s moonshot

IBM’s boldest promises have been about the ways it will impact health care. From the very beginning IBM was banking on the medical community accepting Watson. At a recent health care conference, Rometty said when she became CEO in 2012, soon after Watson won Jeopardy!, she decided using Watson to change parts of health care would “be our next moonshot.”

Watson Health is a separate unit developing and acquiring AI that can process massive amounts of health-related data—clinical studies to skin images to genome sequencing—and creating tools that will help medical professionals. There are two main branches within Watson Health—Watson for Oncology, which essentially suggests cancer treatments based on the patient information that a doctor inputs, and Watson for Genomics, which works similarly but based the genomic sequencing data of a patient’s tumor.

The dream is bespoke medicine and therapies. In the IBM Watson Experience Center in New York City, IBM demonstrated for me a simulated version of Watson that pulls information from millions of medical journals, and processes a patient’s personal information, symptoms, and gene sequencing and, then serves up a diagnoses and path to recovery.

“In the data-science community the sense is that whatever Watson can do, you can probably get as freeware somewhere, or possibly build yourself with your own knowledge.”

There’s no telling if or when we’ll see that version of Watson. But hospitals in China, India, South Korea, and Slovakia—as well as one in the US, Jupiter Medical Center in Jupiter, Florida—use Watson for Oncology for virtual second opinions on treatment options. Sujal Shah, an oncologist at Jupiter Medical Center, told Gizmodo he has been using Watson for Oncology for about five months—mostly during weekly cancer conferences where all the surgeons and cancer doctors discuss treatment options. He and his colleagues enter data into Watson so it can come up with a treatment plan. “Occasionally we disagree, and so that’s where Watson really comes into use,” Shah told Gizmodo, suggesting the computer can serve as a sort of tie-breaker.

In nations like India, where there is a low oncologist-to-patient ratio, doctors can use all the help they can get, and oncologist and chairman of Manipal Comprehensive Cancer Center told Gizmodo that Manipal Hospitals have benefited from Watson and his patients appreciate being able to see recommended treatment options on the screen.

Using technology to help doctors treat cancer is a noble effort—no one would disagree with that. AI can have a tremendous impact on health care and many other industries. But the technology doesn’t seem to be advanced enough to have a transformational impact just yet. And Watson technology isn’t especially unique. But IBM’s buzzy marketing is trying to make us believe it is leaps and bounds ahead of anything else.

Watson’s health problems

“IBM Watson is the Donald Trump of the AI industry—outlandish claims that aren’t backed by credible data,” said Oren Etzioni, CEO of the Allen Institute for AI and former computer science professor. “Everyone—journalists included—know[s] that the emperor has no clothes, but most are reluctant to say so.”

Etzioni, who helps research and develop new AI that is similar to some Watson APIs, said he respects the technology and people who work at Watson, “But their marketing and PR has run amok—to everyone’s detriment.”

Former employees who worked on Watson Health agree and think the way that IBM overhypes Watson for Oncology is especially detrimental. One former IBM Watson Health researcher and UX designer told Gizmodo of a time they shadowed an oncologist at a cancer center that has partnered with IBM to train Watson for Oncology. The designer claims they spoke with patients who had heard of Watson and asked when it could be used to help them with their disease. “That was actually pretty heartbreaking for me as a designer because I had seen what Watson for Oncology really is and I was very painfully aware of its limitations,” the designer said. “It felt very bad and it felt like there was real hope that that had been served by IBM marketing that could not be supported by the product I know.”

The designer thinks that false hope came from the Watson ads. For instance, one commercial depicts two doctors in a rural hospital that can do genomic analysis thanks to an intelligent black box that advises the doctors. In another commercial a soon-to-be seven-year-old talks to a fictional square about how she’s not sick anymore. After Watson reads her health data she asks if Watson is a doctor. “No, I help doctors identify cancer treatments.” Watson responds, as the copy on the screen reads: “IBM Watson is helping doctors outthink cancer, one patient at a time.”

“Outthink cancer” is deceptively vague. Rometty was even more vague in a 2015 Wall Street Journal interview. “We will change the face of health care,” Rometty told writer Monica Langley. “If you think solving cancer is cool, then we’re cool.”

The experience meeting the hopeful patient made the designer view the company in an entirely different light. “I would not put money on Watson helping patients on a grand scale,” the designer said. “IBM needs to be held accountable for the image that it’s producing of its successes compared to what they’re actually able to deliver, because at a certain point it becomes an ethical issue...You’re telling cancer patients that they should have a higher feeling of hope about their outcome and then under-delivering on that—to me, that’s just dirty.”

Another former employee who worked as a design researcher lead at Watson for Oncology also said they were uncomfortable with how commercials portrayed the platform. “You watch those commercials and you think it’s finding new ways to cure cancer,” the designer said. “Why confuse people and make them think it’s going to find something that a physician couldn’t possibly find?... Then you’ve moved into what strikes me as unethical territory when you’re potentially giving hope to people who should never have placed hope in that kind of a system because it’s not a magical box that does that stuff. It’s not a god.”

But IBM Chief Health Officer Kyu Rhee said that Watson commercials aren’t misleading. “I think those commercials are a representation of how Watson interfaces,” Rhee said, referencing the commercial with two rural-based doctors. “That, in my mind, is an accurate representation of people being able to use it.”

“There is no way to validate what we’re getting from IBM is accurate unless we test the real patients in an experiment.”

If the designers making the products think they’re being marketed in a confusing way, how do the medical professionals who are using them feel? IBM set up (and listened in on) interviews in which doctors spoke highly of working with Watson. But the medical professionals we reached out to separately were less enthusiastic. Babu Guda is a researcher who has used Watson for genomic analytics at the University of Nebraska Medical Center for two and a half years. After the center treats breast cancer patients, Guda identifies hundreds or thousands of mutations using his own tools, then feeds that information to IBM, which picks out some of the mutations that have drug treatments. Guda is happy with the speedy results. “But the unknown thing here is how good are the results,” Guda said. “There is no way to validate what we’re getting from IBM is accurate unless we test the real patients in an experiment.”

Guda said if IBM wants researchers to have more confidence in Watson, the company needs to get top research institutions to recruit thousands of patients, test results, and publish studies. “Otherwise its very difficult for researchers,” he said. “Without publications, we can’t trust anything.”

IBM said Guda was using a beta version of Watson for Genomics. The Augusta University Georgia Cancer Center recently began using a new version. Ravindra Kolhe, a lab director at the center told Gizmodo the speed that he gets data back from Watson has added value to his clinical reports.

Watson Health took a huge hit in February when MD Anderson, University of Texas’ cancer center, put its Watson partnership on hold. IBM announced the project in October 2013, stating that the goal was to “eradicate cancer.” MD Anderson paid IBM $39 million for the project. None of the doctors who used Watson would speak about working with the system, but a spokesperson told Gizmodo. “While a variety of approaches have been examined, a final approach using this technology [Watson] to benefit patients has not been determined at this time.”

A recent MIT Technology Review article providing a “reality check” for IBM’s AI ambitions assessed that the failed MD Anderson collaboration did not mean that Watson was technologically inadequate, but rather it “seemed to show IBM choking on its own hype about Watson.”

Regardless, Watson is making strides in health care research. Last month at the 2017 American Society of Clinical Oncology annual meeting Watson Health shared five studies showing that Watson has gotten better at providing the same treatment recommendations as doctors with certain cancer types. But according David Howard, professor of health policy and management at the Rollins School of Public Health at Emory University, these are “early-stage studies” and there is a big gap between those results and demonstrating a benefit for patients. Next, researchers would need to perform studies that show physician’s treatments changed after seeing Watson’s recommendations and that those changes resulted in improved patient survival rates.

“When we did Watson back in the day, AI was a four-letter word. Well, we got to call it something. If we call it AI no one will take us seriously.”

Howard said many doctors are still sceptical about Watson Health. But he thinks health professionals are hesitant to criticise the technology. “You don’t want to be that guy who is quoted saying this is never going to work and then it becomes the future,” Howard said. “It’s kind of a black box and I don’t fully understand what it’s capable of. Maybe it’ll be wonderful. But based on the criteria we use for currently assessing advances in health care, there doesn’t seem to be face validity or actual data behind it.”

Cognitive dissonance

To be fair, it is challenging to market AI and machine learning to a mainstream audience. It was especially challenging back when Watson was born in 2011 and AI was the stuff of sci-fi. That’s why IBM came up with their own description: “cognitive computing.”

Michael Karasick, vice president of cognitive computing at IBM research, has been with the company for 29 years. He discussed the origins of the term with Gizmodo. “When we did Watson back in the day, AI was a four-letter word. Well, we got to call it something. If we call it AI no one will take us seriously.”

Now, thanks largely to IBM, it is no longer a risk for tech companies to focus on AI. Rather, it is a risk to ignore it. But because IBM wanted consumers to take it seriously in the early days, the company came up with its own flashy, imprecise branding for the fantastic new technology. As other companies have started investing heavily in AI in a time when it’s safer to do so, IBM has stayed on the same course, and Watson is trapped in the same black box.


More AI Posts: