Machine Learning and the InnerEye for Cancer Treatment with Dr. Antonio Criminisi

Published February 21, 2018

Share this page

Antonio Criminisi – Principal Researcher

Episode 13, February 21, 2018

With all the sensational headlines about artificial intelligence, it’s reassuring to know that some of the world’s most brilliant minds are developing AI systems for entirely practical reasons. One of those minds belongs to Dr. Antonio Criminisi, a Principal Researcher at Microsoft Research in Cambridge, England. And one of those reasons is to help medical professionals provide better healthcare to their patients.

Today, Dr. Criminisi talks about Project InnerEye, an innovative machine learning tool that helps radiologists identify and analyze 3-D images of cancerous tumors. He also gives us some insight into his work on deep neural decision forests and tells us how gaming algorithms made their way into medical technology, moving from gamer to patient, and turning outside-in imaging… inside-out.

Related:

Microsoft Research Podcast: Visit our podcast page on Microsoft.com
iTunes: Subscribe and listen to new podcasts each week on iTunes
Email: Subscribe and listen by email
Android: Subscribe and listen on Android
Spotify: Listen on Spotify
RSS feed

Transcript

Antonio Criminisi: Look, our work is very, very practical. We want to develop technology to help oncologists, radiologists and, eventually surgeons, as well. That’s all. So, it is a productivity tool. They are not the type of AI that people say will take over the world at all. They are just very practical, concrete tools to help reduce cost and save time.

(Music plays)

Host: You’re listening to the Microsoft Research podcast. A show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

(Music plays)

Host: With all the sensational headlines about artificial intelligence, it’s reassuring to know that some of the world’s most brilliant minds are developing AI systems for entirely practical reasons. One of those minds belongs to Dr. Antonio Criminisi, a Principal Researcher at Microsoft Research in Cambridge, England. And one of those reasons is to help medical professionals provide better healthcare for their patients.

That and much more, on this episode of the Microsoft Research Podcast.

Host: Thanks Antonio, again for joining us from MSR Cambridge in the UK via Skype. It’s great to have you with us.

Antonio Criminisi: No problem. Thank you.

Host: So, you’re a principal researcher on InnerEye AI for cancer which uses machine learning algorithms to treat cancer. So, give us an overview of the work you are doing in computer vision and medical imaging analysis.

Antonio Criminisi: Sure, my pleasure. So, what we do in project InnerEye is, we apply state-of-the-art machine learning technology for the analysis of radiological images. In particular here, we’re talking about CT as in Computer Tomography, and MR as in Magnetic Resonance images. And we’re looking, specifically, at images of patients who have already been diagnosed with some form of cancer, unfortunately. And what the technology does is, analyzes those images, at a pixel-by-pixel level, to figure out exactly where the tumor is. But also, to do what’s called the delineation or contouring, of organs around the tumor. They are called organs-at-risk. And the reason why this is important is because, for instance, in radiation therapy, you need to instruct the machine that delivers the radiation, the therapy, to exactly where the target is, i.e., the tumor, but also the organs that need to be spared from nasty radiation. This is normally a process that is done manually, with somewhat archaic tools. And we can help, precisely in that area, to make the delineation, the contouring, and therefore, the radiotherapy planning, a lot quicker and also more cost-effective.

Host: So tell us how gaming technology algorithms are now working in medical technology. How did you go from gamer to patient and inside-out imaging to outside-in imaging?

Antonio Criminisi: Sure. So, our expertise is in machine learning. So, for a decade or more, we’ve been working on experimenting with new, better, more efficient, more accurate machine learning algorithms for doing predictions from images. And those are pretty much any type of images. It could be your holiday snaps, it could be videos, or it could be medical imaging. So, when we were working on the technology, we developed some algorithms which turned out to be both accurate and particularly efficient. And at that moment, we thought, “Hey, if these algorithms work on decked images from the outside of a person, in that case a player, perhaps they can also work on images where we are looking at the inside of a patient body in that case.” And that’s where the project, you know, was born, really.

Host: So, what are the unique challenges that radiologists and clinicians face that your work helps address? You mentioned delineation earlier. You also talked about quantification at one point. Can you talk about how your work has an impact on these two big ideas?

Antonio Criminisi: Yes, absolutely. So, there are many medical experts in modern hospitals who are faced with a number of issues, and normally they spend an enormous amount of time trying to tackle those issues. In particular we already mentioned the work of radiation oncologists where they need to delineate, with great accuracy, the tumor and the organs-at-risk so that they can deliver safe and effective, you know, therapy. In this case we’re talking about radiotherapy. On the other hand, there is radiologists who have got a very different role. In most cases, radiologists, they look at images of patients and they need to assess what they are looking at; the disease, not only the type of disease, but also whether the disease is progressing over time or it is responding to the treatment. And, unfortunately, nowadays, they do not have very good tools for doing the latter, this, you know, assessment and the quantification of the disease. There are no very good quantification tools where you can actually measure, say, the volume of the tumor from a radiological image. And that’s where we can help. That’s the idea. That potentially our technology can be embedded within a radiologist’s workflow and help translate those radiological images into measuring devices. That’s our goal, turning those images into measuring devices where the radiologists can actually write in the radiological report, “This is how big the tumor is today. This is how big it was last week. This is how big it was two weeks ago.” And so, they can then plot the path of progression of the disease with great accuracy and rigor.

Host: And that’s super important in treating cancer, is to see how aggressive tumors are, yeah?

Antonio Criminisi: Yeah, absolutely. That’s just one of the examples. Where, also, it could be used for instance to figure out which drug works best, right? So, if I’m trialing different drugs, I want to know which one is more effective, faster-acting, and so on. Those are just some of many possible examples.

Host: You know, it’s such a fascinating field that you’re working in and so important for so many people. I mean, I don’t know anyone who hasn’t been touched by cancer… As you are looking at an image, you say it’s important that people can tell the difference between the bad tissue and the good tissue. How does it do that?

Antonio Criminisi: Right. So, as all machine-learning algorithms – the algorithm needs to be optimized or trained. And so, what you do is, you collect a number of anonymized images which show the same type of cancer, solely tumor, for example. And you have experts delineating the tumor and delineating the different organs-at-risk around the tumor. And then you feed that to an algorithm who looks at a variety of patterns within the image: the intensity or the brightness of the pixels, color, if you have color images. But more importantly, the texture around them. And also, what we call semantic context, i.e., a pixel in the heart is defined as such, not just because of the way it looks. Because that wouldn’t be sufficient. There is a lot of pixels in the human body; images of the human body that look alike. And so, it’s much more important to look around and see whether that pixel resides in-between two very dark regions which normally represent the lungs. Right? So, if I know that I can see the lungs, the left lung and the right lung, then I know in-between those there should be, you know, the heart. You know, I expect the heart to be there of course. So, other structures like the spine and so on. But that gives you a little hint of how these techniques work and what they do. They look, not just to the pixels and features extracted from the pixels or voxels, but also, they look around to see whether there are other patterns that reinforce – the reference voxel should be the heart or the pelvis or the prostate and so on.

Host: So, you have to – your algorithms have to train on good tissue as well as bad tissue so that it knows the difference…

Antonio Criminisi: That’s right. They have to look at the whole image really to make sure that they identified the correct region and they classify the correct region as such.

Host: That’s fascinating because when you are being treated for cancer, I would imagine, you know, “Please don’t wreck the other stuff.” I mean, that’s what people are looking for is the magic bullet to only kill cancer and not destroy everything else about your body, right?

Antonio Criminisi: That’s right, absolutely.

Host: When we talked about what machines are good at and what humans are good at, how does this particular machine-learning technique augment what humans are already doing in radiology and cancer treatment?

Antonio Criminisi: Yes, that’s a very, very good question. So, we’re very proud of the fact that we’re designing the technology around medical experts. We are working with a number of medical experts who are giving us a lot of instructions and guidelines. And so, for instance, through this process, we have learned very early on that doctors are extremely good, in most cases, at the task of diagnosis, which means looking at, you know, radiological images and figuring out what is wrong with their patient. So very often, in most cases, they can look at an MR image of a patient’s brain and very quickly say, “That looks like a glioblastoma,” or another type of brain tumor for instance. And that is a very quick process. Again, there are edge cases, not everything is so easy. But for the most part, that’s quick. But what we have discovered, through working with many clinicians, is that measuring tools, that’s the problem. That’s what they do not have. And so, our technology augments their skills, or amplifies their skills by providing expert doctors or radiologists, in this case, with measuring tools, something that they desperately need and they do not have right now.

(music plays)

Host: Let’s switch over to your concept of decision forests. Most of us have heard of decision trees, but decision forests is fascinating to me, particularly with your novel approach to machine-learning, what you called Deep Neural Decision Forests or DNDFs. Explain the difference between the forest and the trees… and what’s unique about DNDFs.

Antonio Criminisi: Sure. I should also set all of this in the background of, you know, the most modern wave of machine-learning which goes under the name of “deep learning.” The whole world is talking about deep learning, and in particular they are talking about convolutional neural networks as a very effective and accurate technology. We know, and we’re very well-versed, both with convolution neural networks, but also with decision trees and decision forests. And we have explored advantages and disadvantages of both techniques. And we believe that we are onto a new set of techniques which we call deep forests, where we manage to marry the benefits of both worlds. And in fact, it looks like, from a theoretical and algorithmic point-of-view, those two worlds are really two ends of a continuous spectrum. They are not so different from one another. But to go back to your question, a decision forest is a collection of decision trees, in practice, where those decision trees are all slightly different from one another, and the advantage of using a collection of trees translates into better generalization, which is this issue of, “Okay I’ve got a machine learning algorithm that works very well on the training data, but what guarantees do I have that they would work equally well on previously unseen data, what goes under the name of testing data?” And so, the use of un-sampled techniques, i.e., a decision forest, gives us a little bit more guarantees in that sense.

Host: How would you frame the work that you’re doing? What specific targets are you aiming at with what you are doing?

Antonio Criminisi: Yeah, good question. Look, our work is very, very practical, okay? So, we want to develop technology to help oncologists, radiologists and eventually surgeons, as well. That’s all. So, it is a productivity tool. Like, Microsoft is particularly good at delivering productivity tools. They are not the type of AI that people say will take over the world at all. They are just very practical, concrete tools to help reduce cost and save time.

Host: Yeah, and that’s one of the – you know, there’s a lot of scary headlines out there about AI taking over the world or at least getting us all fired. And so, as you frame this as a tool to help radiologists in what they are already good at, and augment them, I hear that over and over at MSR, this “augment versus replace.” I find that fascinating with what you are doing with InnerEye.

Antonio Criminisi: Yes, thank you.

Host: What other broader potential might this tool have? Is it really focused just on cancer and radiology and that kind of thing, or do you see applications in other areas of medical technology as well?

Antonio Criminisi: We, in our team, are focusing on image analytics, and so any clinical workflow that can potentially use images of different types would, in theory, benefit from this technology. And so, you could think of pathological images, you could think of malaria, which is hematology images. You could think of 2-D x-rays. You could think of, you know, higher dimensional images. There are many, many, many options. Obviously in our team, we want to be concrete and deliver your value, so we are starting small, and radiotherapy area is our target initial domain really.

Host: So, as you are working in this area and that’s your goal, do you have any kind of corpus of evidence or data that this kind of machine-learning technologies are actually working and helping the radiologists and cancer treatment professionals?

Antonio Criminisi: Yes. We are gathering that evidence as we speak. And so through our many collaborators throughout the world, we are working with many different hospitals in many continents to make sure the technology that we are building together works, you know, across different countries, not just, you know, in the UK, not just in the US, but you know, as much as possible for everybody. And, we are starting to get evidence through our partners that the technology is starting to get really good, and we keep partnering with them to make it even better.

Host: Talk about the accessibility of your technology. How does a medical professional get access to it, how do they use it?

Antonio Criminisi: So, we are working with partners who goes often under the name of ISVs. And so, we are going to deploy our technologies through third-party software providers. So, there are many corporations, many companies who are very, very good at building what’s called medical devices, software-only medical devices, in many cases. And so, they are the people, they are the companies who then sell on those devices to healthcare providers. What we do, is work with those software providers and we provide them with our own state-of-the-art AI machine-learning technology to make those products better for their end customers.

Host: So, they incorporate what you’ve done into their products and then pass that on to them.

Antonio Criminisi: Absolutely.

Host: Okay, that’s interesting.

Antonio Criminisi: Absolutely. So, the technology we are developing will be exposed as a set of Azure services. From a medical point-of-view, you can think of them as medical components which then get incorporated into a third party, end-to-end product. And that way, we can make our partners better.

Host: So, as we talk about work in the medical field, there’s been discussion about the delicate balance between progress and privacy. And it’s particularly acute when you are dealing with sensitive health data. Are there any challenges you face with this technology in light of legislative or policy safeguards?

Antonio Criminisi: Absolutely. So, you know, we are dealing with very sensitive patient information here. We are talking about radiological images. And so, we need to be extremely careful in the way we handle them. And at Microsoft, we are incredibly, you know, aware of all the issues to do with patient privacy, anonymity and so on. And so, we make sure that we comply with all regulations, but we go beyond there. We are super-transparent with what we do with those patient images. As an example, our algorithms have been designed to need only the pixels, nothing else. In order to train algorithms and to optimize them, for them to deliver value, all we need is the pixel information. We don’t need any patient-related information, or any information related to the hospital of origin. And that’s a big advantage for us. You know, the algorithms have been designed precisely to be as strict and rigorous as possible in terms of preserving the patient’s privacy.

Host: Are you running into any of the same issues with InnerEye, with the GDPR regulations?

Antonio Criminisi: We are very well aware of the GDPR and for InnerEye, we are already compliant with GDPR. We know exactly what the regulation requires and does not require. As a quick example, again, all the data that is ingested by our training algorithms is completely anonymous. It is impossible to go from the pixels that we have to the identity of the patient even if there was an attack and someone maliciously wanted to record the ID of the patient, it wouldn’t be possible to do so.

Host: That’s awesome. You refer to pixels and I saw a term called Voxels.

Antonio Criminisi: That’s right.

Host: Tell the difference between those.

Antonio Criminisi: A voxel is a 3-D pixel. That’s all. Radiological images often come in a 3-dimensional format. Think of a 3-dimensional grid in space, and so each element in that grid rather than being called a pixel is called a voxel. That’s the only difference.

(music plays)

Host: So, if I’m a person that’s interested in medical science, I might not consider computer science as a way to get to my career goal, but this feels like it’s kind of a crossover between the two. Can you explain how maybe somebody interested in working on diseases and helping in that area would find computer science as a good path to that?

Antonio Criminisi: Yeah, so as you know computer science is everywhere nowadays, right? There aren’t many fields which haven’t been touched by computer science. And the same applies to medicine, and in particular to radiology. And I see more and more radiologists being extremely savvy about computer technology, being able to write code and program themselves into a little bit or maybe a lot of image analysis, themselves. So, this is really refreshing to see because obviously the more crosstalk there is between pure computer scientists and say pure oncologists, the better for both worlds.

Host: So, what was your path to medical image analysis research, Antonio?

Antonio Criminisi: I’m an engineer and I have always been passionate about images. Therefore, I ended up doing a PhD in computer vision, which is everything to do with algorithms for analysis of any type of images. And then I became passionate about applying those techniques to radiological images because I clearly saw an immediate benefit there for patients. Then, you know, within Microsoft, I was fortunate enough to be allowed to start looking to that space a little bit more deeply and work with radiologists and hospitals across the globe. And I found it extremely fascinating and refreshing and inspirational as well.

Host: How did you end up at Microsoft Research in the UK?

Antonio Criminisi: I did a PhD in Oxford in the UK; then I was hired into Microsoft straight after that. So, a very simple path.

Host: Pretty straight. You know, one of the things I’m hearing from researchers all across the organization is that a lot of what they are doing is very interdisciplinary and they are working with people that aren’t necessarily all from the same field they came from and that there’s a lot of wonderful cross-pollination. Are you finding that in your work as well?

Antonio Criminisi: Absolutely. You know, the cross-disciplinarity is, you know, one of the biggest things, you know, the best things you can do for innovation, really, and it’s not just me who says that. It is incredibly rewarding also to be able to learn new things from people who don’t necessarily speak exactly the same language as you or don’t do exactly the same things as you do. It’s, you know, it’s a great growing experience, learning experience, and at the same time, this sort of cross-disciplinary interaction has got a lot of, you know, provides a lot of impetus into innovation really.

Host: As a researcher, part of your life is just discovery and asking questions and then digging in and finding what you find. And sometimes you get one topic as a life work and other times it’s like, “Okay here’s another thing that I’d like to chase after.” Do you have any other projects, or research interests, on the go right now?

Antonio Criminisi: I have way too many to share. But at the moment I would like to just be concrete and deliver something of great value on this project. So that’s why for the last couple of years or so, I’ve been focusing only, and entirely, on this project.

Host: Yeah. You’ve got a bunch in your brain that once you deliver you can move onto.

Antonio Criminisi: That’s right.

Host: It’s like the writer who has a hundred stories they want to tell, but…

Antonio Criminisi: Yeah, one at a time.

Host: Yeah. It’s interesting to me that your framework is super-practical and that isn’t always the case, which is great. You want researchers to be looking at things from a variety of angles. But I imagine that radiologists appreciate the singular focus of what you are doing to make their lives better.

Antonio Criminisi: Yeah, that’s right. I think it’s very, very important. If you want to deliver something concrete, and really, you’ve got to focus. And that’s difficult to do believe it or not because you have to learn to say no, as well as yes. We get flooded with requests, all the time, from many different doctors, many different hospitals or organizations saying, “Hey, I’ve read about InnerEye. You are doing great work. Can that technology be applied to problem X and Y?” And, more often than not, the answer is, “Yes in theory, it could be applied in those other domains, but I cannot do it! I have to say no, because you know, the resources, of course, are limited and time is limited.”

Host: Yeah. “Sorry, I’d like to say yes, but I have to say no.” Oh, gosh. So, any thoughts as we close our Skype session and you go off to happy hour and I get a second cup of coffee here in Seattle?

Antonio Criminisi: No, I’m happy!

Host: Antonio, thank you so much for taking time out of your day to join us on the podcast. And it’s great talking to you.

Antonio Criminisi: Thank you very much.

Host: To learn more about Dr. Antonio Criminisi, and how machine learning technologies are helping medical professionals provide better healthcare, visit Microsoft.com/research.