EPISODE #62: How Explainable AI Keeps Decision-Making Algorithms Understandable, Efficient, & Trustworthy

( Click here to listen to this podcast episode )

There’s a quote circulating on the internet that “trust takes years to build, seconds to break, and forever to repair”. Its author was almost certainly referring to human relationships, but it turns out this sentiment applies equally to AI-driven decision making. Complex machine-learning models feeding AI algorithms can take a long time to develop. However, if they result in people’s credit applications being erroneously rejected or medical conditions being misdiagnosed, confidence in that AI will evaporate, and possibly never be restored. That’s where Explainable AI comes in.

Explainable AI seeks to transform a “black box” model into a fishbowl, so that its outputs are understandable, explainable, and ultimately trustworthy. One leading proponent of Explainable AI is Krishna Gade, a software engineering veteran of Facebook, Pinterest, Twitter, and Microsoft who Co-founded Fiddler Labs, an Explainable AI company. Krishna joins us to discuss how Explainable AI can address unconscious or unintentional bias, expedite debugging of AI models, and accelerate acceptance of and trust in AI-driven decision making.

Guy Nadivi: Welcome everyone. My name is Guy Nadivi and I’m the host of intelligent Automation Radio. Our guest on today’s episode is Krishna Gade, Founder and CEO at Fiddler labs. Fiddler is on a self-described mission to make AI trustworthy for all enterprises via something called “explainable AI”. Krishna knows a thing or two about explainable AI. Having led the team that built Facebook’s explainability feature you’re all familiar with known as, ‘Why am I seeing this?’ located in your newsfeed. Prior to Facebook, Krishna held senior engineering leadership positions at Pinterest, Twitter, and Microsoft. His current venture Fiddler Labs counts the Amazon Alexa Fund and Lockheed Martin Ventures among its investors. Krishna, welcome to Intelligent Automation Rdio.

Krishna Gade: Thank you, Guy. Thanks for the kind introduction and looking forward to the podcast.

Guy Nadivi: Krishna, what led you to start Fiddler and how did you come up with that name?

Krishna Gade: That’s an interesting story. As you said, I was an engineering lead at Facebook, working on explainable AI for newsfeed, and that’s actually how I got introduced to the field. And after working through that problem at Facebook and seeing all the impact that it can have in creating transparency and trust around AI and machine learning technology there, I wanted to start an enterprise company that would help different businesses across the world that are adopting AI and machine learning and do it in a trustworthy manner. So, we were starting this company and the first feature that we built was this ability to fiddle with the models or fiddle with the inputs in the model and observe the changes in the outputs. And so we really loved that and we wanted something that is very easy to remember and fiddling with AI was something that we wanted people to be able to do to make it transparent to trust it and that’s how the name Fiddler came along.

Guy Nadivi: Okay. Now, fiddling with parameters is something that kind of is a good segue to the next question which I think is on a lot of people’s mind is what is the performance impact of explainable AI on an AI system?

Krishna Gade: Yeah, that is a great question. If you were to step back, what is AI and machine learning? What are these technologies doing? They’re essentially looking at the data and eliciting patterns inside data to construct what we call a model which can then be used to predict the future. So for example, let’s say I have a history of all the loans in my bank that people defaulted on and people paid correctly on. Using that data I can learn a machine learning model or an AI model that, with some high degree of confidence, can predict when a future loan applicant comes to me, how likely this person is going to default on the loan or how likely is this person going to pay back it? So that is what an AI system does. It uses the historical data set to predict the future. The problem though is that these AI systems, these models that are created today using whatever technology that people are trying out with an open source or vendor solutions, happen to be a black box, which means that a human, whether that’s a business user or a developer cannot open up the system and look at all the rules that it is using to do the predictions. And so explainable AI is a set of techniques that can make this black box AI system or the model transparent so that you can look into what’s going on. You can play with the system interactively and understand what is the reason the model is saying that this person is going to default on the loan? What will happen if this person’s profile was like this, for example, if he was requesting a lower loan amount, or if he had a 20 points higher FICO score or would the model still approve the loan or reject the loan? Those are the things that you can ask with explainable AI and get answers from so that when you start observing and understanding how the models work, you can start to learn to trust the system and start using it more and more in your business processes. So that is kind of the main impact of explainable AI. Obviously there are other benefits of actually being able to open up the black box — you can debug when things go wrong. So when you are running these models at scale in your environment, when they’re making these predictions you can go look into them. Once you start knowing how the model is performing and observing any image predictions you can then start correcting it by figuring out how to retrain it better and all. So there are a lot of other benefits of being able to understand how the model works. So actually ultimately make the overall AI system better over time.

Guy Nadivi: Okay. So let’s talk about some outcomes from explainable AI. Can you, Krishna, speak about some of the more interesting use cases Fiddler has applied explainable AI to and the results you achieved?

Krishna Gade: Absolutely. So as I mentioned AI and machine learning is this new technology spreading across the world, across the industries. So financial services, recruiting, e-commerce, advertising, different industries are using explainable machine learning and AI techniques today to improve business process efficiencies, increase revenue, to save time of people and whatnot. So one of the use cases that Fiddler has recently worked with is in recruiting. So we work with this company called hired.com, which uses machine learning and AI to process resumes. And so they are a marketplace between users who are trying to find jobs and recruiters who are trying to find relevant candidates. And so in that marketplace when I’m a candidate I’m trying to apply for a job, I can post my resume to hired.com. And hired.com’s AI matching engine will match me to a particular recruiter who might be looking for a candidate with my experience. So during this process, Hired was building very sophisticated, deep learning models, AI models to do this matching and what they felt was they did not have a good way to understand how these models are working so that if their internal users, their own developers who are building the models or business stakeholders ask a question like, “Why did we reject this candidate?” Or “Why was this candidate accepted into our pipeline?” Or “Why was this candidate matched to that other job or to this recruiter?” They would not get answers to those questions. And so by using Fiddler in their workflow today, they’re able to understand how their candidate curation is working? How is their system is matching the candidates to their recruiters? And then ensure various different things. To ensure things like there is no bias in their matching environment, they are treating candidates across different genders and ethnicities equitably and in fixing defects in their models, bugs in their models as they may come. And so that’s kind of how they’re using Fiddler and to improve trust in the system and also improve the efficiency and reduce the amount of time debugging the system that they had in the past when they would get these questions.

Guy Nadivi: So it’s a strange thing to ask, perhaps with regards to explainable AI, but is there a single metric, like ROI that best captures the impact of incorporating explainable AI into an AI project?

Krishna Gade: Yeah, that’s a great question. So here’s how I would think about it. So machine learning and AI is becoming a first class citizen in software stack in many companies. When you are actually using them in production and these machine learning models when they make predictions they can actually increase your business impact like for example, increase your revenue… Let’s say, you’re using the model for predicting which items the user will buy on your e-commerce site or you’re using a model to surface up which ad to show to the user or to predict whether a loan would be a good loan or not. Each of those cases it’s associated with your business metrics. If you make good decisions, then you will sell more items, you’ll issue more good loans and it’s a business value for the company. And that’s why they are implementing AI and ML. However, the problem as I mentioned with machine learning and AI is it’s a black box. It means that you cannot understand why it’s doing what it’s doing. And then another problem with that is its performance can vary based on the data that it is processing. So for example, this has actually happened in real life after coronavirus. So there’s a company called Instacart that would use machine learning for managing the inventory of their items. As everyone knows it’s in the grocery delivery market. And managing inventory is very important for them so that users when they come to them and they can find the relevant items. Now what happened is after Coronavirus, their model started performing worse. Why? Because as I mentioned at the beginning, an AI or a machine learning system learns from the historical data to predict the future. But when the data itself shifts so dramatically that your history cannot really predict the future. Then your machine learning and AI no longer very effective. And so what they’ve seen is, the models who were trained with this prehistoric pre-coronavirus date where the user behavior is normal, and then post Coronavirus, all of a sudden their models were not predicting with high accuracy. And so that was actually causing them problems with their users. And so it was dropping their business metrics. So those are the two aspects that machine learning models suffer with today. One is this black box nature, which makes it hard for us to debug it. Second is the stochastic nature or there’s variability in their performance because of… or even call it non-determinism where their performance can change over time or you can call it monitor drift. And so these are the two aspects where explainability can help with, where you can observe the model over a period of time, observe its performance and then debug what’s going on with the model so that you can actually surface up if you’re in a sort of any business metric issues or whatnot so that you can actually safe guard your business to course correct. That is one big use case for explainable AI. And the metric there is how effective it is to help you catch issues with respect to model? How can it save developers time in finding bugs? Can it reduce the amount of time that you’re spending to understand what’s going on with the model? Can it spread transparency with an organization? How many people are able to use this model effectively and trust the model? So, that’s one aspect. The second aspect obviously is the regulatory aspect. When you’re using AI and machine learning in regulated use cases like under credit, underwriting or recruiting, it’s important to make sure that some third party is going to… It needs to understand how your models are working. In those cases, explainability is a must have because otherwise you would not be compliant with respect to the regulations. So those are the two main sort of metrics. One is this business ROI, ensuring your models are performing correctly and reducing the time to troubleshoot. And the second is the compliance, meeting the compliance regulations in your industry with respect to AI.

Guy Nadivi: Concerns have cropped up from time to time about the misuse of AI and machine learning. And that’s led to speculation that legal and or political headwinds might slow adoption of these advanced technologies. Krishna, how will tools like Fiddler’s explainable, AI platform mitigate these concerns?

Krishna Gade: Yeah. So, as I mentioned, as machine learning and AI is spreading across industries, especially to high stakes use cases, which has touched human beings directly. Like, for example, if my mortgage is approved, my quality of life changes. If I can buy a better house, if I get a better job, then it affects my life directly. If I go to a doctor, if machine learning is being used for clinical diagnosis, it affects me directly. So there are these high stakes use cases where AI is being used today. And in those cases it’s very important that companies that are operationalizing AI, do it in a responsible manner or a compliant manner. And within certain industries where pretty well defined regulations exist for example, in financial services especially in banking and insurance there are very well defined regulations that any model needs to follow. And in those cases you need explainability to understand how the model is working, to ensure there is no bias in the model, to ensure you’re able to create a report whenever you need to know what’s going on with the model. And so what’s going on is similar regulations are now coming up for the AI industry in general. So for example, there is this bill sitting in Congress called Algorithmic Accountability Act which is essentially to govern and regulate AI across the board, across the use cases, and create some sort of accountability on businesses too, so that they’re using the AI properly. And so when those things become like actually… And operationalize those regulations, then tools like Fiddler become extremely important so that every company needs to explain their models, create those reports, and mitigate whenever they may get like a sort of a visit from a regulator and whatnot, they can actually show to them how their models are working and they are actually understanding it for themselves how their models are working.

Guy Nadivi: Harvard Business School published an article not long ago calling for the auditing of algorithms the same way companies are required to issue audited financial statements. Given that AI developers can incorporate their own biases into algorithms, even unintentionally or unconsciously, what do you think about the need for algorithm auditing in addition to a tool like Fiddler?

Krishna Gade: Mm-hmm (affirmative). Yeah, it’s a very good question. See, I think one of the things that we need to understand is how does this concept of bias get into a model and how do we audit this algorithm to ensure that it’s working properly? So if you think about it in the process of machine learning goes like this, you start with collecting some data, let’s take that credit card use case. If you go back like a year, year and a half ago, we had this famous new story called Apple Card gender bias. So where Apple started issuing credit cards on phone, and people were getting different credit limits set automatically, and what happened was within the same household, the husband and wife got 20x difference in terms of the credit limits that were being set by their algorithm. Now, when the card was obviously sponsored by or created by the Goldman Sachs team. And so when the customers complained to the Goldman Sachs customer support, the answer that they got was, “Oh, we don’t know. It’s just the algorithm.” And so, what could have happened there? We don’t know. There was a regulatory probe that went into Goldman Sachs. But what happens is when developers train these algorithms, they take this historical data and train a model. And when you do that, if you don’t validate the model, if you don’t audit that model before you launch it to your users, it can carry unintentional, unconscious bias that you yourself may not be aware of. And I think I believe that organizations do not create biased models knowingly. I believe that there’s still goodness in the society. I don’t think people do it… A lot of people do it for the sake of it. But it happens because they’re not paying attention to it. And so what do you need in those cases is a tool that can give you insight into how this algorithm is working. So that you can say, “Okay, we have this credit card algorithm, it’s going to set credit limits. How is it going to set credit limits across different genders, across different ethnicities? Have you found any sort of discrepancies here? Can you do like some sort of ‘What if’ analysis? Like what if you flip the gender of the person, would the algorithm predict the same way? Even though the gender may not be part of like the variable. So, if you change certain characteristics of the profile, how would the algorithm predict their credit limit?” So you need those type of tooling so that developers can, validate their models, audit their models thoroughly before they launch them to production. And when they find those defects, they can go back and fix them and figure out how they’re working. So, I think we are working on a tool that can help this algorithmic validation. There are obviously other tools, there are some open source tools that are also coming up as well. It’s a very important topic, especially when algorithms are being used for high stakes use cases.

Guy Nadivi: There’s a lot of excitement about AI for young people entering the workforce, and even for more established professionals thinking about a career change. what kinds of skills does Fiddler covet the most when hiring talent for explainable AI?

Krishna Gade: That’s a very, very good question. See, I think first up we are an extremely mission-driven company. Our mission at Fiddler is to build trust into AI, and that’s what brings the team together. That’s what brings the investors together. That’s what led to us to form the company. So what we look for for young people joining the workforce is firstly this passion towards this mission, passion towards explainability, passion towards transparency and building trust with AI and solving this problem. We believe that this is not just like, like a business problem, this is a societal problem. And that’s what excites us every day to come to work and spend our blood and sweat building this product. So the first thing is that passion. The second thing is obviously from the technical side and as we hire data scientists, we look for people who have either prior data scientist experience or some educational sort of knowledge around machine learning, data science. It doesn’t necessarily have to be like a PhD in computer science with machine learning. We also have hired people from different disciplines, from physics, from other disciplines in science and people who have entered data science from other scientific disciplines as well. And the third thing is obviously the technical sort of knowledge around ability to code and all of that. So from a technical point of view those are the three things that we look for. The passion for trust and explainability and machine learning and data science kind of skill set. And then obviously the technical skill set to actually be able to build these tools for real, like be able to write code and whatnot. And then from people that are looking for a career change, if you’re thinking of going into machine learning or data science from a different field, we actually have quite a few data scientists on our team who have done that. I would really encourage you to either sort of take up like a Coursera course or kind of look into… There are lots of these very good academies these days that teach data science or even do a small like a master’s program or a program that can teach data science in a university. That way you can learn all the subject matter and sort of prepare yourself to join this workforce. It’s a very exciting area. Lots of companies are hiring data scientists and machine learning as a whole is taking off. And so it’s a very, very exciting time for people to learn this field and join the workforce in this area in general.

Guy Nadivi: Overall Krishna, given your high level perspective, what makes you most optimistic about AI and machine learning?

Krishna Gade: Yeah, it’s a very, very good question. I think it’s a lead up to what I’ve just said, right? So I think what’s going on is if you kind of think about what has happened in the last two decades, data has just exploded. Companies, individuals, we are just in the age of where we are surrounded by data. Especially when it comes to enterprise, you’re collecting so much data these days from various different channels about your users, about your own business, about how customers are interacting with your products and services. So all of this data is now available and we have built this two decades worth of data infrastructure, data warehouse, and data platform infrastructure. Thanks to a lot of successful cloud companies like Amazon, Snowflake, and so many other data warehousing sort of tools that came out. And so we are sitting on this goldmine of data. Now how do you make sense of this data so that you can actually build actionable products? So how do you improve your credit decisioning system? How do you improve your e-commerce recommendation system? Or how do you improve your recruiting system that is processing resumes? How do you improve your clinical diagnosis? So there is an opportunity there to use all of this data to leverage AI and machine learning to make better decisions and to also make decisions at scale so that it’s not like there’s one person sitting and underwriting loans and kind of in a manual manner. So to be able to automate it at scale. So that’s the opportunity that every company is sitting on. However and the catch is that it’s important to not only do it at scale and we have a lot of tools to help you do it, but to do it right and to do it in a responsible manner so that you don’t lose your customer trust, so that you don’t hurt your business metrics accidentally, so that you don’t get into regulatory compliance issues or brand reputation issues. And so that’s where things like explainable AI and machine learning monitoring come into picture. And so being able to do this in a responsible manner, you can then not only leverage this amazing technology of AI to do great things for your customers and improve their experiences, but also do it right and in a sustainable manner for the future of your company. And so that’s what excites me in general. And that’s what I believe this decade is going to be the decade where AI will be sort of the first class citizen in enterprise. And pretty much every company will be deploying it at scale.

Guy Nadivi: Krishna for the CIO, CTOs and other IT executives listening in what is the one big must have piece of advice you’d like them to take away from our discussion with regards to deploying explainable AI at their organizations.

Krishna Gade: That’s a great question. I think a lot of CIOs and CTOs are being asked either by their board or their CEOs that, “Hey, there’s this whole AI and ML hype going on, what are we doing about it in our company? What is our strategy there?” Right? So I think the first step is to go and look into what business problems that you can apply this technology towards. And do you have data collected for those business problems? And if you have data collected, then do you have the teams in place to actually do this work. So this technology is not an easy technology, you have to sort of hire the relevant staff, you need to hire these data scientists and machine learning engineers to be able to build these models and sort of increase your business metrics over time. Now, once you have it then you need to think about what is the right toolkit? What is the right set of tools that you need to employ so that you’re doing this at scale, but also in a responsible manner? And that’s where tools like Fiddler explainable AI platform can come into picture. So when you are creating these machine learning models, how do you ensure that they’re working properly? How do you monitor them in production? How do you explain defects or problems when things go wrong with them? Having those tools will help these data science teams to move faster and do and make and fix their models faster to actually be able to deploy AI faster to production, to get that sort of business ROI much, much faster to their CIO. And we have heard a lot of times, I mean I was having a chat with the SVP of Analytics of a very large insurance company this week. And he was telling me that there are so many models that they create that don’t end up in production. Why? Because they don’t know which model to launch, they don’t know what’s a good model, whether this model will perform properly in production or not, they don’t have the good tools to measure and monitor the performance of the models, they don’t have good tools to understand how the model is working in a deeper manner. So by having this explainable AI platform like Fiddler, you can leverage these things so that you don’t have to just create these models and keep them on the shelf. You can actually operationalize them faster, you can get them to your customers, you can increase your ROIs. And so that’s kind of how I would think about it. First of all, the problem statement that you’re looking at, the staffing, that you have the right team in place and then that you have the right tools to actually deploy machine learning to your end users and build products that depend on machine learning for your customers.

Guy Nadivi: Alright. Well, looks like that’s all the time we have for on this episode of Intelligent Automation Radio. Krishna, I love bringing on guests who can shed insight on a topic that I’m trying to get educated on and that my audience will benefit from learning about as well. You’ve really raised our collective IQ on explainable AI today. And I thank you very much for coming on to the show.

Krishna Gade: Thank you so much, Guy. It was a pleasure and I loved the questions and it’s a very, very important topic from a business standpoint, from a societal standpoint, and from a technology standpoint, and we are very excited to be at the forefront of this new field. And yeah, we are super excited to share whatever knowledge and experiences that we have accumulated over a period of time to all the audience.

Guy Nadivi: Krishna Gade, founder and CEO at Fiddler Labs. Thank you for listening everyone and remember, don’t hesitate, automate.

KRISHNA GADE

Co-Founder and CEO of Fiddler

Krishna is the co-founder and CEO of Fiddler, an Explainable AI Monitoring company that helps address problems regarding bias, fairness, and transparency in AI. Prior to founding Fiddler, Gade led the team that built Facebook’s explainability feature ‘Why am I seeing this?’. He’s an entrepreneur with a technical background with experience creating scalable platforms and expertise in converting data into intelligence. Having held senior engineering leadership roles at Facebook, Pinterest, Twitter, and Microsoft, he’s seen the effects that bias has on AI and machine learning decision-making processes, and with Fiddler, his goal is to enable enterprises across the globe to solve this problem.

Krishna can be reached at:

LinkedIn: https://www.linkedin.com/company/fiddler-ai/

Twitter: https://twitter.com/fiddlerlabss OR https://twitter.com/krishnagade

Website: https://fiddler.ai