From an AI Algorithm to a FAIR and Robust AI Algorithm
It’s a long challenging journey from an AI algorithm to an AI product. These days there is a lot of focus on removing the bias of many AI algorithms, which is absolutely critical, but just one of the many things that must be done to go from an AI algorithm to an AI product.
Biased Algorithms
I spent over a decade building and deploying machine learning predictive models in regulated industries, such as financial services and insurance, and the assumption was that your models tended to be biased and you worked hard not only to identity and remove any biases, but also in preparing a case to regulators showing that your models were fair. Today, with all the advances in AI, we are in a different place. We have plenty of data and machine learning frameworks that make it much easier to build models. We now build models over large, obviously biased datasets, don’t take the time to look for biases, and then act surprised when third parties examine our models and find biases.
This reminds me a bit of the gambling scene in the film Casablanca:
Rick: How can you close me up? On what grounds?
Captain Renault: I’m shocked, shocked to find that gambling is going on in here!
[a croupier hands Renault a pile of money]
Croupier: Your winnings, sir.
Captain Renault: [sotto voce] Oh, thank you very much.
Captain Renault: [aloud] Everybody out at once!
Source: Dialog from the film Casablanca released on January 23, 1943 and directed by Michael Curtiz [1].
These days we are shocked to find biases in our models when we build models over large datasets, in which it is very challenging to remove the biases. In practice, almost all “found” data is biased and, in general, it requires a careful experimental design to generate data that is not biased.
There is a growing understanding of the importance of removing biases from machine learning models before deploying them in products, but much less understanding of all the work required to go from an AI model or algorithm to an AI product, which is the subject of this post.
Let’s assume that you have worked and trained an algorithm so that it is not biased. Note that a lot of this work is identifying the biases that you need to check and then accumulating enough data so that they can be carefully checked.
Robust Algorithms
Although it is not mentioned as frequently, it is also important to make sure that the algorithm is robust in that no matter what data is provided to the model, it continues to perform and gracefully detects and responds appropriately to any inputs. Often fuzzing is used for this purpose. In practice, deployed algorithms in the wild, get all sort of inputs that were never expected by developers. Also, robust models have the property that small changes to the input data result in small changes to the output data (more precisely that the map from inputs to outputs is smooth). Of course, many classification models don’t have this property, which is one of the reasons that, in practice, robust classification models are often implemented in two or three stages. First, continuous scores are produced that can have the property that small changes to the inputs result in small changes to the outputs. Second, the scores are mapped to normalize them and to account for drift over time. Third, the remapped scores are thresholded to produce categories. Of course, categories can be produced in this way using multiple scores. Another technique when the outputs of models are based on a temporal sequence of inputs (sometimes called event based scoring), is to exponentially dampen (or smooth) the inputs so that the resulting scores are smoother.
An Example: Explainability for AI Devices
There are a number of deep learning algorithms that have been developed that will take a hematoxylin and eosin stain (H&E) stained histology image and output a diagnosis or important feature of the image. There are also algorithms that will provide an explanation for the output of the deep learning model. Let’s consider the work required to transition from the deep learning algorithm and associated explainability algorithm for a medical image to a FDA-approved product.
As Figure 1 shows, it is useful to think of an algorithm as deployed in a system, and as a system as the main component of a product. The system and the product will of course be different depending upon how they are used and who uses them. For example, if the product is used by a pathologist, then it must be integrated into the pathologist’s workflow. If it is used to QA work by pathologists, then the workflow will be different. For example, the workflow might be used for an AI-based “second opinion,” followed by sending out any cases in which the two opinions are different to get a third human-based “third opinion.” If it is used to assign work to pathologists with different skillsets and expertise, the product will work another way. Finally, if it is used to replace pathologists, the workflow will be different again.
A deployable version of the algorithms must be developed that is part of a system that is integrated with the inputs the algorithms, the outputs the algorithms produce, and the actions taken by the outputs that are part of the workflow.
Before the product can be marketed it must either get approval as a FDA medical device or what is becoming known as Software as a Medical Device (SaMD), that is as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device [2].
From AI Models to AI Systems
Perhaps the most basic distinction is between building a model and performing an action with a system. When building a model, an important relevant question is: “Is the model valid?” When developing a system, the analogous question is: “Is the system usable?”
- Does the system perform the task it was designed to perform?
- Is the system usable by the person who actually uses the system?
- Is the system well integrated into all the required upstream and downstream systems?
- Does the system integrate all the rules required from a business, compliance and regulatory viewpoint?
From AI Systems to AI Products
Once you have a system, the next question is whether it is an AI product that can be sold effectively?
- Is the product complete?
- Is the product competitive?
- Does the product provide a large enough business value so that it’s deployed, integrated and used by those that buy it? A good rule of thumb is for a product that is replacing a current product requires a new product with “10x” improvement along some axis.
Perhaps the most important question to ask is whether the product is complete? A complete product solves a business problem from end-to-end, while a system simply performs a function. For example, a fraud algorithm produces a fraud score. A fraud system uses the score to take an action, such as approve the transaction, deny the transaction, or ask for more information. A fraud product also includes a case management system to manage contacting the customer, gathering more information and making a decision. A fraud product also includes substantial reporting to understand and manage the fraud present.
Moving from an AI system to a complete product takes time and requires a process. I discuss how to use a continuous improvement process for this purpose in my post Continuous Improvement, Innovation and Disruption in AI.
From AI Products to AI Business
Once you have an AI product, the next question is: can it be turned into an AI business that can grow and is sustainable?
- Is there an effective sales model to sell the product?
- How scalable is the sales model?
- Is the business sufficiently profitable to be sustainable?
- Is the market big enough for the business to grow?
- Is the market position defensible to keep current and emergent competitors out?
The journey from an AI product to a sustainable AI business can take years and it may be helpful to consider using some of the lessons of a lean analytic start-up.
References
[1] IMDb, Quotes from the film Casablanca, retrieved from https://www.imdb.com/title/tt0034583/quotes on February 10, 2021
[2] Benjamens S, Dhunnoo P, Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ digital medicine. 2020 Sep 11;3(1):1-8. https://doi.org/10.1038/s41746-020-00324-0