THINKING OUTSIDE THE BLACK BOX

9 August, 2018

In recent years, a great deal of attention—and a fair amount of hype—has surrounded the resurgence of neural networks, particularly those using a “deep learning” approach, in the field of artificial intelligence and machine learning.

Much of the core science involved techniques dating back decades, but which have only become practically applicable in the past few years due to the huge growth in computing power and the size of available data sets. At Logical Glue, we know neural networks can produce great results, and our platform incorporates a state of the art neural network implementation that delivers highly accurate models.

But, for all their power, neural networks are not a panacea, and not without their drawbacks. Perhaps the single biggest problem with neural networks is that they are a black box. In science and engineering terminology, a “black box” system is one whose inputs and outputs are accessible and understood, but whose inner works are either obscured or unintelligible. Black box systems are contrasted with “white box” systems, whose internals are visible and comprehensible to their users.

 

 

An example of a black box system in everyday life would be a door lock. The inputs to the system—a key, and human action such as pushing and turning—are obvious and well understood, as are the outputs—the lock unlocks or locks. However, the lock’s internal mechanism is hidden and unexplained. This does not preclude their general usefulness, but it is does cause problems in certain situations. For example, if I cannot open a lock, it is because I have the wrong key, or because the lock is faulty? How can I be certain the lock has not been tampered with? How can I know whether a particular lock is more secure than another?

An example of a white box system would be a crank-operated lottery ball machine, which are used to produce random numbers for games like bingo. Such machines are deliberately designed to be simple and transparent. The goal of the mechanism is not just to prevent cheating, but also to be visibly and obviously fair to anyone watching. This helps to boost confidence in the system, and empowers the audience to verify its security themselves, instead of having to rely on the guarantees of the operator.

In general, as systems become more complex and more locked down, there has been a tendency towards more black boxes and less white boxes. Cars and consumer electronic devices have seen a shift away from user-serviceable internals to highly integrated components that can only be prised open and serviced with specialist equipment.

The issue of transparency however, remains important for many businesses and sectors. The ability to understand the systems we use empowers us to be more than passive providers of input and consumers of output. In particular, understanding those systems that are critical to the outcome of our businesses can be the difference between success and failure.

In the field of applied machine learning, the inputs to our system are the feature values we supply to a model, and the output is the classification decision. A black box model is one for which the output, or decision, is not relatable in terms of the input in a comprehensible way. As mentioned earlier, neural networks are black box models. That is, even the best trained and most accurate model cannot explain why a decision was made in a way human beings can understand. Even with simple, single-hidden layer neural networks, it can be almost impossible to determine why an instance was classified in a particular way.

For some applications of machine learning, a black box model is not a problem, but for others it is a deal-breaker. For businesses that need to explain and justify decisions to regulators, simply presenting a result may not be enough. For businesses who wish to use knowledge acquired via machine learning to better understand their market or profile their customers, the reasons why decisions were made may be as important, or more important, than the decisions themselves.

Attempts to solve the black box problem of neural networks have focussed on algorithms that extract rules and decision trees from models after they have been built. At Logical Glue, we have a better solution. Our Fuzzy Logic models are white box by their nature, with human-understandable reasoning a fundamental part of how they produce decisions. Each Fuzzy Logic model consists of a set of learned rules that can be expressed in natural language as if-and-then sentences.

For example, a sample rule for a credit decisioning model might say that “if a customer is young and their income is high then they are likely creditworthy”, if this is a conclusion supported by the input data set. The “fuzziness” inherent in the algorithm means “young” age and “high” income are defined not as strict ranges, but fuzzy sets that instances can belong to a greater and lesser extent, mirroring the way humans categorise things. The output of any prediction includes the set of these rules that fired for the instance in question, each contributing to the overall decision. This allows the user to easily understand why the model made the decision that it did.

Having this kind of white box output changes the relationship between the predictive model and the user. Rather than being a mysterious oracle, producing decisions that users must take, almost on faith, as being accurate, the model is instead a source of reasoned advice. The user can use the decision, the reasoning behind it, or both, in order to guide their actions and their thinking about the problem. For example, in credit decisioning, the output of a Fuzzy Logic model can be used to assist underwriters, by applying a consistent baseline to all decisions. The model’s white box output can inform the underwriter, drawing their attention to the most salient facts about the application, and giving a recommendation. The underwriter is then able to apply their own experience to borderline cases, achieving nuanced decisions that combine the strengths of both human and machine reasoning.

As the use of machine learning in business continues to grow, demand for white box decisioning is likely to rise as well. Models that justify their decisions with human-understandable reasoning can overcome natural skepticism about their accuracy, and integrate into existing management processes more easily than black box models, where a prediction must simply be believed or not. Fuzzy Logic models, as provided by the Logical Glue platform, provide sophisticated white box output without sacrificing model accuracy, and are uniquely positioned to capitalise on this demand.