How the machine ‘thinks’: Understanding opacity in machine learning algorithms

Text Exploration

In this article, the author identifies that algorithms may lead to unequitable and unpreferred social outcomes, but these algorithms are opaque: the opacity of how they work makes it difficult to understand if they are making discriminatory classifications. It’s argued that one form of opacity in particular—the use of machine learning—makes it nearly impossible for humans to detect this form of discrimination. The author concludes that managing problems of opacity will require a variety of approaches, rather than simply auditing or regulating an algorithm creator.

The author begins by showing how algorithms show up in popular media; negative implications of the use of algorithms have been increasingly discussed, but in an overly simplistic manner. Opacity limits the ability to really understand what is happening when a corporation makes decisions that impact people based on algorithmic output, and the author explains that there are three primary ways this opacity is introduced or manifested in software development.

Opacity is sometimes used intentionally to hide proprietary code and intellectual property. This may be because that code is a differentiator in the market, or it may be because knowing how the code works would allow people to “game” how a given product works. Search engine optimization is a field that works to understand the algorithmic rules at play in Google’s search rankings so a website author can position their page earlier in the search results; the more transparency an author has, the more they can manipulate that positioning. One response to purposeful opacity is to audit the proprietary code, but the author argues that corporations would likely be unwilling to expose their code, and the actual audit might be far too complicated to work in practice.

Opacity is also due to a lack of technical knowledge by those who might read algorithmic code with an intent to find bias. The public may have a hard time understanding the code, and some who author it may be less skilled in writing it in a clear, readable manner.

Opacity also results due to machine-learning, where a human—even one who wrote the code—may not be able to understand what exactly is happening when it runs. This is the focus of the remainder of the article. The author provides a brief primer on how machine learning works, and illustrates this by means of a handwriting algorithm; simply looking at how the handwriting is recognized does not provide any useful information visually. A more complex example of spam filtering is used to show how weighted words are used to indicate a spam message, but because no one word leads to the resulting decision, there’s no real way to make sense of causality: signals are used instead of deterministic models, and so a human can’t work backwards from an output to a set of clear causes.

This ability (or lack thereof) to understand classification in action is called interpretability, and there have been attempts to show the interpretability alongside a decision to a consumer or end-user for a given algorithmic classification. But the explanation of interpretability is often overly simplistic because of the complexity of the rules, and so systems resort to broad and easy to understand, but less valuable, statements.

The author concludes by offering a recommendation on how to best manage the opacity of algorithms, which is primarily to establish partnership between various parties involved. These partnerships “between legal scholars, social scientists, domain experts, along with computer scientists may chip away at the challenging questions of fairness” and “user populations and the general public can give voice to exclusions.” Additionally, “some combination of regulations or audits… the use of alternatives that are more transparent (i.e. open source), [and] education” will be necessary.