BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Defusing The Perils Of Enterprise AI

Forbes Technology Council
POST WRITTEN BY
Keith Strier

In the months following the failed Apollo 13 mission, investigators discovered that a seemingly benign event two years earlier was the root cause of this near national disaster. Engineers handling one of two oxygen tanks built for the service module accidentally let one slip and fall. The total distance of this fall: two inches. I once dropped my iPhone from my seat at a hockey game and watched helplessly as it fell 15 feet toward the cement floor. Miraculously, it landed at just the right angle and survived. Apollo 13 wasn’t so lucky.

In a fateful moment years before launch, at the North American Aviation plant in Downey, California, a simple slip of just two inches created enough structural damage to set in motion a series of failures that nearly killed three astronauts. An article by Jim Banke on Space.com summarized the situation pointedly: “No one knew it, but when Apollo 13 lifted off, it carried the makings of a small bomb inside its service module.”

As enterprises race to unlock the potential benefits of artificial intelligence (AI), they are focused on vetting use-cases and quickly moving to scale, but often with limited enthusiasm for the unique risks associated with such systems. Potentially, enterprises are blasting off with the makings of a small bomb inside their AI program.

The performance of traditional enterprise software, once implemented, is typically measured by a simple question: “Is the system up?” Having stable and secure access is the primary metric, and governance processes and controls have evolved over the decades to manage the known risks to this state of performance. The very nature of AI, however, renders traditional IT operating and risk models less relevant, if not dangerously out of touch. Unlike traditional software, AI is not implemented but applied, and at every stage of the data science process, novel risks emerge about which we, as an industry, have limited knowledge.

Championing an AI initiative without a fluency in the risks of AI is, arguably, akin to handing car keys to an 11-year-old and suggesting a joyride. The latter sounds patently absurd after all -- who would be that irresponsible? -- and yet this is the current state of enterprise AI. There are two issues leading to this heightened risk:

First, the “operate” model for traditional IT systems does not fit here, but it is still commonly misapplied. As referenced above, AI must be dynamically monitored, trained and retrained as part of an iterative journey, while software is procured, installed and maintained as part of a predictable program with clear boundaries. This difference has implications for governance, controls, resources, key performace indicators (KPIs) and the continuous operations needed.

Second, trust must be purposefully engineered into AI. It is not something to be assessed “post go-live.” More than just the data model, the entire system in which intelligence is embedded must be ethically aligned. While AI can emulate human-like cognitive abilities, such as learning through experience, it may also become corrupted, acquire human biases and amplify them at enormous scale.

On this last point, consider the tragic death in March 2018 in Tempe, Arizona, when an autonomous vehicle struck and killed a bystander crossing the street at night. The company that manufactures the Light Detection and Ranging (LIDAR) sensor on that vehicle quickly (and understandably) released a statement noting its product did its job. The failure point, in their view, was unrelated to the performance of the sensor. The key takeaway here is not to pass judgment, but to observe the importance of designing trust across the entire system. The primary KPI for AI performance is not “uptime,” but rather it involves a complex byproduct of mathematics, hardware, software, APIs, integrations and humans in the loop that may appear to be working, until a tragedy occurs.

Enterprises adopting AI are bound to have their own “crashes” -- HR systems that ignore qualified candidates for unlawful reasons, finance systems that learn the wrong way to detect fraud, procurement systems that fail to predict supply disruptions, etc. — but steps can be taken to defuse these risks. As a baseline, it is important to institutionalize a holistic approach to the design of automated systems a) emphasizing the need to leverage the full spectrum of robotic, intelligent and autonomous capabilities when automating a task; b) encouraging managers to blend machines and humans to optimize outcomes based on each entity’s natural strengths and weaknesses; and c) persistently scanning to stay current and mindful of the critical new risks that could jeopardize benefits realization.

To be more prescriptive, I recommend that executive teams become more informed about the many dimensions of trusted AI, starting with the following four criteria:

• Ethics: Is this use-case consistent with your corporate values? This analysis would encompass the design and behavior of an automated system, whether it’s a physical machine or virtual agent. Considerations may include moral implications, as well as the potential for issues of fairness and bias.

• Social responsibility: Does this use-case have implications at the macro level? This would include implications for the financial, physical and mental well-being of humans in the broader economy, as well as in your workplace, including suppliers and contractors.

• Accountability: Do we know how this intelligent system would operate? This pertains to legal liability as well as regulatory compliance. Considerations include chain of ownership, control, transparency and explainability.

• Reliability: Can it be trusted to perform as intended? What is the optimal definition of performance, in this context? This pertains to the need not only for testing but a commitment to testing innovation, developing novel methods that map to the complexity of the systems they are intended to measure. Considerations include operational thresholds, curative methods, secure coding, back-out procedures and kill-switches.

In sum, enterprises will need to upgrade IT operating and risk models to accommodate the nuances and emerging state of data science, shifting the focus from managing risk to sustaining trust. The path to sustained value is pragmatic enthusiasm, evangelizing for AI while respecting the minefield of new risks that could unhinge this critical lever for competitive advantage.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?