Defining “Value” – the Key to AI Success
I recently conducted a 3-day, remote “Data Monetization: Thinking Like a Data Scientist” workshop for a transportation agency in the Middle East. Doing this training remotely is a personal challenge as I miss the face-to-face interaction in ideating, validating, and prioritizing the business areas that can benefit from data and analytics. However, conducting the workshop remotely did provide some valuable learnings for me.
One learning was my “Thinking Like a Data Scientist” visual was outdated (Figure 1).
Figure 1: Original “Thinking Like a Data Scientist” (TLADS) Visual
Figure 1 portrayed the “Thinking Like a Data Scientist” (TLADS) process as a linear process, where you would complete one step and then cleanly move onto the next step. But in reality, the process is highly iterative where it is common for learnings from one step to impact an earlier step such as refining the KPIs against which the targeted business initiative’s progress and success will be measured. So, I created an updated “Thinking Like a Data Scientist” visual in Figure 2 to reflect the highly iterative nature of the TLADS process.
Figure 2: Updated “Thinking Like a Data Scientist” Visual
One other learning from the workshop was the need to spend more time thoroughly understanding and defining the “value” that the business initiative sought to create. And that’s where things get tricky. Too many organizations limit how they define and measure “value”. And defining a robust and diverse set of KPIs and metrics against which to measure business initiative progress and success becomes critically important as we apply AI to continuously-optimize the business initiative.
To understand the AI “value” quandary, one must first understand how an AI model works:
- The AI model interacts with its environment to gain feedback in order to continuously learn and adapt its performance (using backpropagation and stochastic gradient descent).
- AI model’s continuously learn and adapt process is guided by the AI Utility Function, which are the metrics and KPIs that define AI model progress and success.
- The AI model seeks to continuously make the “right” decisions, as framed by the AI Utility Function, as the AI model continuously interacts with its environment.
- In order to create an AI model that makes the “right” decisions, the AI Utility Function must be comprised of a holistic and robust definition of “value” including financial, economic, operational, customer, society, environmental, and maybe even spiritual.
Bottom-line: the AI model determines “right and wrong” actions based upon the definition of “value” as articulated in the AI Utility Function (Figure 3).
Figure 3: AI Rational Agent Makes the “Right” Decisions based on AI Utility Function
Consequently, we must invest the effort across a diverse set of stakeholders to thoroughly explore and validate a diverse set of metrics and KPIs against which the AI model is seeking to optimize. And that’s exactly one of the key objectives of the TLADS process.
If we want AI to work for us humans (versus us humans working for AI), then we must thoroughly define “value” before we start building our AI ML models. Consequently, I expanded the TLADS process to drive a more thorough exploration and definition of “value” (Figure 4).
Figure 4: How Does Your Organization Define “Value”
A more holistic suite of “value” dimensions that today’s organizations need to consider are represented in red in Figure 4. To support the ideation around the exploration of these expanded value dimensions, I updated TLADS Step #1 and Template #1 (Figure 5) to include:
- What is the targeted Business Initiative? A clear statement about what the business initiative is trying to accomplishment.
- What are the KPIs or metrics against which business initiative progress and success will be measured? There should be at least 6 – 8 KPIs and metrics against which the organization is measuring the progress and success of the targeted business initiative.
- What are the Ideal Outcomes from this business initiative? This is a “future visioning” exercise to envision what successful execution of the business initiative looks like.
- What are the Benefits from the business initiative from the value perspectives of financial, customer, product, operational, environmental, societal? There should be at least 6 to 8 benefits across the broader dimensions that define value.
- What are the Potential Impediments to successful execution of the business initiative? There should be at least 6 to 8 potential impediments across technology, data, skills, personnel, competitive, market, and organizational factors.
- What are the Ramifications of the Failure of this business initiative? This one is the most fun because it gives everyone a chance to envision and explore all the different ways where things can go wrong.
Note: capturing a robust set of KPIs, benefits and impediments should not be difficult if 1) you have a diverse group of stakeholders participating in the brainstorming process and 2) you ensure that everyone has an equal voice in the ideation process.
See Figure 5 for an updated Template #1 using my traditional Chipotle example, where the items in red are related to the expanded definition of “value” for Chipotle.
Figure 5: Updated Template 1 of the “Thinking Like a Data Scientist” methodology
By the way, how your organization defines “value” probably says more about your organization than whatever your charter and mission statement states. Or said another way:
You are what you measure, and you measure what you reward
Yea, you may say that your organization’s charter is such and such, but your organization’s charter is actually defined by the metrics and KPIs against which you measure (and reward) business success. Period.
Value definition is critical from an AI execution perspective. Unfortunately, in a rush to get to the fun part of the AI job and start playing with the AI algorithms, organizations sometimes shortchange the upfront work in thoroughly and holistically defining the value (metrics and KPIs) against which business initiative progress and success will be measured.
Organizations must be thoughtful and thorough in how it defines the values against which the operations of the business will be measured. Getting those “values” wrong can lead to unintended, biased, and disastrous consequences in your AI models (check out Terminators, VIKI, and ARIIA…your homework assignment for my next blog).