analytic governance

Five Steps to Improve the Analytic Maturity of Your Company – 2021 Edition

December 14, 2020 by Robert Grossman

Half of AI Projects Fail

A good rule of thumb is that about half of all AI and analytic projects fail to bring business value. Here are some recent articles that remind us of this:

Gil Press writing in Forbes [Press2019] summarized some of the statistics around the failure rate of AI projects. One of the most relevant ones was:”25% of organizations worldwide that are already using AI solutions report up to 50% failure rate; lack of skilled staff and unrealistic expectations were identified as the top reasons for failure [Press2019].”
John McCormick writing a column for the Wall Street Journal discusses a Gartner report: “A just-released Gartner Inc. report found that in the last two years, companies with artificial-intelligence experience moved just 53% of their AI proof of concepts into production. A similar survey in 2018 by the research and advisory firm showed that only 47% of AI proof of concepts were fully deployed [McCormick2020].”

Companies that consistently and repeatedly build models, deploy models and extract business value from models generally use processes that share some common characteristics. The role of an analytic maturity model is to identify these processes as a first step to improving them.

Two Dimensions of Analytic Maturity

I have been involved in assessing the analytic maturity of organizations for about twenty years. Based upon this experience, in an (open access) article [Grossman2017], I introduced a framework (call it AMM-17) for evaluating the analytic maturity of an organization that was based on the software maturity capability model [Paulk93].

I have found this framework quite useful and it captures many of the important characteristics of developing analytic and AI models, but it doesn’t address an important difference between developing software (the subject of the capability maturity model) and developing analytic and AI models. In general, for most organizations, software is developed by a single department or division, whereas it is important that analytics be used throughout an organization, wherever that it can add value.

In this post, I’ll remind readers of the AMM-17 model (for more information, see the post “Improving the Analytic Maturity Level of Your Company”). The first four levels (Analytic Maturity Levels D1 – D4) can be best thought of as applying to a single department, center or business unit developing analytic models. In the section that follows, I’ll introduce a second dimension of evaluating the analytic maturity as analytics is replicated throughout an organization (Analytic Maturity Levels E1 – E4).

The Maturity Level of a Project Team, Department or Center to Build and Deploy a Model

The first dimension is the ability of a project, department or analytic center of excellent to build and deploy and analytic model. There are four essential levels of analytic maturity for a department or similar unit, which we call D1 – D4.

D1: Build Reports. An Analytic Maturity Level (AML) D1 organization can analyze data, build reports summarizing the data, and make use of the reports to further the goals of the organization.

D2: Build and Deploy Models. An AML D2 organization can analyze data, build and validate analytic models from the data, and deploy a model into an organization’s products, services, or internal operations.

D3: Use a Repeatable Process to Build and Deploy Models. An AML D3 organization follows a repeatable process for building, deploying and updating analytic models. For most organizations, a repeatable process for building and deploying analytic models usually requires a functioning analytic governance process.

D4: Strategy Driven Repeatable Analytics. A team, no matter how talented, can usually only build a handful of genuinely new analytic models each year. An AML D4 organization has an analytic strategy, aligns the analytic strategy with the organizational strategy, and develops and deploys the models as prioritized by the analytic strategy.

Of course, with the appropriate automation, any given model or category of model can be built many times, but when something new is required, it takes a team, and this is usually what is in short supply. With a analytic maturity level of D4, the effort of the team is spent on the right models, that is, the models that bring the most value to the organization. In other words, the choice of which models to build is driven by an strategy that is congruent with the organizational strategy.

These four levels are the same first four levels in the AMM-17 model described in my earlier post.

Analytic Maturity Levels for Enterprise-Scale Analytics

As companies grow in size, it is critical that analytic be replicated in any department, center of division that it can add value.

E1: Enterprise support. In AML E1 organization, IT support for analytic efforts, so that analytic projects get the data they need, get the computing infrastructure they need, and the support they need to deploy the models that they build?

E2: Replicate. An AML E2 organization replicates analytics throughout the organization, across the departments/divisions that can benefit from it. Different departments/divisions have the support they need from enterprise IT services

E3: Effective analytics governance to coordinate. An AML E3 organization has enterprise analytic services supporting different analytic projects, efforts and groups that provide enterprise data and services when required, while providing the support, flexibility and autonomy each effort requires to move at an optimal velocity. An AML E3 organization has services for enterprise data and metadata that provide a uniform level of integrity for the data, models, and scores across the enterprise.

E4: Holistic management and integration of analytics. An AML E4 organization aligns and integrates the analytic models used by one unit with those used by another unit. For example, in a company that provides credit, analytic models for customer acquisition are aligned with analytic models for risk determination so that new customers are the right customers and not customers who are likely to default on their payments 18 – 24 months later. As another example, for social media companies, analytic models that rank news items and posts for displaying to users to maintain customer engagement are aligned with analytic models for identifying news items and posts for inappropriate content so that items prioritized for users are consistent with the organizations policies and values.

Five Steps to Improve Your Company’s Analytic Maturity

Here are five basic steps to take to understand and quantify the analytic maturity level of your enterprise.

Step 1. Identify where in the organization analytics is done and evaluate the analytic maturity of each of these efforts (using D1 – D4)

Step 2. Identify the enterprise level of analytic maturity that supports these efforts and harmonizes them (using E1 – E4).

Step 3. Review:

the level of IT support for each analytic effort and how it can be improved
the effectiveness of analytic governance in helping each of these analytic efforts (vs slowing them down) and how it can be improved.
what are analytic opportunities and analytics risks that fall between the cracks of existing efforts and are the responsibility of an enterprise analytics office?

Step 4. Identify one to three new analytic efforts that can provide the most value to the organization as a whole.

Step 5. Identify one to three risks associated with analytics that can provide the greatest harm to the organization as a whole.

Improving the Analytic Maturity of Your Company
Analytic Governanc e and Why it Matters

References

[Grossman2017] Robert L. Grossman, “A framework for evaluating the analytic maturity of an organization,” International Journal of Information Management, 1 February 2018, Volume 38, Number 1, pages 45-51, available online 22 September 2017 at https://doi.org/10.1016/j.ijinfomgt.2017.08.005 (open access).

[McCormick2020] John McCormick, “AI Project Failure Rates Near 50%, But It Doesn’t Have to Be That Way, Say Experts”, Wall Street Journal, Aug.7,2020, available here.

[Paulk1993] Paulk MC, Curtis B, Chrissis MB, Weber CV. Capability maturity model, version 1.1. IEEE software. 1993 Jul;10(4):18-27.

[Press2019] Gil Press, “This Week In AI Stats: Up To 50% Failure Rate In 25% Of Enterprises Deploying AI,” forbes.com, July 19, 2019, available here.

Twelve Rules for a Chief Analytics Officer

December 6, 2019 by Robert Grossman

About the picture: George Heilmeier was the Director of DARPA from 1974-1977, where he developed seven questions that he used to determine whether to fund research projects. The questions became known as Heilmeier’s Catechism and are still used today. The image is from: https://en.wikipedia.org/wiki/George_H._Heilmeier

George Heilmeier (1936-2014) was a engineer and technology executive who started his career working in liquid crystal displays (LCD) at RCA (1958-1970), became a White House Fellow (1970-1971), a senior executive at DOD (1971-1974), the Director of DARPA (1974-1977), a technology executive at Texas Instruments (1977-1991), and the President and CEO of Bell Communications Research Inc. (Bellcore) (1991-1996).

He was elected to the US National Academy of Engineering in 1979, and in addition to other awards and recognitions, he won the 2005 Kyoto Prize for his invention of the liquid crystal display.

While working at DARPA, he came up with a simple set of rules for selecting and managing technology projects that became known as Heilmeier’s Catechism. There are several formulations, including the following list of questions:

Heilmeier’s Catechism

What are you trying to do? Articulate your objectives using absolutely no jargon.
How is it done today, and what are the limits of current practice?
What’s new in your approach and why do you think it will be successful?
Who cares? If you’re successful, what difference will it make?
What are the risks and the payoffs?
How much will it cost? How long will it take?
What are the midterm and final “exams” to check for success?

He became the CTO at Texas Instruments and developed a framework for CTOs that is lesser known and consisted of twelve rules. The rules can be found in a December 30, 2009 blog post by Rich DeMillo, who is a Distinguished Professor of Computing and Management at Georgia Tech in Atlanta.

In 2012, I adapted these 12 rules to create 12 rules for Chief Analytic Officers or Chief Data Officers. I used to go over these when I taught a course at the University of Chicago Booth School of Business on the Strategy and Practice of Analytics (2013-2015) . You can learn more about these rules in Chapter 10 (Analytic Governance) of my upcoming book The Strategy and Practice of Analytics. Here are the rules:

Twelve Rules for Chief Analytic Officers (or Chief Data Officers)

For each “client” establish/conceive a list of technologies and initiatives that drive the business and a list of technologies and initiatives that could change the business.
Use the Catechism to get people to focus on essentials when making analytic investment decisions and establishing analytic priorities.
Understand the limits of analytic technologies and capabilities today.
Establish a good working relationship with your peers in marketing, sales, finance, HR, etc.
Align analytic strategy with the real priorities and the strategy of your organization, as understood by your CEO and Chairman.
Establish metrics for the success of your programs in their eyes.
Don’t shy away from doing some near term problem solving and working basic projects in analytics. It builds credibility and respect.
Never have your peers or clients come to your office for meetings with you. Go to theirs.
It is easy to be arrogant when discussing a technical subject. But remember, any display of arrogance will cost you. Don’t do it.
Compile a list of innovations that are needed in big data and analytics by your organization. How are these the same and how are they different from the innovations required by the broader analytic community.
Make sure that each program or initiative is output oriented not activity oriented. Output = value to your organization.
Learn your company’s culture. It is unique.

As mentioned, these are basically Heilmeier’s rules for a CTO, just lightly adapted.

Improving the Analytic Maturity Level of Your Company

October 2, 2019 by Robert Grossman

The figure shows the five levels of analytic maturity – Analytic Maturity Level (AML) 1 -5.

Over the past 10 years or so, I developed and used an analytic maturity model that was broadly motivated by the software Capability Maturity Model (CMM) that is used in software development.

You can learn more about the model in this article: A Framework for Evaluating the Analytic Maturity of an Organization, International Journal of Information Management, 2018, which is open access and available here: doi.org/10.1016/j.ijinfomgt.2017.08.005. This article introduces five Analytic Maturity Levels (AML) and discusses the analytic processes required to achieve each level.

Over the past few years I have been working on a book called the Strategy and Practice of Analytics and have been thinking about analytic maturity levels again. Here are the five levels as defined in the book:

AML 1: Build reports. An AML 1 organization can analyze data, build reports summarizing the data, and make use of the reports to further the goals of the organization.

AML 2: Build models. An AML 2 organization can analyze data, build and validate analytic models from the data, and deploy a model.

AML 3: Repeatable analytics. An AML 3 organization follows a repeatable process for building, deploying and updating analytic models. In my experience, a repeatable process usually requires a functioning analytic governance process.

AML 4: Strategy driven repeatable analytics. An AML 4 has an analytic strategy that aligns with the corporate strategy, uses the analytic strategy for selecting which analytic opportunities to pursue, and follows a repeatable process for building, deploying and updating analytic models.

AML 5: Strategy driven enterprise analytics. An AML 5 organization uses analytics throughout the organization and analytic models in the organization are built with a common infrastructure and process whenever possible, deployed with a common infrastructure and process whenever possible, and the outputs of the analytic models integrated together as required to optimize the goals of the organization as a whole. Analytics across the enterprise are coordinated by an analytic strategy and analytic governance process.

Five ways to improve the analytic maturity level of your organization. Here are five suggestions for improving the analytic maturity level of your organization.

Set up a committee to quantify the analytic maturity of your company. If you cannot measure it, you cannot improve it.
Set up (or improve) the environment for deploying analytic models, using a model interchange format, analytic engines, or similar technology. (Helpful for AM Level 2).
Set up your first analytic governance committee or improve the operational efficiency of your current analytic governance. (Helpful for AM Level 2).
Set up SOPs for building and/or deploying analytic models so that the process is faster, repeatable & replicable. (AM Level 3 Requirement)
Volunteer to lead a process to integrate two different models from two different parts of your company to improve the relevant actions. (Helpful for AM Level 5)

What’s different between AMM-12 and AMM-17? Although the paper describing the analytic maturity model was published in 2017, the work dates back to period 2010-2012. I first gave a talk about the AMM in 2012 at a Predictive Analytic World (PAW) conference that took place on June 25, 2012 in Chicago. Since the AMM didn’t change between 2012 and when it was published in 2017, let’s call this the AMM-2012 model.

In the AMM-2012, AML 1-3 are the same, but AML 4 is Enterprise Analytics and AML 5 is Strategy Driven Analytics. The problem with this approach is that strategy driven analytics can occur at any level, so it is not particularly helpful to reserve it for level 5. From one perspective whether, strategy can be integrated into processes for building reports (AML 1), for building models (AML 2), for building models in a repeatable fashion (AML 4), or for building models across the enterprise in a repeatable fashion (AML 5). With this approach, reports -> models -> repeatable models -> enterprise models would be one axis and the degree of strategy integration would be another axis. This seemed too complicated though for most applications, so for the AML-19 versions of the AMM, AML 4 is simply strategy driven repeatable analytics and AML-5 is strategy driven enterprise analytics.

I speak about analytic maturity models from time to time, most recently at the Predictive Analytics World (PAW) Conference that took place in Chicago on June 29, 2017 and The Data Science Conference Chicago (TDSC) that took place in Chicago on, April 20, 2017. I also occasionally give in house training courses that include material about helping a company increase its analytic maturity level.

You can find more information about Analytic Maturity Models in Chapter 14 of my book The Strategy and Practice of Analytics.