I find it helpful to have simple rubrics when evaluating analytic projects as an advisor. I have written previously about George Heilmeier and the rubric he used when he evaluated projects as Director of DARPA from 1975-1977. The seven questions he asked when evaluating projects are now back on the DARPA website [1].
Eight Questions
Here are 8 questions that I find helpful to ask when evaluating a project that works with data and produces machine learning or analytic models:
Question 1. What datasets (or data products) will this project produce? How will they be made available so that they are findable, accessible, interoperable and reusable (FAIR)?
The most basic question to ask is what datasets will be produced and how they will be made available so that others can find them, access them and use them [2].
Question 2. How is the use and value created from the datasets being measured and tracked?
The second most basic question is how is the use and the value produced by the use of the data being measured and tracked? This is a surprisingly difficult question and often one simply counts the times the data is used by others. Even better is to track what is produced by others using the data, such as any reports, publications, products, or services produced using it.
Question 3. What is the expected impact? If you are successful, what difference will it make?
This question is one of the questions that George Heilmeier always asked. It is related to Question 2, but goes deeper. It’s not, for example, the data is used by two other processes in our organization, but, rather, it is only using this data that our organization can do X, and X has impact Y for our organization.
Question 4. Will the datasets be in a format and available via an API that supports range queries (“sliceable data”) or must the entire dataset be transferred?
Question 4 is more technical, but important. Just because data is available does not mean it is easy to use by other applications. Question 4 is about whether the data produced by your project will be available via an API that allows other applications to get the data they need without accessing, hosting and parsing the entire dataset.
Question 5. What machine learning (ML) or AI-ready datasets is the project producing and making available?
Data is often produced in transactions, events or encounters, (we’ll use the term “events” for simplicity), but usually machine learning or AI models are based upon the entities that produce the events. For each entity, there is a feature vector and the input to a machine learning model is often a matrix, data frame or similar structure with rows corresponding to the entities and columns corresponding to the features. These days, data structured this way is beginning to be called AI-ready. To clean and curate the data and produce features from the events can be quite labor intensive. It usually also requires quite a bit of domain knowledge.
Question 6. How is the curation, provenance, and processing of the data being managed and made available? What steps are being taken to make sure that the curation and analysis of the data is reproducible?
If the data and data products are being used by others, then others can usually benefit by reusing the processes to curate and process the data, since sometimes they need to make minor (or major) changes to the processes.
Question 7. What is the sustainability plan for maintaining the datasets and continuing to make them available?
For business, this is a function of the return generated by the data produced by the project, including downstream projects that reuse and repurpose the data. For not-for-profits, this is a more complex function of the value generated by the data, how the value generated by the data impacts the mission, and a comparison involving how much mission value and impact could be generated by alternate uses of the funds required for the project.
Question 8. What steps are you taking, what standards are you following, and what software services are you using so that the data you are producing and the software applications and services that you are developing can be used by other projects?
This question is related to Question 2, 4, 5 and 6, but is framed a bit differently. Almost always extra work is required to take data from one project and make it more easily used by other projects. This question begins to address the trade-off about whether additional work should by undertaken by the project so that other projects can more readily leverage the data it produces. This is a trade-off at the organizational level or at the mission level.
Two Contexts
There are different contexts in which to ask and to evaluate these questions. The first is within the context of your organization in order to generate a competitive advantage for your organization as part of an analytic strategy.
Another context is is as part of a not for profit mission in which you are using data sharing to advance your mission. For example, to accelerate research by sharing data. From this perspective, these questions can help answer how the project can advance what is being called open data, and, more broadly, open science [3].
On the Origin of Rubric
Today, in education, a rubric is a scoring guide used to evaluate student performance or a student project, assignment, or exam. The term rubric has becoming widely used and many of us use, for example, rubrics for evaluating job candidates in interviews.
Rubric derives from the Latin ruber or rubeus. The Latin word rubrica was the name for red earth, red ochre or red chalk. As another example, the English word ruby, a red gemstone, derives from the Latin rubinus lapis for red stone.
In medieval manuscripts, the first letter of paragraphs and section headings was sometimes in red ink for emphasis [4]. Adding the red section headings and related markings was called rubrication. Later, with the advent of printing, in printed liturgical texts, black ink was used for what should be recited and red ink was used for what should be done. The instructions in red ink were called rubrics.
Rubric has also been used in a number of different figurative senses over time. The Oxford English Dictionary dates its use to the 1400’s and included as one of its definitions: an established custom; a set of rules, an injunction; a general prescription [5]. For example, “an important rubric of open data is to make the data FAIR.”
References
[1] DARPA, The Heilmeier Catechism, retrieved from https://www.darpa.mil/work-with-us/heilmeier-catechism on March 1, 2022.
[2] Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg et al. “The FAIR Guiding Principles for scientific data management and stewardship.” Scientific Data 3, no. 1 (2016): 1-9.
[3] Fecher, Benedikt, and Sascha Friesike. “Open science: one term, five schools of thought.” In Opening science, pp. 17-47. Springer, 2014.
[4] Wikipedia, rubric, retrieved from https://en.wikipedia.org/wiki/Rubric on March 1, 2022.
[5] Oxford English Dictionary, rubric, retrieved from https://www.oed.com/view/Entry/168394 on March 1, 2022.