How Much Precision is Really Required in Diversity Measurement?

Any time I have presented on the subject of measuring diversity and inclusion’s impact on the bottom-line, issues of accuracy and precision invariably arise. It has been argued that to measure diversity activities convincingly or with any high degree of confidence is difficult. Diversity measures based on “report cards” don't allow for competing hypotheses in assertions such as “managing diversity leads to increased profitability.” Some would argue that other efforts could just as easily have caused the profitability to increase. Another contention is that there is lack of control over thousands of factors that influence a large organization’s profitability. Factors such as inflation, labor market conditions and cost of money can make it virtually impossible to effectively measure diversity with accuracy.

In general, this argument is correct, though similar conditions prevail throughout the organization. Everyone knows certain factors are not controllable. The marketing department does not have control over the product or the customer, and the finance department does not control the cost of money or inflation. Yet both departments are able to evaluate much of their work quantitatively. If we are willing, we might admit that there are more issues out of control than in the control of any organization. The task of management is to reduce the uncontrollable variables and to instill as much order as possible.

C-suite management does not require accuracy at the .05 level of statistical significance for general reports of progress in diversity efforts. In research, precision is critical, obviously. In pharmaceuticals or medicine, extreme care must be taken with procedures and measurement. Results are often required to be statistically valid beyond the .001 level. That kind of measurement with a capital “M” is not required in the operational measurement of diversity just to get a sense of the progress being made in your diversity efforts. We are not operating in a laboratory but in the field, with all the problems inherent in field research and experimentation.

Accuracy is necessary, but precision is naturally limited by internal and external conditions. Though we can't control the variables in the environment completely, we can still come up with usable numbers and “compelling evidence” of diversity’s contribution to the bottom line. However, there are some exceptions. For example, diversity measurement leading to diversity ROI impact does require the use of scientific processes and analytics to demonstrate correlation and causality. A seven-step methodology like the Hubbard Diversity ROI Methodology can be used to highlight these relationships and reveal what drove the results using evidence-based outcomes.

Seven Types of Data Capture

The Hubbard Diversity ROI methodology captures seven types of data, with the actual diversity ROI calculation being only one of them. The seven types of data include:

  • Level 0: Needs analysis
  • Level 1: Reaction, satisfaction and planned action
  • Level 2: Learning
  • Level 3: Application and behavioral transfer
  • Level 4: Business impact and consequences
  • Level 5: ROI
  • Level 6: Intangible benefits (includes measures that cannot be crediblyconverted to monetary values)

Here is a sample of a few of the most-debated questions:

1. Isn’t diversity ROI calculations based on nothing but estimates that can be too subjective?
No. Estimates are only used when isolating the effects of a diversity program or diversity intervention’s business impact, when converting data to monetary values and when tabulating program costs. Estimates are used only when other methods are not available or become too time-consuming or expensive to use. Also, estimates are adjusted using “confidence level” factors to improve credibility.

2. How does ROI in diversity and inclusion differ from the ROI used by the finance and accounting staff?
The classic definition of ROI is earnings divided by the investment — no matter what the application. In the context of calculating the ROI of diversity and inclusion, the earnings become the net benefits from the program/initiative (monetary benefits minus the costs) and the investment is the actual program cost. The difficulty lies in developing the actual monetary benefits in a credible way.

3. Do I have to be a whiz at finance and statistics to understand the ROI methodology?
No. Most of the basic principles of finance and accounting do not relate to what is needed to develop the ROI in diversity and inclusion. However, it is important to understand issues such as revenue, profit, and cost. Only basic statistical processes are required to develop most diversity ROI impact studies.

4. Doesn't ROI cost too much?
No. When external resources are used, the cost for a diversity ROI study may be as little as 5 percent of the entire project. A large banking group and a large telecommunications company using similar approaches report that the average costs for their ROI studies range from $3,000 to $5,000 per study. The total cost of all evaluation, including selected ROI studies, is usually in the range of 3 to 5 percent of the total project budget.

5. Does the ROI process reveal program weaknesses? Strengths? Recommendations?
Yes. At low levels, data always capture deficiencies or weaknesses in a process. At the application level, the process requires collecting data about the barriers, which inhibit success, and enablers, which help success. Each study contains a section for recommendations for improvement. You must know what your weaknesses are in order to improve them. In addition, you want to avoid losing strengths that are working for you and providing value.

6. Is it appropriate to calculate ROI for every program?
No. Only a few select programs should be subjected to evaluation at the ROI level. Ideal targets include programs that are very expensive, strategic, operationally focused, highly visible, involve large target audiences, and have management’s attention in terms of their accountability.

7. What types of applications are typical for ROI analysis?
The applications vary, but usually they include a range of programs such as leadership competencies for a diverse workforce training program, unconscious bias training, measuring the ROI of ERG and BRG initiatives, multicultural marketing and sales initiatives, innovation and diverse work team performance interventions, etc.