Today’s chief learning officers must show accountability for their learning investments. Many CLOs have found that measuring the return on investment of a few selected, high-profile programs is an excellent way to show fiscal responsibility for key projects and initiatives. Our best guess is that about 30 percent to 40 percent of CLOs are using ROI as a tool. According to the Corporate Executive Board, almost 80 percent of major learning and development organizations want to use ROI in the future. Actual use versus desired use underscores the misunderstandings and misconceptions of ROI as a legitimate part of the measurement mix. For almost two decades, The ROI Institute has been assisting organizations with this important issue. In the past five years, we have kept track of the many questions that are asked about ROI in conferences, workshops and consulting assignments. Here are the 20 most frequently asked questions about ROI.
1. How does ROI for learning initiatives differ from ROI used by financial staff? The classic definition of “return on investment” is earnings divided by the investment—no matter what the application. When calculating the return on investment for learning and development, the earnings become the net benefits from the program (monetary benefits minus the costs), and the investment is the actual program cost. The difficulty lies is developing the actual monetary benefits in a credible way.
2. Do I have to know statistics to understand ROI? The very basic statistical processes are all that are necessary to develop most ROI impact studies. Rarely are statistics needed beyond simple averages, variance and the standard deviation. Sometimes, hypothesis testing and correlations are necessary. These are very simple concepts and, by design, are simplified as much as possible.
3. Is ROI just a single number? How can you communicate a program’s value with a number? The ROI methodology (or the ROI process) develops six types of data, with the actual ROI calculation being only one of them. The six types of data are:
- Reaction, satisfaction and planned action.
- Learning and application.
- Implementation.
- Business impact.
- ROI.
- Intangibles.
4. Aren’t the levels of evaluation out of date and not applicable? The original four levels developed by Donald Kirkpatrick, Ph.D., show how the data must be developed to generate value from learning and development programs. The data are arranged in a chain of impact that must exist if the learning has business impact, which ultimately becomes business value. The chain of impact can be broken at any point, thus correlations do not always exist between the levels, because there are barriers to success at any level. Although a few researchers take issue with the four levels, it is still the most widely used foundation for evaluation. ROI becomes the fifth level and is the consequence of the program expressed in monetary terms.
5. Isn’t ROI based on nothing but subjective estimates? Estimates are usually used in these areas:
- Sometimes the amount of improvement is estimated when records are not readily available to show the improvement or, in a forecast situation, where it is not known.
- When isolating the effects of a program.
- When converting monetary values.
- When calculating the costs (acceptable finance practice).
Estimates are used only when other methods are not readily available, or become too time-consuming or expensive to obtain. When estimates are taken, they are adjusted for the error of the estimate to improve their credibility. In essence, results are understated. In every case, there are many alternatives to estimates, and they are recommended. Estimates are used routinely in some situations because they become the preferred method and are accepted by stakeholders or because they may be the only way to obtain the needed data.
6. Isn’t ROI too complicated for most learning and HR professionals? The ROI calculation itself is a very simple ratio: benefits divided by costs. Benefits and costs are both developed using methodical, step-by-step processes with guidelines and principles. The many options within each step of the process make it more complicated. These options are critical due to the many different situations, programs and projects that need to be evaluated and the different environments and settings in which they occur.
7. Doesn’t ROI cost too much? The cost for a study all the way through to ROI may represent as much as 5 percent to 10 percent of the entire project. This number varies considerably. In most organizations, every program is evaluated at some level. The total cost of all evaluation, including selected ROI studies, is usually in the range of 3 percent to 5 percent of the total learning and development budget.
8. Is it possible to isolate the effects of my program from other factors? This is the most difficult and challenging issue, but it is always possible, even if estimates are used. Sophisticated and credible processes involve control groups, trend-line analysis and forecasting models. Other techniques are expert estimation and customer input. Always strive to carve out the amount of data directly related to the program or project. When estimates are used, the data should be adjusted for possible errors.
9. Is it true that the ROI process does not reveal program weaknesses or strengths? No, of the six types of data captured by the ROI methodology, Levels 1, 2 and 3 always capture deficiencies or weaknesses in the process. At Level 3, the process requires collecting data about the barriers (which inhibit success) and enablers (which help success).
10. Is it true that the ROI process does not result in recommendations for improvement? No, each impact study using the ROI methodology contains a section for recommendations for improvement. This tool must be used as a process-improvement tool, even when studies have reflected a very successful project.
11. Is it appropriate to do ROI for every program? No, of the few select programs that should be subjected to evaluation all the way through to Level 5 (ROI), ideal targets include programs that are very expensive, strategic, operationally focused or highly visible, and those that involve large target audiences and have management attention in terms of their accountability. In most organizations, only about 5 percent to 10 percent of programs are selected for ROI analysis each year.
12. Who is using the ROI methodology? All kinds of organizations in the United States and around the world are using the ROI methodology:
- More than 2,000 private-sector organizations have formally implemented ROI through skill building and ROI certification. Others have informally implemented the methodology.
- Some 200 government agencies and nonprofit organizations are implementing the process.
- Almost 20,000 specialists and managers have taken one-day or two-day ROI workshops.
13. What types of applications are typical for ROI analysis? Applications can vary, but usually include sales training, supervisory training, team building, executive development, communications, competency systems, software utilization, leadership development, diversity, orientation systems, compensation and benefits, reward systems, skill-based pay, career management, major projects and wellness initiatives. These topics make excellent applications and have been documented with case studies.
14. Can ROI be applied on a pre-project basis in a forecasting mode? ROI forecasting is an important part of the ROI methodology. Using credible data and expert input, and estimating the improvement (projected benefits) that will occur upon implementation, projected benefits are compared to projected costs to develop the forecast ROI.
15. How does ROI compare to a balanced scorecard? The ROI process generates six types of data: reaction, learning, application/impact, business impact, ROI and intangibles. This data essentially is a scorecard. The balanced scorecard process developed by Robert S. Kaplan and David P. Norton (“The Balanced Scorecard” from Harvard Business School Publishing, 1996) suggests four categories of data: learning and growth, internal business processes, financial and customer. The data generated with the ROI methodology may be grouped into one of these four categories. In addition, the ROI process adds two additional capabilities not normally contained in balanced scorecard methodology: It provides a technique to isolate the effects of a program, and it shows the costs versus benefits of a particular program or initiative. Thus, the ROI methodology will compliment the balanced-scorecard process.
16. How can I secure support for ROI? Building support for the ROI methodology is an important issue. Top executives will usually support the process when they realize the types of data that will be generated. Most of the resistance comes from those directly involved in programs because they do not understand ROI and how it is used in the organization. When they are involved in implementing the methodology and the data are properly used to drive improvements, it helps to lower the resistance. The efforts to implement any major change program will apply with the implementation of the ROI methodology.
17. How can I minimize staff resistance? Most learning and development and HR staff will have some resistance to ROI, unless they see the value it can bring to their work. Involvement, education and process improvement are key issues. It is often the fear of ROI that generates resistance—a fear based on misunderstandings about the process and how the data will be used. The ROI methodology should be implemented as a process improvement tool and not as a performance evaluation tool for the HR or learning and development staff. No one wants to develop a tool that will reflect unfavorably on his or her performance review. Improvement in key decisions about the use of ROI will help minimize resistance. Also, resistance will be minimized when steps are taken to ensure that the data are communicated properly, improvements are generated, and the data are not abused or misused.
18. Should I conduct an ROI study on my own program? If possible, the person evaluating the program should be independent of the program. It is important for the stakeholders to understand that the person conducting the study is objective and removed from certain parts of the study, such as the data collection and the initial analysis. Sometimes these issues can be addressed in a partnering role or limited in outsourcing opportunities—whether data collection or analysis. In other situations, the issue must be addressed and the audience must understand that steps are taken to ensure that the data were collected objectively, analyzed and reported completely.
19. Are there any standards for ROI? The ROI methodology contains standards labeled “Guiding Principles.” These provide consistency for the analysis with a conservative approach. The conservative approach builds credibility with the stakeholders. Additional standards are under development and will be made available in the near future.
20. How is ROI on e-learning developed? Applying the ROI methodology to e-learning is the same as any other process, program or solution. The monetary value of the benefits from the e-learning is compared to the cost of the e-learning. Many individuals assume that the benefits of training remain the same in that only the costs to provide training changes. What makes e-learning studies somewhat different is that these individuals assume that the e-learning is more cost-effective on a larger scale. An ROI study should be conducted to show the actual value of the training. When an instructor-led program is compared to an e-learning program, the effectiveness of the training when comparing the impact data (impact data from both programs) is compared with the respective costs for each of the programs. A higher ROI shows a more successful program in terms of providing value that exceeds costs.
Jack. J. Phillips, Ph.D., is chairman of the ROI Institute. He developed and pioneered the ROI process and has written more than 15 books on the subject. Jack can be reached at jphillips@clomedia.com.
