More on Re-evaluating Evaluation – Jack Phillips and ROI

Dashe & Thomson

I have been blogging a lot about Training Evaluation this year—mostly Kirkpatrick , but also Brinkerhoff and Scriven. I just realized that I haven’t included a single word about Jack Phillips , who introduced Return on Investment (ROI) as Level 5 to Kirkpatrick’s Four Levels of Evaluation. My first exposure to Phillips’ ROI—although I didn’t realize it at the time—was through a colleague who introduced me to Kirkpatrick’s Four Levels. Collect Pre-Program Data.

Extend Your Training Evaluation To Include The Phillips ROI Model

eLearning Industry

Including the Phillips ROI model within your training evaluation will serve to quantitatively justify the investment into your training programs. Training ROI eLearning ROI Measure Training Online Training Training EvaluationThis post was first published on eLearning Industry.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The Kirkpatrick-Phillips Model – Part 4

CommLab India

In the third part of this series, we looked at the advantages and disadvantages of the Kirkpatrick Model of Evaluating a training program. I mentioned the Kirkpatrick-Phillips Model of Evaluation in passing, and promised you more on this “ evolved ” model. I call this an “ evolved ” model because, the Kirkpatrick-Phillips Model of Evaluating a training program is based on the Kirkpatrick Model of Evaluation. Phillip ROI Methodology TM.

An Overview: Phillips' Model for Learning Evaluation


When we originally published our Learning Evaluation eBook and blog series, we covered four learning evaluation models representing a range of approaches and perspectives: Kirkpatrick, Kaufman, Brinkerhoff, and Anderson. While there are many other learning models we’ve yet to cover, the one we’re asked about most often that’s not mentioned in our original series is Phillips' Learning Evaluation Model. Meet the fifth level of learning evaluation.

Training Evaluation – 5 Best Ways to Evaluate Training Effectiveness and Impact


Training evaluation refers to an attempt to obtain relevant information on the effects of a training program. The primary objective of evaluating any training program is to develop an understanding of whether it has achieved its stated objectives. There are several types of training evaluation methods and tools available that enterprises can use to evaluate and significantly improve the outcome of future training as well. The Phillips ROI Model.

Showing the ROI of key programs on a budget

CLO Magazine

The talent development team is challenged with showing the value of key programs, particularly the important, expensive programs involving a lot of people. At the same time, you don’t necessarily have the resources to devote to a comprehensive evaluation. Here are 7 ways to minimize resources and still show the value of a major program at the impact and ROI levels. Everyone has a role in making the program work. 7: Be selective when evaluating to this level.

ROI 56

Interview with Jack Phillips on ROI for eLearning


Jack Phillips is founder and CEO of the ROI Institute, Inc., Phillips developed the ROI Methodology™, a critical tool he has used for measuring and evaluating programs such as training, human resources, technology and quality programs and initiatives. The challenge is that people who own programs may not want to measure them at the impact and ROI levels. Yet the executives who are funding these programs want to measure them.

Learning to Learn from Evaluation of Learning

The Performance Improvement Blog

The Kirkpatricks have four levels, the Phillips have ROI, and Brinkerhoff has the Success Case Method. Each approach to evaluation of training has something to contribute to assessing the impact of formal training on employee learning. However, the value of evaluation is not in the data. The real value is in organizational learning from evaluation. Learning from evaluation depends on asking the right questions.

Short Course on Evaluation of Training and Learning in Knowledge Economy

The Performance Improvement Blog

LAD Global , in partnership with the Singapore Training and Development Association , has made my short course on evaluation of training and learning available for free online. My emphasis in this course is on using measurement and evaluation for learning. Much of evaluation in organizations today is still focused on formal training programs and limited to Kirkpatrick’s “level one”.

Kirkpatrick’s Four Levels of Evaluation


It was while writing his thesis in 1952 that Donald Kirkpatrick became interested in evaluating training programs. In a series of articles published in 1959, he prescribed a four-stage model for evaluating training programs, but it was not until 1994, that he published “ Evaluating Training Programs: The Four Levels “ According to Kirkpatrick, evaluating training programs is necessary for the following reasons: 1.

Evaluating Training Effectiveness and ROI


To evaluate the effectiveness of education and its practicability from the financial standpoint, it is necessary to calculate the expenses connected with creating and conducting the trainings, gauge the results achieved by the employees who have taken part in them, and decide whether the increase in the employees’ efficiency and company profits was sufficient to recoup the associated costs. The Donald Kirkpatrick’s Learning Evaluation Model consists of four levels: Level 1.

ROI 100

What Is Learning Evaluation?


Knowing how to aggregate and evaluate all types of learning is a priority for many learning and development practitioners. But due to challenges, such as employees using multiple avenues for learning, many practitioners find it challenging to measure and communicate the value of their learning programs. Technologies are becoming available that enable L&D professionals to aggregate, evaluate, and measure many types of learning data in one place. Learning Evaluation



It was while writing his thesis in 1952 that Donald Kirkpatrick became interested in evaluating training programs. In a series of articles published in 1959, he prescribed a four-stage model for evaluating training programs, but it was not until 1994, that he published “ Evaluating Training Programs: The Four Levels “. According to Kirkpatrick, evaluating training programs is necessary for the following reasons: 1.

Is Kirkpatrick’s Model of Evaluating a Training Program The Best? – Part 3

CommLab India

In the first part of the series, I covered the need to evaluate any training program and the basics of the Kirkpatrick model of evaluating a training program. Here’s a quick diagrammatic representation of the gist: The eLearning industry relies tremendously on the 4 levels of the Kirkpatrick Model of Evaluating a training program. Therefore, it would not be right to make changes to a program based on learners’ reactions.

Evaluation and ROI in Compliance Training

Interactive Services

The chances are your program includes some or all of the following: A focus on behavior change and employee conduct rather than just knowledge. Elements of micro-learning, collaborative learning, and comprehensive communication programs designed to reinforce critical lessons and ensure they stick. But what about evaluation and return on investment (ROI)? . Evaluation and ROI are important indicators of how well a program is working and how much risk is mitigated.

Three Superheroes of Training Evaluation

Growth Engineering

Well, let’s look at the heroes of training evaluation and how they helped make sense of training evaluation! Followed by Roger Kaufman and then Jack Phillips. This trio laid the foundations that help you evaluate your training. KNOWN FOR: The four levels of learning evaluation. QUOTE: ‘ The immediate objective of any training can be stated in terms of the desired knowledge and understanding that the program is trying to impart to the trainees’.

Predicting Learning Success

CLO Magazine

Phillips is chairman and Patti P. Phillips is president and CEO of the ROI Institute. Part of the evaluation mix should be the predictions of success. Think about the logical chain of value from any learning program: level 1, reaction; level 2, learning; level 3, application; level 4, impact; and level 5, ROI. We know from experience that programs are evaluated infrequently at level 3, less often at level 4 and rarely at level 5. Jack J.

Evaluating Training – Capturing the Benefits Aspect of ROI

Obsidian Learning

New post Evaluating Training – Capturing the Benefits Aspect of ROI on Obsidian Learning. Training evaluation is necessary and, in many ways, critical to the success of a business. And even if training evaluation is undertaken, it is usually at the easiest and lowest level: the measurement of student reactions through simple surveys or happy sheets. When BCR is greater than one, the benefits outweigh the costs and the program is considered a success.

Watershed's 7 Steps of Learning Evaluation


Out of the dozens of different learning evaluation models currently in practice, which one is right for you? Meet our "super method" for learning evaluation. We've been using Watershed Insights to dig into several models of learning evaluation in light of what's possible with modern technologies such as xAPI. Here's an overview of our practical seven-step evaluation process that you can implement in your organization: Step 1: Align learning program evaluation with priorities.

Determining The ROI Of eLearning – Using Kirkpatrick’s Model Of Training Evaluation

EI Design

In this article, I outline how you can use the Kirkpatrick’s model of training evaluation to measure training effectiveness, its impact, and the ROI of eLearning. What Is Kirkpatrick’s Model Of Training Evaluation?

Impact and ROI of Learning: Worth Pursuing or Not?

CLO Magazine

The primary reason is that senior corporate leaders like the CEO and CFO want to see it and so should the department head and program director. In conclusion, we must not shy away from higher levels of evaluation because they are harder or more subjective.

ROI 64

Commonly Used Training Evaluations Models: A Discussion with Dr. Will Thalheimer

Convergence Training

And that’s especially true when it comes to issues regarding learning evaluation. We were excited to be able to talk with Dr. Thalheimer about four common learning evaluation models, and we’ve got the recorded video for you below. If you prefer your learning evaluation information in written form, just scroll down for the transcript of our discussion. Many thank to Will for participating in this discussion on learning evaluation and for everything he does.

ROI: Determine the Impact of Your Training Programs


And, measuring the ROI for your training programs is no exception. Industry Growth Means Managers Need to Understand the Business Impact of Their Training Programs. Industry growth suggests that companies will continue to invest in training programs – online and offline. It’s clear that there are lots of advantages to implementing online training programs. Return on Investment is a popular financial metric for evaluating the financial consequences of investments.

ROI 40

Determining The ROI Of eLearning – Using Kirkpatrick’s Model Of Training Evaluation

Adobe Captivate

In this article, I outline how you can use the Kirkpatrick’s model of training evaluation to measure training effectiveness, its impact, and the ROI of eLearning. ROI determination methodology: One of the popular models used for ROI determination is the Kirkpatrick’s model of training evaluation. What Is Kirkpatrick’s Model Of Training Evaluation? Kirkpatrick’s model of training evaluation is one of the popular models used to evaluate the effectiveness of training.

ROI 56

ROI Goes to School and Church

CLO Magazine

How are these organizations using the ROI Methodology to show the value of their projects and programs? The Phillips ROI Methodology is a 10-step process organizations can use to show the value of programs and initiatives. Across the United States, school systems are implementing the ROI Methodology in their education programs, departments and functions. Programs such as these can have tremendous payoff, particularly in student and educator outcomes.

ROI 40

Putting Thought to Work: Evaluation in Practice

CLO Magazine

Organizations that have reached higher measurement levels use a blended approach for the various frameworks and find ways to customize evaluation. The company organizes its learning evaluation data using a tool, Metrics That Matter, from KnowledgeAdvisors that connects directly to its learning management system. Jaime Maas, manager of organizational development at Qualcomm, said its program managers are assigned to specific learning topics.

Demonstrating Your Training Program’s Return on Investment

Learning Wire

As distance‐learning programs grow ever more popular, companies have discovered that technology‐ driven learning can be a critical tool to communicate new information and regulations with far‐flung workers, and to help businesses that are experiencing significant growth get employ- ees quickly on the same page. We address these issues, as we discuss return on investment (ROI), and the importance of establishing clear business needs and program objectives.

ROI 43

Is this thing on? Tips for measuring course effectiveness and return on investment

Obsidian Learning

The Kirkpatrick four levels of training evaluation. While later stages of evaluation measure more obvious aspects of quality—such as the impact of the training on the learner—it’s important not to overlook the less obvious factors, such as instructional design or the use of technology. Usability Testing: During course design and prototype development, evaluate the course for usability. Evaluation: The Four Levels. Level 2: Learning Evaluation. Phillips, J.

Ratio 63

Timing is Everything Series: Part 6 Structured vs. On-Demand Learning

InfoPro Learning

A central premise of the learning curve framework is that the training program is well-structured and planned out in advance: the program’s budget, timelines, content, personnel, and administration are all accounted for at the outset. Using a structured approach lets managers maintain control over the training program, evaluate its effectiveness, and budget for it.1 Sources: Phillips, J. Handbook of training evaluation and measurement methods.”

How do you measure how training creates value? – The 7 learning principles

Learning Wire

Let’s begin with the evaluation of learning. Why evaluating? Are there credible measurement methods to evaluate value creation? The Kirkpatrick/Phillips model shows us how and why to assess training outcomes. What should we evaluate? Evaluating value creation involves comparing the cost of an investment with what you get in return. According to Phillips (2012), intangibles are benefits that cannot be given a monetary value. Phillips, 2006).

How do we measure value creation from training?

Learning Wire

In a previous article we looked at the five levels of impact in any exhaustive training evaluation process that reflect the key challenges involved in high-quality evaluation. We also looked at the limitations of evaluations due to non-measurable (intangible) factors. Training evaluation at a glance. Evaluating training is no easy task. If you restrict yourself to sending out a satisfaction form, that’s not going to be enough for a complete evaluation.

E-Learning vs. In-Person: A False Dichotomy

The Performance Improvement Blog

Although this is obviously a marketing piece to promote Citrix Online’s GoToMeeting Corporate product and does have some useful suggestions for selling Web-based programs internally, it sets up a false dichotomy: online vs. in-person. No one can deny that online programs offer a substantial cost savings over in-person training and conferences. In fact, when the argument shifts to results from online programs, the authors can’t help disagreeing with themselves.

Measuring the Effectiveness of Your Training

Training Industry

Something must be done so that business leaders can find useful and pertinent information about the effectiveness of training programs, make better use of the training budget, and obtain the results they seek. Jack and Patti Phillips’ model of evaluation emphasizes the importance of tying evaluation to learning objectives. For evaluation to be truly effective, we must start with the end in mind. While valuable, reaction evaluation is not enough.

Do You Have a CEO-Friendly Scorecard?

CLO Magazine

Phillips is the chairman, and Patti P. Phillips is president and CEO of the ROI Institute. The scorecard should offer insight into how L&D programs contribute to improvements in bottom-line measures. First, consider the inputs — or indicators — such as the number of people involved in programs, their involvement time and the investment in learning (per person). Additionally, consider showing the connection of each program to key business measures.

Instructional Design Basics: What Is ADDIE?

Convergence Training

ADDIE is an acronym that stands for each of the steps of the model–analyze, design, develop, implement, and evaluation (or you might see it listed out as analysis, design, development, implementation, and evaluation). Evaluation. Evaluate or Evaluation.


Here’s the latest thinking on measurement, analytics and reporting

CLO Magazine

Third, programs and computing power are facilitating the use of microdata, which simply means that data are available at the individual level for further analysis or for follow-up with the employee. Ken Phillips, founder and CEO of Phillips Associates, (not related to the ROI Institute’s Patti or Jack Phillips) is doing some exciting work using microdata to reduce the scrap rate and increase application. It is an exciting time to be a learning professional.

A Legend’s Look at Learning

CLO Magazine

Phillips is the chairman, and Patti P. Phillips is president and CEO of the ROI Institute. To see learning as an investment, strategic, expensive and important programs must add business value. This pushes the evaluation of learning to the impact level with a clear connection between learning and business impact. When programs are designed to drive the business measures, they deliver results, removing the fear of a negative ROI. Jack J.