More on Re-evaluating Evaluation – Jack Phillips and ROI

Dashe & Thomson

I have been blogging a lot about Training Evaluation this year—mostly Kirkpatrick , but also Brinkerhoff and Scriven. I just realized that I haven’t included a single word about Jack Phillips , who introduced Return on Investment (ROI) as Level 5 to Kirkpatrick’s Four Levels of Evaluation. My first exposure to Phillips’ ROI—although I didn’t realize it at the time—was through a colleague who introduced me to Kirkpatrick’s Four Levels. Forecasting methods.

The Phillips Model and How to Use It for Evaluating Employee Learning

Designing Digitally

If you're in the business of…well, any business, really…your strongest method of gauging success is by looking at your Return on Investment (ROI). The same principles apply when discussing learning evaluation


Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Employee performance goals; Choosing the right training evaluation model

Learning Pool

Since the four-step Kirkpatrick/Katzell model of learning evaluation was first introduced some sixty years ago there have been numerous revisions and new versions, each of which takes it in a slightly different direction. Evaluation projects don’t all have the same aims.

The Phillips ROI MethodologyTM – Measuring Data at All Levels – Part 5

CommLab India

Part 1 , Part 2 , and Part 3 of this series dealt with the Kirkpatrick Model of evaluating a training program. In the fourth part of the series, we saw how the Phillips ROI Methodology TM suggests that measurements be taken at every level. Methods to measure: This is done by verbally asking learners what they thought of the course, conducting surveys on their reaction to the course. Methods to measure: Measurements must be ongoing and taken over a long period of time through.

Training Evaluation – 5 Best Ways to Evaluate Training Effectiveness and Impact


Training evaluation refers to an attempt to obtain relevant information on the effects of a training program. The primary objective of evaluating any training program is to develop an understanding of whether it has achieved its stated objectives. The Phillips ROI Model.

Learning to Learn from Evaluation of Learning

The Performance Improvement Blog

The Kirkpatricks have four levels, the Phillips have ROI, and Brinkerhoff has the Success Case Method. Each approach to evaluation of training has something to contribute to assessing the impact of formal training on employee learning. However, the value of evaluation is not in the data. The real value is in organizational learning from evaluation. Learning from evaluation depends on asking the right questions.

Kirkpatrick’s Four Levels of Evaluation


It was while writing his thesis in 1952 that Donald Kirkpatrick became interested in evaluating training programs. In a series of articles published in 1959, he prescribed a four-stage model for evaluating training programs, but it was not until 1994, that he published “ Evaluating Training Programs: The Four Levels “ According to Kirkpatrick, evaluating training programs is necessary for the following reasons: 1. Was the method of delivery effective?

What Is Learning Evaluation?


Knowing how to aggregate and evaluate all types of learning is a priority for many learning and development practitioners. Technologies are becoming available that enable L&D professionals to aggregate, evaluate, and measure many types of learning data in one place. Key Learning Evaluation Takeaways. So, what is learning evaluation? We'll explain how you can effectively evaluate your learning and development process from design through implementation.



It was while writing his thesis in 1952 that Donald Kirkpatrick became interested in evaluating training programs. In a series of articles published in 1959, he prescribed a four-stage model for evaluating training programs, but it was not until 1994, that he published “ Evaluating Training Programs: The Four Levels “. According to Kirkpatrick, evaluating training programs is necessary for the following reasons: 1. Was the method of delivery effective?

What’s LEO’s Chain of Evidence Learning Evaluation Model?


It roughly maps to Kirkpatrick’s learning evaluation levels one to four; however, LEO’s model has been developed through practical experience to be more pragmatic in a blended learning context. Learning Evaluation

Evaluating Training – Capturing the Benefits Aspect of ROI

Obsidian Learning

New post Evaluating Training – Capturing the Benefits Aspect of ROI on Obsidian Learning. Training evaluation is necessary and, in many ways, critical to the success of a business. And even if training evaluation is undertaken, it is usually at the easiest and lowest level: the measurement of student reactions through simple surveys or happy sheets. Determining the benefits of the training is more difficult, and involves knowing how the training program should be evaluated.

Watershed's 7 Steps of Learning Evaluation


Out of the dozens of different learning evaluation models currently in practice, which one is right for you? Meet our "super method" for learning evaluation. We've been using Watershed Insights to dig into several models of learning evaluation in light of what's possible with modern technologies such as xAPI. evaluate alignment with strategic priorities. Step 2: Define metrics for effective evaluation. Step 4: Design programs for effective evaluation.

Evaluation and ROI in Compliance Training

Interactive Services

But what about evaluation and return on investment (ROI)? . Evaluation and ROI are important indicators of how well a program is working and how much risk is mitigated. When it comes to evaluating and determining the ROI of your compliance initiative, heed the advice of the popular business book written by John Doerr and measure what matters. Evaluating Compliance Training Programs.

Showing the ROI of key programs on a budget

CLO Magazine

At the same time, you don’t necessarily have the resources to devote to a comprehensive evaluation. For example, a guiding principle in the ROI Methodology states, “When planning a higher- level evaluation, the previous level of evaluation is not required to be comprehensive.”

ROI 84

Is Kirkpatrick’s Model of Evaluating a Training Program The Best? – Part 3

CommLab India

In the first part of the series, I covered the need to evaluate any training program and the basics of the Kirkpatrick model of evaluating a training program. Here’s a quick diagrammatic representation of the gist: The eLearning industry relies tremendously on the 4 levels of the Kirkpatrick Model of Evaluating a training program. Here’s what we know about the benefits of the model: The model can be used to evaluate classroom training as well as eLearning.

Commonly Used Training Evaluations Models: A Discussion with Dr. Will Thalheimer

Convergence Training

And that’s especially true when it comes to issues regarding learning evaluation. We were excited to be able to talk with Dr. Thalheimer about four common learning evaluation models, and we’ve got the recorded video for you below. If you prefer your learning evaluation information in written form, just scroll down for the transcript of our discussion. Many thank to Will for participating in this discussion on learning evaluation and for everything he does.

Three Challenges Faced by Today’s L&D Leaders

Your Training Edge

Rob Brinkerhoff, a professor at Western Michigan University and thought leader in the field, puts that number 80 to 85% (Phillips, 2016)! This is due largely to outdated training methods. In a 2008-2009 study, Jack and Patti Phillips surveyed the CEOs of Fortune 500 and large, private-sector employers. Jack & Patti Phillips, Measuring What Matters: How CEOs View Learning Success 2009). Phillips, Jack & Phillips, Patti. Phillips, Ken.

Is this thing on? Tips for measuring course effectiveness and return on investment

Obsidian Learning

The Kirkpatrick four levels of training evaluation. While later stages of evaluation measure more obvious aspects of quality—such as the impact of the training on the learner—it’s important not to overlook the less obvious factors, such as instructional design or the use of technology. Usability Testing: During course design and prototype development, evaluate the course for usability. Evaluation: The Four Levels. Level 2: Learning Evaluation. Phillips, J.

How do you measure how training creates value? – The 7 learning principles

Learning Wire

Let’s begin with the evaluation of learning. Why evaluating? Are there credible measurement methods to evaluate value creation? The Kirkpatrick/Phillips model shows us how and why to assess training outcomes. What should we evaluate? Evaluating value creation involves comparing the cost of an investment with what you get in return. According to Phillips (2012), intangibles are benefits that cannot be given a monetary value.

Timing is Everything Series: Part 6 Structured vs. On-Demand Learning

Infopro Learning

Using a structured approach lets managers maintain control over the training program, evaluate its effectiveness, and budget for it.1 Sources: Phillips, J. Handbook of training evaluation and measurement methods.” Timing is Everything Series: Intro , Anxiety , Information Overload , Knowledge Formation , Optimum Performance , Growth , Stagnancy , or Decline , Structured vs. On-Demand Learning, ( Download the complete white paper ). Structured vs. On-Demand Learning.

How do we measure value creation from training?

Learning Wire

In a previous article we looked at the five levels of impact in any exhaustive training evaluation process that reflect the key challenges involved in high-quality evaluation. We also looked at the limitations of evaluations due to non-measurable (intangible) factors. Training evaluation at a glance. Evaluating training is no easy task. If you restrict yourself to sending out a satisfaction form, that’s not going to be enough for a complete evaluation.

Are learning objectives really that important?

The eLearning Nomad

3) And they are useful when we evaluate courses: have the learners achieved the objectives we have set for the course? The predetermined objectives approach was first introduced in a school-based setting in 1932 by Ralph Tyler, an influential American educator, specialised in assessment and evaluation. To this extent it is unsuitable as a general purpose model of evaluation for adult learning.”.

Z490 Basics: Your Intro to the ANSI/ASSP Z490 EHS Training Standards

Convergence Training

addresses all EHS training, no matter the delivery method (instructor-led, online, etc.). Section 6, Training Evaluation. A is for Analysis; D is for Design; the next D is for Development; I is for Implementation; and E is for Evaluation. Section 6, Training Evaluation.

Unlocked Learning—Training analytics made easy


Before online training, there were in-office methods that involved presentations, manuals, and in-person training. However, many believed there should be a fifth level of assessment to evaluate the cost/benefit of the training.

Instructional Design Basics: What Is ADDIE?

Convergence Training

One of the things people in this situation sometimes don’t know if that there are processes, models, or methods that already exist that make the process of creating training more orderly, more effective, and more systematic. And even if you ultimately wind up using a different method.


The Essential Guide to Learning Analytics in the Age of Big Data

Lambda Solutions

How to use learning analytics for evaluation. Evaluating Learning Analytics and Measuring ROI. Evaluating Learning Analytics and Measuring ROI. The Kirkpatrick Evaluation Model is the gold standard for assessing online training programs. The Kirkpatrick Evaluation Model.

Make a Credible Business Connection

CLO Magazine

As the value of learning unfolds, five levels of evaluation are possible: reaction, learning, application, impact and ROI, driven by objectives at each level. In the evaluation, you track the impact measure, “sales from existing customers.” When estimates are compared to other methods, such as an experimental versus a control group, estimates are often more conservative because they are provided by people who know their work best, who are actually driving that impact measure.

Learning by Creating: Turning Bloom’s on Its Head

TechSmith Camtasia

Guest post by Library Media Specialist, David Phillips. At the bottom of the pyramid, the most basic skill is remembering; built upon that is understanding; then applying, analyzing, evaluating and creating. Because we want students to learn with mastery, and learning is more important than my method of presentation. If we want graduates to be prepared for the technology-driven workforce they are entering, we have to make best use of those tools and methods in the classroom.

Bloom 86

Unlocked Learning—Training analytics made easy


Before online training, there were in-office methods that involved presentations, manuals, and in-person training. However, many believed there should be a fifth level of assessment to evaluate the cost/benefit of the training. In 2003, Dr. Jack Phillips introduced the fifth level as the return of investment for the training. Being able to measure your team’s training is crucial to its effectiveness.

What is an Instructional Design Model?

CommLab India

An ID model helps to evaluate learners’ needs and design and develop the course based on this. The types of activities and exercises that will be used in the course are planned and how learners are to be evaluated is also determined. Evaluation. ID models help the designer execute the designing of the course using the process or method prescribed in the model. E-learning Design: Instructional Design Models Vs Strategies The Kirkpatrick-Phillips Model – Part 4.

Brewing CALDO: How communications and learning professionals can thrive using DevOps

CLO Magazine

DevOps methods and principles have been shown to be able to ride (and generate) these kinds of waves. Classical systems methods aren’t keeping up. Is the learning and development profession overwhelmed? Odds are, you might be feeling overwhelmed.

Is Your Online Training Working? Here’s How to Tell


There is a method for finding this out called the Kirkpatrick Model for evaluating training effectiveness. Building on Kirkparick’s Model, Jack Phillips added a fifth level to measure the return on investment of training. The secret to evaluating online training is to put all five levels together and make a determination as to whether the training was effective. There are many reasons to start developing online training.

ROI of continuous training: HR Directors’ unsolvable problem?


In the 50s, PHD and former president of the American Society for Training and Development Donald Kirkpatrick described an evaluation model of training efficiency in a 4 levels pyramid: The Four Levels of Training Evaluation. Jack Phillips then completed this model in the 90s in order to calculate ROI, with a 5th layer. Some use the ‘ Method of loci ‘ to improve memorization, others simply follow training programs.

Demonstrating Your Training Program’s Return on Investment

Learning Wire

Measuring ROI: Evaluation framework. Jack Phillips, Chairman of the ROI Institute, developed a five‐ level method that you can use for measuring the ROI of your digital learning course: Reaction and planned action: Measures participants’ opinions of and satisfaction with the learning program as well as their future plans. Each level of evaluation provides important data that you can use on its own to measure the efficacy of distance learning.

ROI 43

Free L&D webinars for December 2019

Limestone Learning

While business measures may improve after a program’s implemented, unless evaluators take steps to isolate the effects of the project on that improvement, they’ve no credible claim to the improvement – leaving stakeholders to wonder, “What caused it?”

Free 70

Free L&D webinars for April 2020

Limestone Learning

Instructor-led classroom training remains the top learning-delivery method, but as the COVID-19 crisis overruns the workplace, the digital transformation of learning must occur immediately. Follow the “workflow learning” methods that are already built into the methods of productive work.

Free 77

Effective Learning – Musings from the UnConference

Learning Cafe

Evaluation and Measurement – Bob Spence. It was agreed that evaluating the success of learning is something we know should be done but, in reality, it’s rarely completed well. The Session participants noted that the most common form of “evaluation” was the completion of “smile sheets” by participants which typically provide some vague and thinly veiled opinion about the training session just attended.