Learning initiatives fuel strategic progress. Leaders need actionable, timely information about program impact to keep their goals on track. However, many organizations lack effective learning evaluation programs that could provide these insights. Of the 556 respondents to TiER1’s 2023 Learning Trends Survey, just 5 percent report that they generally use research-supported evaluation practices, while 64 percent feel unable to do the kinds of learning evaluation that they want to do.
This is the story of how a Canadian energy company developed both the capabilities of their learning team and a suite of powerful evaluation tools in less than nine months.
Breaking ground
In mid-2022, the learning team at this company approached TiER1 for help evaluating a key program intended to provide foundational training for shovel supervisors in one of their mines.
To get the most from the partnership, we adopted a hybrid model where the company’s team could learn from and work alongside Will Thalheimer, an internationally recognized learning evaluation expert and author of an award-winning book on learning evaluation. Through this hybrid model with TiER1, the team would come away with several things:
-
- Actionable, high-impact insights to improve the pilot course
- The skills to repeat this process for other courses
- Valuable experience with this learning evaluation approach to share with others across the company
Digging in
To be full partners in evaluation, the core team spent time building shared values and a common language.
We conducted research and interviews with key stakeholders, instructors, and recent learners to better understand the needs of shovel supervisors and the goals for the course. Digging in further, the team engaged in a series of six two-hour collaborative workshops to develop foundational skills in the design of metrics, studies, surveys, and scenarios and to apply these skills to our project context.
We then developed appropriate metrics and measuring tools, using the Learning Transfer Evaluation Model (LTEM) developed by Will Thalheimer to guide our work. LTEM helps teams make decisions about the kinds of data to collect for a multidimensional picture of a learning program’s effectiveness. For instance, to measure job performance we identified key behavioral indicators (KBIs) that would show if course objectives were really being applied on the job.
We planned a time-series study, in which we would compare the data before training and at multiple points after the training, to assess the impact the training had on the learners.
What We Wanted to Measure:
- Job performance
- Task competence
- Decision-making skills
- Learner perceptions of performance impact
How We Measured It:
- Anonymous self-evaluations of KBIs
- Capability checklists
- Scenario questions
- Performance-focused learner surveys
Developing the evaluation instruments was an iterative process, involving multiple stages of team discussion, review, and revision. For example, we validated scenario questions with both experts and non-experts. Listening to experts “think aloud” through their answers helped us ensure that the questions assessed decision making based on the intended learning points. Testing with non-experts helped us identify and remove wording that could make the answers easier to guess.
Staying grounded
The project did face challenges, but the team persevered in excavating insights and expanding their learning evaluation capability.
We ran into obstacles with implementing the time-series study as envisioned, as the number of learners entering the program dropped significantly right before our planned start for the study. Since our sample size was too small to perform meaningful data analysis, we ultimately derived the most reliable insights from research-inspired benchmarking based on our initial interviews and observations.
Despite this setback, we still achieved several important outcomes:
-
- We identified strengths and opportunities for improvement in the shovel supervisor training program.
- We developed evaluation tools that can be used in the future for additional mine training programs.
- We informed the learning evaluation strategy for programs at our company.
- We gained skill and confidence in evaluating learning programs that we were able to transfer to other contexts.
Fueling discovery
Research-based evaluation methods such as LTEM help us leverage learning and performance experiences of all kinds to uncover insights and achieve our goals. This project illustrates how collaboration and learning science can combine for a solution that delivers high performance, develops teams, and drives business results.
Learning evaluation projects comes in many shapes and sizes. Talk to TiER1 about your goals, and we can help design the right solution for your team.
Author Bios
Heather Cole is a Senior Learning Consultant at TiER1 Performance. She believes in building memorable experiences that help people reach their goals and do their best work. With a background in instructional design and education, Heather brings a unique suite of skills and valuable perspective to performance challenges. Outside of work, Heather loves reading, hiking, and acrylic paint pouring.
With over 20 years of experience in corporate, public, and non-profit sectors, Alex Brierley has had the opportunity to develop, design, and deliver learning experiences for topics ranging from systems and software to web design and life skills. Curious by nature, she seeks new ways to help learners understand and apply the principles and skills they need to be successful. Outside of work, Alex loves to paddleboard, ski, hike and ride anything with two wheels from bicycles to motorcycles. She also has a weakness for pizza and electronic music.
