Why Facts, Evidence & Data are the Only Way to Demonstrate the Impact of Learning

Kevin M. Yates, the L&D Detective who solves measurement mysteries, joins Blake Proberts on the Strategic L&D Podcast to discuss overcoming the perception issue of L&D, transitioning L&D from being reactive to proactive, getting optimal impact by mixing human performance and business performance metrics, and how it’s the volume of content linked to your performance metrics that matter rather than the volume of content overall. Listen to the full episode above or watch below.

Kevin, it’d be great to start with a little bit about your background and what’s led you to where you are today.

Thank you. Thank you. I’m excited to have this conversation. You know, I like to say that, where there is talk about measurement for training, learning and talent development there am I, so I am right where I want to be right now at this moment.

So yeah, I’ve been training, learning and talent development now for a little over 25 years. I am fortunate to have worked in different industries, all in training, learning and talent development. I’ve served in different roles. So I have had roles that were focused on instructional design and roles that were focused on learning operations or learning administration, curriculum development, program management.

And most recently, the work that I’ve done is narrowly focused on what I like to call impact analytics. So my work is really focused on measuring fulfilment of purpose for training and learning. A lot of people call that measuring impact. It’s semantics, I think, on some levels, or maybe it is a little bit more than semantics. But where I am focused, in the work that I’m most passionate about the work, the work I’m most interested in, is using facts, evidence and data to answer the question, did training and learning fulfil its purpose?

You know, I’m known in the, in the global training, learning and talent development industry as the L&D Detective. And I think that that’s because on some levels, I’m a lot like Sherlock Holmes, you know, although he was solving the murder mysteries, on solving measurement mysteries. So that’s, that’s a little bit about who I am, what I do and where I focus.

How does somebody at the ground floor go about starting to measure that impact?

Yeah, great question. And as cliche as this is going to sound, when it comes to measurement, you really do have to start with the end in mind. And so for me, there are a series of questions that need to be answered—not only to design in the best way possible, and to align training and learning solutions with business needs, in the best way possible—but also, so that you’re setting yourself up for measurement success, right?

So those first conversations are really not, you know, what’s the training? What type of training do you need? How long do you need the training to be in duration? What kind of modality do you want for your training? Quite often, and that’s where the conversation starts, right? But for me, if you want to measure successfully and set up yourself, so that you are in the best position possible, to be able to produce evidence for learning, trainings, influence on performance, and behaviour and ultimately business goals, you have to gather information that answers the question, first and foremost, what is your business goal?

“The business goal is not to train X number of people by X date, but rather, the business goal is… how you want people to be able to perform and behave and act towards a… specific business outcome.”

And so the business goal is not to train X number of people by X date, but rather, the business goal is what is the intention for how you want people to be able to perform and behave and act towards a specific business strategy or toward a specific business outcome, right. So I am diving into those conversations that give me insight into performance requirements that give me insight into what are some of the risks to performance. And I’m also looking at all the things that have the potential to drive performance beyond training. I’m also taking a look at what are some of the metrics and measures that the business is using to, to monitor or to measure its own success or its own performance? Because those are the metrics and measures that I believe are training and learning solutions need to be hooked into you.

So to answer your question about how do you start, right, on this measurement journey? You really start by not asking questions about training upfront, but rather asking questions about business needs, performance requirements to achieve what those business goals are. Metrics and measures that the business are using to monitor and measure performance. Those are some of the things that I believe are going to set you up quite nicely for, for your strategy for measuring outcomes and measuring. Does that make sense? Did I answer your question?

So: Starting with aligning your L&D strategy with the business strategy, and reverse engineering where learning fits in?

It is, it is, it is absolutely that and you use the word that is key. And that is critical. And that is alignment, alignment, alignment, right. And so when we design training and learning solutions that are intentionally and specifically aligned to producing certain types of behavioural outcomes or performance outcomes, then we are doing our best service to the organisations that we serve, because we want to make sure that, for what we are creating, what we’re designing, what we are delivering, what people are using, there is that connection between how they will use what they learn in ways that help them use their performance to help the business achieve goals, and to execute a strategy. So that’s exactly it. You’re spot on.

What are some of challenges when you’re trying to take those business objectives and goals and build that learning strategy out?

Yeah, there are definitely challenges, no doubt. If there weren’t challenges, we probably wouldn’t be having this conversation. Because the reality is that as an industry, training, learning and talent development has really struggled with this idea of measurement, right. And I think that there are a lot of reasons for that; I’m going to try to zero in on two or three here.

I think that one of the challenges that we have is the way in which we are viewed by the organisations in which we work quite often, we are viewed as a fulfilment centre, or as I like to say, as like a fast food restaurant, right? Because what usually happens is that someone will come to us with their order for training, right? And that means they come to us and they say, you know, like, one instructor-led, two eLearnings and a side order of virtual training to go. So they give us their order. We go off, and we make our thing and we build our thing, and then we come back, and we give me give them their order, right?

“The biggest challenge is just the perception of who we are, and how we do what we do. And so, if you really want to get to alignment, that’s going to go beyond a training order. In order to get to a performance-based outcome, we have to have a performance-based discussion.”

So the biggest challenge is just the perception of who we are, and how we do what we do, right. And so if you really want to get to alignment, that’s going to go beyond a training order, right? When we are, when we are focused on the end, or specific outcomes, we’re having very different kinds of conversations. And quite often, it’s hard to have a conversation with someone who is not used to having that kind of conversation. So in other words, if you’re used to just giving me your order, and now I come back and I say what, you know, let’s dive a little deeper. Let’s talk about what your needs are. Let’s talk about what the performance requirements are that you have. Let’s talk about what some of the threats are to performance. Let’s talk about what are some of the triggers or activators for performance, including but not limited to training. So now we got to have a very different kind of conversation. And not only are we not used to having that kind of conversation, but our business partners and their stakeholders are not used to having that kind of conversation with us.

So that is one of the things that makes measuring results a little difficult, because in order to get to a performance-based outcome, we have to have a performance-based discussion. And that’s not always easy to do, again, because we aren’t used to having it, and because our business partners and our stakeholders aren’t used to having it with us.

So if I were to think about a barrier, or the biggest challenge, I would say that the biggest challenge is setting ourselves up for success, for measurement by having very different kinds of conversations. I honestly believe that that’s the biggest challenge and then the other challenge that exists is—I’m going to be very transparent and say the reality for measuring the impact of training is it’s difficult to do. It’s just hard. It’s hard work, right? I scratch my head all the time, which is probably why I don’t have as much. And so and so, you know, perhaps a difference between me and anyone else, and what I have in common with other practitioners who are doing the work that I do is that we just dig in our heels, right? So just because it’s difficult, doesn’t mean that it’s not possible. So you have to have a real commitment to finding the answer.

And it can be discouraging, it can be frustrating when you can’t find them. And then that leads to this idea. That is difficult, right. And again, full transparency, measuring the impacts of training, learning is difficult. It is hard work. But it is absolutely possible. So I hope that that answers your question, in terms of where are the challenges of what makes measuring impactful.

What are some of the things that you look for when you do that detective work that you can benchmark yourself and your training against?

That’s a great question. One of the things that I am looking for is data and metrics and measures that give insight into human performance and business performance. The data points are the outputs. The metrics and the measures is what we use to determine the extent to which training and learning [did what] it intended to do. So, you know, depending upon the business and the organisation, you know, there are different metrics and measures that we can use as signals for the extent to which performance has changed, based on training and learning, the extent to which performance is influenced based on training and learning.

So, you know, again, it’s me looking at different human performance metrics and measures that give us a signal about the effect or the influence of training and learning. And then the same is true for business performance metrics and measures, right. So there may be, you know, different types of operational metrics that a business uses, right, in terms of productivity or cycle time or quality. And then we can even talk about different scores over measures in terms of customer satisfaction, and we can talk about sales, or we can talk about volume, or we can talk about a number of different quality metrics or performance metrics that businesses use to get signal for business performance.

“It’s a matter of connecting the dots between… how training and learning influences human performance, and how human performance influences business performance.”

So here’s what I’m saying, right: What I’m looking at first is the measures and metrics, they get insight into human performance. Because ultimately, human performance influences business performance, right? So for me, it’s a matter of connecting the dots between training and learning, how training and learning influences human performance, and how human performance influences business performance.

So those are the key, core critical metrics and measures that I am looking at, I am investigating and evaluating to determine training amongst fulfillment purpose.

How you would advise people to go about looking into those human performance metrics, and taking them into more of a business performance area?

Yeah, what’s worked really well for me is partnership and collaboration, right? Because here’s the reality: Those human performance metrics and business performance metrics, we don’t own. We don’t normally own that, right, as the training, learning and talent development team.

So what is critical and absolutely necessary for success with measuring outcomes, is creating the right partnerships with the owners of those business metrics, the owners of those human performance metrics. That is how I work and that is what makes me successful, because in order to have access to the signals that are going to show training and learning’s influence, I need to work with the people who owns those metrics and who owns those measures.

“If we’re trying to measure impact, and investigate impact, what we’re looking for are metrics and measures to show training and learning’s influence on… human performance, and ultimately, human performance’s impact on business performance.”

And so that may be a relationship that historically, training and learning has not established, right. But for me, those relationships are the most critical, again, because we don’t own those data points. The data points that we own are usually learning analytics, right. And so, you know, learning metrics and learning measures include things like, how many people were trained, how many hours of training that we deliver? What are the different modalities through which we deliver training? Those are learning metrics. And there’s nothing wrong with that, right. But if we’re trying to measure impact, and investigate impact, what we’re looking for are metrics and measures to show training and learning’s influence on performance, human performance, and ultimately, human performance’s impact on business performance.

So, to go back to your question like, how do we do that? We do that by, first of all, having that very different conversation that I described at the onset, and then creating partnerships and collaboration with owners of human performance metrics and business performance metrics, because then we can connect the dots between our learning measures, our learning metrics, and our learning analytics with human performance metrics, and different business performance metrics. Does that make sense?

“[Create] partnerships and collaboration with owners of human performance metrics and business performance metrics, because then we can connect the dots between our learning measures… with human performance metrics, and different business performance metrics.”

It’s typically hard for HR and L&D practitioners to get the ear of those leaders. What ways have you seen work to get that buy in?

Yeah, I think that, you know, it’s classic, right? It’s relationship building. There’s no magic for that. There’s no simple solution for that, right. At the end of the day, it’s all about collaboration, partnership, and relationship building.

So, you know, I have been in situations where I’ve had to reach out to people in teams outside of training and learning. And what I’ve tried to do to bring them along on the journey, is paint the picture for what we are doing, right. And I also tried to paint the picture for where there’s opportunities for collective success. And not just success for the training and learning team, but success for those teams that we are partnering with as well. Because when we can tell a story that shows how training and learning measurably influence human behaviour in a way that ultimately helps people in teams achieve a business goal, then there’s, there’s quite a few people in that story, right? Because what I’m doing is telling stories.

So that story includes the business partners and collaborators, who owns those metrics, that are going to help tell that story. And so what I have found to be successful is the ability to create a win-win situation. And create win-win situations when we are able to articulate, demonstrate and show how working together ultimately helps us all be successful.

That’s, that’s the best way I can think of to do it. Again, it’s, there’s no sexy way to do it. No, you know, no pixie dust, no magic spray. It’s just that basic relationship-building process.

I guess that probably goes back to your first point, when we first started chatting around the perception. It’s like, we’re a team here and we want to impact your performance.

That’s it. When we talk about performance—I love that you said that. It’s not an L&D thing. Performance is a collaborative effort that is centred on people, that helps and support people in being the best that they can be. And that is not just up to L&D, right. L&D is a trigger for performance, but it is only one trigger, it is not the only trigger. So I love that you kind of painted this picture of us all coming together for the benefit of serving people in the organisation and supporting them in the best way possible. And how we support people in the best way possible is with our training learning solutions.

“L&D is a trigger for performance, but it is only one trigger, not the only trigger.”

It sounds like data is fundamental and key to the strategy going forward. How have you seen data used really well the L&D environment?

Yeah, so I’m going to limit the answer to your question to using data that shows training and learning’s influence on human performance and business outcomes. And I’ve talked about that a lot already. But just to bring it home, I believe that the best signal for learning effectiveness is measures and metrics that show how people are performing, acting and behaving in their roles, in their work, day-to-day.

Again, I don’t want to limit performance and behaviour and actions to being solely influenced by training. And there are other triggers, there are other influencers. And we need to be aware of what those are. But, if we’re trying to kind of isolate the impact of training and learning, what we want to be able to do is use metrics and data to evaluate how training and learning specifically has, has changed performance or again, has influenced performance.

So when we talk about data, it’s not like all the data that there is, it’s very specific data that we want to use. And that data—again, starts to sound like a broken record—but that data is focused on business performance, and human performance. And I think that by doing that, we kind of narrowed down what we should be looking at, because again, in any given business, there’s probably tonnes of data that you can look at. What you really want to focus on is data, metrics and measures that reveal the extent to which training and learning is measurably influencing human performance and ultimately, business outcomes.

How have you found the best ways to measure those behavioural changes?

Yeah, and and I’ll also add here, as we’re having this conversation, that I have found that there are times where people use words like behaviour and actions and performance interchangeably, right. And I do the same thing. Actually, I do the same thing.

So for me, behaviour and actions kind of fumbled up to performance, right? So, having said that, your question was, how do you measure behaviour? I’m gonna give you a very simple example, right? Let’s just say that you work in a business where the business is in the business of creating purple widgets, right. So you are on the assembly line, and you are someone who is responsive for painting, the widget purple, right? So you’re gonna go to a training class that gives you instruction on the proper mix of colours to create the colour purple, right. So your behaviour then is using the right types of colours and paints to come up with the colour purple or mixing the proper proportion of different colours to come up with the colour purple, right. So let’s just say that you are inadvertently and mistakenly coming up with a colour that is not purple, let’s say that you’re mixing colours in a way that the colours turned up green. So you’re producing green widgets. Well, you need to produce purple widgets, right. So there’s an opportunity there to change your what your behaviour.

So you go to a training class, and you’ll learn the proper methodology for mixing the appropriate proportions of pellets to create the colour purple. So you go to that class, you go back to the production line. And now you are starting to paint your widgets purple, right? Well, what we can, what we can say—amongst some other data points—and this was just a real simple example. But what we can say here is that one of the things that contributed, if you are successful, one of the things that contributed to you successfully painting your widgets, purple is the training that you completed.

Now another thing that might be supporting you is the coaching and guidance that you’re getting, say, from your manager or from coworkers as well. But that just, that just demonstrates it illustrates the fact that, you know, as I said earlier, training is going to be one of the triggers, but not the only trigger. But this in example, in this example, training is one of the triggers that supports you painting your widgets purple. The other thing that’s supporting you in painting your widgets purple is, you know, coaching and guidance from managers coaching and guidance from peers.

But again, what you can now do is go back to the data that shows what number of widgets that you produced and created off of the assembly line are coming through as purple, and are coming through as green. So we can consider the ones that are coming through as green as your error rate. And we can consider the ones that are coming through as purple as your success rate. So this is an example where we can use an error rate and a success rate as an indirect signal. For the extent to which training and learning fulfils its purpose. Does that make sense? Did that answer your question? I tried to come up with an example on the fly.

How do you decide what learning activities work best to influence some of those human behaviours and business behaviours?

Yeah, so now what you’re talking about is instructional design techniques. You know, in a prior life, I was an instructional designer. And here’s where that alignment, I believe, is critical. And this is where asking those questions upfront, that are focused on performance, are critical.

So if you are clear about performance requirements to achieve business outcomes, along with that, the essence of what you are looking at is the different types of skills, capabilities, again, and behaviours that are required to be successful. So if you know what the required skills and behaviours are to be successful, you then design training and learning solutions that have potential and power to produce those behaviours as outcomes, to produce those skills and capabilities as outcomes.

“If you know what the required skills and behaviours are to be successful, you then design training and learning solutions that have potential and power to produce those behaviours as outcomes.”

So here’s another example, right? Let’s just say that you are a manager of people and teams, and you get feedback from your employee engagement survey that shows you are not leading, guiding and supporting your employees and coworkers’ progression the best way possible. Which says there’s an opportunity for you to improve with supporting employees on the career path and on the career journey. So if we can create training and learning for that, then what we want to do is immerse, I would say, immerse the trainee—the manager being that trainee—in experiences that allows her or him to rehearse the types of conversation with the support employees in their career journeys. We want to engage them in, maybe, some simulation or some role play.

The essence of what I’m saying is, depending upon the desired skill or behaviour as an outcome, we then need to align the different types of experiences and activities that are part of the training, the training experience or learning experience, that is most likely to produce those desired outcomes. And that’s going to vary from outcome to outcome, or requirements or requirements. But the idea is to deliberately align your instructional design methodology with the intention for desired outcomes and desired behaviours as a result of the training or the learning experience.

What do you find is the best method to map or manage content so that it is relevant to learners?

Yeah, you know, again, I’m gonna sound like a broken record here. And on some level, that’s a good thing, because the story is the same, it doesn’t change, right.

So, you know, if you have, I don’t know, a library of training and learning of, I don’t know, what maybe 100 courses, or a training library of 1000 courses, you’re going to be most successful in that content is going to be most relevant and most valuable when you map it back to skill requirements, capability requirements, and performance requirements, right.

“You’re going to be most successful [with libraries of] content… when you map it back to skill requirements, capability requirements, and performance requirements.”

So if you have a library of 100 courses, it helps employees be better able to assess what is best for them, and what supports them the most, when they have a clear view into a map, if you will, that connects specific behaviours, skill requirements, capability requirements, to specific courses and specific training that is designed to support or produce those behavioural or skill or capability-based outcomes. , right? So again, for me, the answer is simple. It’s all about alignment. It’s not about guidance, it’s all about direction. So if you have a library of 100 courses, or a library of 1000 courses, it’s going to be most effective when you can show employees the journey, the connection, the path, and how specific training and learning is mapped back to specific skill and capability requirements.

So in other words, you know, if I am interested in training, training and learning that helps me be a better communicator, then I need to see what courses and training do we have in a library that are available to me, that helps me be a better communicator. So for me, that is the best way to map that is the best way for alignment. I hope that answers your question.

Prior to that, how do you uncover what what capabilities and skills that you need?

It goes back to what I said earlier, man. And again, I know I’m sounding like a broken record here, but the story doesn’t change. It just really doesn’t. And you uncover what those skill and capability requirements are through discussion, through talking to business, business people. Talking to maybe even HR stakeholders, those who are close to the work, right? How about we’ve talked to people in teams that are actually doing the work day to day, because they’re doing the work and they have a perspective that is unique and very close to the work.

So your question is, well, how do you uncover and how do you know what the performance requirements are? Well, if you know what the job is, right, if you know what the job requires—and, and as a matter of fact, quite often one of the places where I recommend people start, if they’re trying to get a good grasp on performance requirements, I say start with the role description, or the job description. Because that is hopefully the foundation for what the role or the job is built on, right. So if you can go back to a job description, or a role description, that should [make] planning out what the capability, experience and school requirements are to be successful.

“How do you know what the performance requirements are? Go back to a job role description, that should [make] planning out what the capability, experience and school requirements are to be successful.”

It’s not always easy. Again, I want to go back to that idea. Because sometimes, even when we have conversations with our business partners and our stakeholders, and we asked them to describe what performance looks like, when it shows up, quite often we hear people lead with well, people need to know and people need to understand. Well, I didn’t say well, how does knowing show up? Right, and how does understanding show up? So people know something, what does that look like? If people understand something, what does that look like? And then a light bulb goes off, and it’s like, oh, that’s what he means. And so when we can have that kind of compensation, we are better able to get very narrow and specific about what performance looks like when it shows up, day to day, in real time on the job.

You’ve worked across an array of areas within the L&D landscape, you’ve been in the instructional design side, you’ve been in the business strategy side. What are some key lessons that you’d share with an up-and-coming L&D professional?

Yeah, wow. So you’re gonna make me reflect here. Relative to the work that I do, I think one of the key takeaways, and I see this, which is why I’m grateful for this opportunity to, to answer this question. What I see quite often is a struggle where training and learning teams are trying to measure the impact of efforts, right? Quite often, I see we’re training and learning organisations are trying to measure the impact of L&D.

And I’m, I’m challenging that I’m like, so when you say you want to measure the impact of L&D like exactly what do you mean, when you say that right? I believe that we can measure the impact of our individual training and learning solutions. Right. But when we talk about measuring the impact of L&D we’re talking about the extent to which we can show, through facts, evidence and data, how training and learning is contributed, how rather L&D, as a function, is contributing to business goals and contributing to business success. That’s a little different. Right.

So, for me, I think it’s important for training, learning and talent development teams to be very clear about the difference between measuring the impact or the influence of our unique and different training and learning solutions, compared to measuring the impact of L&D. I’m not so certain that you can measure the quote unquote, impact of L&D singularly, but I do believe that you can measure the impact of our unique training and learning solutions, and then that ultimately shows how we are contributing to business goals, and how we are contributing to business success. So I think that there’s that, there’s that struggle there, from a mindset of well, what are we measuring here? Right, are we measuring the impact of L&D, or are we measuring the impact of specific training and learning solutions?

“Training, learning and talent development teams to be very clear about the difference between measuring the impact or the influence of our unique and different training and learning solutions, compared to measuring the impact of L&D.”

So I would suggest and recommend that ultimately, when you are measuring the effectiveness of individual training and learning solutions that, ultimately, then leads to answering the question, what is the impact of L&D versus trying to come up with this singular metric, measure or data point that gives insight into the impact of L&D. I’m not so certain that you can do that. You know, I’ve seen people try to do it with things like Net Promoter Scores. I don’t know that that tells us anything. Right. I do believe that we can get signal for how training and learning is measurably contributing to business goals when we take a look at how the training and learning solutions that we offer and that we provide are influencing people’s behaviour, influencing people’s performance and ultimately influencing how that empowers people to achieve business goals.

So, for me, that would be a big takeaway. And again, to summarise, the big takeaway is be clear, be very clear about the extent to which you are trying to measure the impact or influence of training and learning solutions versus the extent to which you’re trying to measure the impact of L&D.

My last question for you, Kevin, is where can people find you?

I’m all over the place, man.

Well, I am on social media. So you know, people can certainly join me on social media platforms. So I’m on LinkedIn, I’m on Twitter, I’m on Facebook, I’m on Instagram, but then definitely check out my website. And my website is kevinmyates.com.

And as you mentioned, you know, on my website, there are a lot of resources. You’ll find the articles that I’ve written over the years about measuring the impact and influence of training and learning. And what I am most excited about on my website is the L&D Detective Kit, which is a free resource. That’s free dollars, and zero cents. And the L&D Detective Kit has has really received some great reviews and feedback from the community and industry about the extent to which it helps people on the impact measurement journey. So certainly go to my website, kevinmyates.com. Take a look at the free online detective kit. Take a look at the articles that are there. Take a look at the videos, I believe, that there’ll be found to be helpful.

This is a transcript from the Strategic L&D Podcast, where we venture through what key L&D opinion leaders are doing today to ensure they’re delivering a strategically impactful L&D function. If you want to stay up to date with our latest releases, subscribe to our learning and development podcast. We’re on most common podcast platforms, including Spotify and Apple. You’ll also find us in video form on our YouTube channel.

Related Reads on This Topic

How the ‘Mess’ of an L&D Problem Provides Structure & Drives Outcomes

Learn why you should spend more time diagnosing learning needs and how to spend L&D budget more effectively…

Read More

Why Starting at the End of L&D Strategy Pays Off

Find out how lean L&D budgets often drive innovation, why self-directed learning won’t net results, and starting with strategic impact…

Read More

How to Double Your L&D Budget by Aligning with Business Needs

Align learning with performance in stakeholders’ minds and explain the risks of non-conformance to secure the budget you need in L&D…

Read More