Podcast 50: Modern Assessment Strategies with Maria Incrocci

EPISODE 50 – TOPIC SUMMARY AND GUEST:

As digital transformation digs more deeply into every corner of the training industry, organizations continue to face issues with data analytics – and skill assessment, in particular. How should learning professionals respond?

I can’t think of a better person to discuss this topic than today’s guest, Maria Incrocci, PhD. Maria is VP of Psychometrics at Scantron, where she oversees assessment solutions and more.

Maria and I met several years ago, when she participated in one of our monthly webinars, “Using Analytics to Drive Growth.” At that time, she was Associate Director of Assessment and Examinations at the American Academy of Orthopaedic Surgeons, where she led the development of the Orthopaedic In-Training Examination, along with various self-assessment and specialty topic exams.

Today, we look closer at factors shaping high-stakes assessment and the role of certification in professional growth.

KEY TAKEAWAYS:

• For most people, assessment follows us throughout our school and work lives. But digital innovation is redefining the very nature of learning experiences, and assessment approaches are changing, as well.

• Assessment faces multiple digital-era challenges. For instance, many organizations are still trying to catch up with learners’ high expectations. As digital consumers, we’ve all grown accustomed to advanced functionality and useful information at our fingertips.

• In addition, the need for rigor is especially important in high-stakes assessment, where demonstrating skill is essential. But this adds further complexity, with requirements such as branching scenarios, simulations and expert validation.

Q&A HIGHLIGHTS:

Welcome, Maria. Could you start by filling us in on Scantron as a company?

Sure. Most of us associate Scantron with paper-based tests we took in our academic lives, using number two pencils and little bubble sheets. But the company is actually much broader, offering technology, assessment and survey solutions for business, government and educational institutions.

For instance, for educators, we offer K-12 and higher education assessment solutions. But we’re also well-versed in high-stakes professional credentialing, including licensing and certification.

Many of these clients use assessment platforms along with our psychometric services and test development services to create assessments that help them license and certify candidates for credentials they provide.

I’d like to explore that further. But first, could you explain your role as a psychometrician?

Sure! Don’t make me spell it, but psychometrics is really a science of measurement.

We provide expertise and guidance into the validity and evidence of measurements and how to use that information to create a story for end-users or stakeholders. For example, we work with businesses that want to sculpt organizational hierarchies through employee surveys and feedback.

So psychometricians like to play with numbers. Some might say we’re glorified statisticians. But I think our training lends itself to philosophical and theoretical theory.

Find out how real-world companies are achieving more with learning systems that create business value. Get inspiration from dozens of success stories in our free LMS Case Study Directory…

Great explanation. Previously, you mentioned high-stakes assessments. What are some examples?

High-stakes assessments usually involve some sort of decision-making in a credentialing process. For instance, all individuals who practice medicine in the United States must obtain a license. And they all go through credentialing.

In addition, a very large sector of credentialing now involves certification. In other words, as medical professionals move into a specific area of practice, usually at some point, they’re likely to have an opportunity to become certified in that specialty.

Certification is just another area where individuals can set themselves apart from others in terms of their professional knowledge, skills and abilities.

I see…

The technology world is another example where certification is huge right now. Technology is evolving at a very fast pace.

So, in a very large pool of potential employees, certification is a great way to demonstrate your knowledge and set yourself apart.

We consider these exams high-stakes assessments because they result in some sort of validation with an electronic badge, a national certificate or a certification.

Certification seems to have multiple meanings. As an expert, how do you define it?

You can look at it in various ways.

A certificate could be issued as a one-time thing. You may go through a brief training program and upon completion, you receive a certificate that says, “John Leh completed this program on this date.”

That could be the end of it. You may never practice or use it again.

But certification is really a commitment to that credential and that leg of employment. It is a much broader descriptor because it usually conveys some sort of ongoing expertise.

So typically, when an individual accomplishes certification, you’ll also find maintenance or credentialing that evolves over time.

What kind of organizations provide these certifications?

That’s a great question. It’s everyone, from associations and other accredited non-profits, to businesses that create programs so their staff can earn industry-specific credentials and advance their careers.

Are these organizations prepared for sound assessment rigor?

I think the top challenge is a lack of expertise in how to develop an effective assessment, whether the exam is paper-based or computer-based.

And now, many assessments are done with observations. We use all types of technology to create simulations. And those simulations can be scored in terms of a candidate’s ability to respond and react in certain scenarios.

But resources have become a huge challenge for organizations creating any type of assessment. It’s not just about access to the right kind of expertise, but also about assembling groups of experts who can provide validation for that assessment.

No doubt…

That’s where psychometricians come in. We’re really not the coach of the team. Were more like the water boy or water girl. We’re here to make sure you have what you need because subject-matter experts will drive the content.

A psychometrician can help shape that, and ensure the right processes are followed to assemble your program in a valid, defensible way. This means that, if anyone ever challenges your exam or assessment program, all the underlying fundamentals will be in place.

Makes sense. Do you see additional hurdles?

Other resource challenges usually involve technology. It’s very difficult for an organization to advance its assessment programs to the level we’re accustomed to in our daily lives.

We all expect instantaneous information access, now. And we rely heavily on technology for that. But organizations have needed time to evolve their assessment programs to meet what we experience elsewhere online.

Where does this gap tend to appear?

When measuring a person’s skill, sometimes we can get great information by using what I call unidimensional assessments. These are text-based item formats, similar to traditional school exams. There’s a question or another stimulus, and answers to choose in response.

But in other cases, it’s important to observe someone responding to a situation as if they were actually experiencing it on the job. That can be more difficult to do in a unidimensional type of assessment.

Want a better way to manage the business of learning? Find vendor profiles, reviews, case studies and more in our free Learning Systems Directory…

These new assessment formats must generate an enormous volume of data. How do you use all this data to make decisions about someone’s knowledge and skills?

Definitely, that’s a challenge. Even outside of credentialing, being able to tease through the data for meaningful information is important. A lot of stuff comes along with data feeds. It really becomes a matter of determining what is important to the stakeholder and the end-user.

Here’s a simple example. For some stakeholders, demographics don’t matter. The stakeholder is looking at big data and trends. They want to tell stories from lots of information. Who it’s from doesn’t matter much, because it’s all feeding down one lane.

On the other hand, sometimes it’s really important to know things like, “Who answered our survey?” or, “Who responded to our inquiries about each of our business models?”

Right…

I like to work these problems backward. So, we start by working with stakeholders to determine what they’re trying to glean from the information. We’ll explore how it will help their business, or their decision-making, or the future direction of their programs.

Then we’re able to work backward to determine the most meaningful way to assist the stakeholder from an analytics standpoint.

But you’re spot-on, John. There’s just so much data out there, and you really have to figure out the best way to tease through it and put a useful story together.

What about the technology behind that story? What tools are organizations using to analyze and visualize data? How challenging is this now…?

…FOR COMPLETE ANSWERS TO THIS AND MORE QUESTIONS, LISTEN TO THE FULL 30-MINUTE PODCAST!

Need Help Finding the Best Training Content Business LMS?

Fill out the form below to request a complimentary consultation call with John Leh: