Monitoring and Analyzing Player Mastery, Confidence and Engagement in Drive
Drive monitors and reports on three critical player attributes that work together to provide you with information that influence a player’s performance in Drive: confidence, engagement, and mastery. Mastery and engagement can be monitored from the author’s dashboard, as shown in Figure 1, as well as within various reports available from the Reports section of the authoring site. Confidence and mastery are shown on a variety of reports related to topics and learning objectives. (Figure 2) All three attributes are showcased on the player progress report. (Figure 3)
Figure 1. Player mastery and Player Engagement can easily be monitored from author’s dashboard.
Figure 2. A variety of Drive reports available via the Reports section of the authoring site showcase confidence and mastery levels, enabling you to see where learners are performing well and where they need additional support or more time in Drive.
Figure 3. The Player Progress report shows you engagement, mastery, and confidence for every registered player.
The three attributes are interrelated so understanding each of them can help a manager coach employees or a game author to coach the managers whose employees are participating in the Drive game. Let’s look at each one.
Mastery
Mastery ratings indicate how well players can perform the learning objectives specified in a Drive game. Players always start with a “low” mastery rating. The speed with which they move to a “high” rating depends on how well they perform within the mini-games. They need to successfully master all the content within the games, which typically requires 3 to 5 plays of each mini-game at a minimum. (Exact number will depend on how many mistakes they make and how much content your games include.)
- Because Drive, like all the Knowledge Guru apps, is designed to provided learners with spaced repetition/retrieval practice, the goal is for them to acquire mastery over time – a span of a few weeks rather than within a single session or two. Players’ performance mastery rating will adjust based on their most recent scores on any game they play.
What to analyze related to mastery
A learner’s manager can look at these mastery ratings and gauge how well a learner can perform essential skills or demonstrate essential knowledge in his or her role.
- Mastery ratings that are low indicate that the player has not achieved any of the learning objectives within the game.
- Mastery ratings that are moderate indicate that progress toward achievement is underway. By looking at a player’s report, the manager can see which objectives the player is closest to mastering and which ones the player needs to spend more time with.
- Mastery ratings that are high indicate the learner has successfully achieved all the learning objectives.
Confidence
Mastery is only one part of the performance picture. A learner can have high mastery over content but feel low confidence in using this knowledge or demonstrating a skill. Conversely, a learner can be overly-confident and believe their skill and knowledge set is higher than it actually is. The confidence rating within Drive is designed to help learners and their managers spot discrepancies between confidence and mastery.
Drive has learners self-assess their confidence levels on each learning objective that is part of the Drive game. They rate themselves on a scale of 0 (no confidence) to 9 (high confidence). As learners play the various mini-games that are part of a Drive game, Drive compares their performance to their confidence ratings. If they perform better or worse than their confidence rating would warrant, Drive prompts learners to re-rate themselves to a more accurate place.
A player’s confidence ratings, as well as their actual performance within the games, influences the game content that player sees on each login to the game. Drive prioritizes games and content within games in this order:
- Low confidence, low performance
- High confidence, low performance
- Low confidence, high performance
What to analyze related to confidence
- Is confidence significantly higher or lower than mastery over learning objectives? If there is a mismatch is it due to lack of mastery (which could be addressed by more game play) or is it due to over-confidence with actual mastery lower than the desired target level? When confidence is too high, a conversation needs to occur with that learner to get better alignment between perceived skills and actual skills.
- Is confidence increasing over time? Can you see a desirable uptick in confidence levels as mastery gains occur?
- Does any individual or group have low confidence but high mastery? If yes, a manager may want to have a conversation about what is keeping confidence ratings low.
Engagement
Engagement ratings tell you how frequently your learners are interacting with Drive. There are four categories of engagement. Here’s what each category means:
- Highly Engaged = Accessing 3x/week (total of 15 minutes’ time)
- Moderately Engaged = Accessing 1x/week (total of 5 minutes’ time)
- Minimally Engaged = Accessing 1x/two weeks
- No Engagement = No access for more than 2 weeks’ time
Once players achieve mastery, most are not likely to continue interacting with the game. After a player’s mastery level peaks, that player’s engagement will likely fall to “no engagement.” Your focus is to monitor engagement right after you release a Drive game and make sure it stays in the “moderate” to “high” categories.
What to analyze related to engagement
- Is your engagement level matching mastery levels? Ideally, if mastery is low to mid-level, you want engagement levels to be high or at least moderate.
- If engagement and mastery are both low, what do you need to do to encourage play? What might be hindering participation and execution? Are players finding value in the Drive experience? If not, why not?
Helpful Tip
Make sure automatic email reminders are enabled. To check, go to Implement in the left-hand navigation pane. Select Drive Access. The drop down menu under Automatic Email Reminders should say “Reminders Enabled.” With the auto emails enabled, Knowledge Guru will automatically send reminders to players after extended periods of inactivity.
When is a game “done?”
Learners and their managers can consider a Drive game complete for a learner when the learner’s performance mastery AND confidence are both high. A typical learner can achieve high ratings on both metrics after encountering all the game content, which should require several instances of play. Depending on the number of performance objectives in the overall game experience and the content included within a single mini-game, this should require 45 to 60 minutes of total playing time, which translates into 9 to 12 separate days of logging into the game (5 minutes per day). Users can continue to access the experience after achieving high mastery. They are never cut off from using it as long as a valid subscription exists.
