The Roles and Design of Tests in Online Instruction

Recently this thought hit me: “Who cares how employees learn it, so long as they can demonstrate that they do know it – on a test?” This notion may be my learner-control subconscious rebelling after a series of mandated minimum-student-contact-time online courses my team had to develop. In order to ensure that each learner met the minimum prescribed contact time, we felt compelled to force them to see most of the screens as well as listen to the entire audio on each of those screens. (We did try to make it interesting by making the courses scenario-based.)

Editor’s Note: Parts of this article may not format well on smartphones and smaller mobile devices. We recommend viewing on larger screens.

So, on courses that don’t have such mandates, can we go to the other extreme and give learners total control over what content they view, including the option of whether they even view the content we provide at all? Can they just prepare on their own and test out?

Perhaps … it seems that it depends on a few things:

• Are the tests valid? Do they measure the right knowledge and skills?

• Are the tests sufficient? Do they cover enough of the material so there’s a high correlation between test success and job performance?

• Do employees really want to follow self-directed learning or constructivism approaches, or do they really want someone to guide them?

• What about mastery learning, a practice shown to be among the most successful learning strategies of the past several decades?

There is an abundance of research on the above theories in academic settings, but not so much in workplace settings.

In my own experience, employees like it somewhere in the middle. In preparing this article, I researched a variety of terms while looking for one that described the concept I was after. I believe the concept of “mastery learning” comes closest. With mastery learning, the learners receive the resource materials but otherwise expect to study on their own, progressing to each successive level after passing the prescribed test(s).

Many schools use this practice. I learned of two examples of mastery learning in the workplace. In one, a new employee had to take a comprehensive test after one year. If she passed it, she received a sizable pay raise. If she failed, she lost that job. In another example, the setup was the same but after the employee passed the test the company had no money for the pay raise, so the employee left on his own. I plan to do more research on the use of this practice in the workplace.

I suspect mastery learning holds the most promise for enlightened workplace learning, but no matter where we are on this spectrum, there is a need not just for valid online end-of-course tests, but for tests with slightly different purposes, as well. That’s what we’ll cover in this article.

Tests: macro view

Where do test questions come from? If you immediately said, “the learning objectives,” then you get an A+. Before we look at construction of questions themselves, let’s look at tests from a macro viewpoint.

The role of tests

What role do tests play in online workplace learning?

“…our first concern with assessment is knowing what students know.” (sic) [Stoner and Smith, 2007 – see References at the end of this article]. But I have come to view tests as having much broader application than merely assessing knowledge, especially in Web-based training (WBT) where there’s no instructor to moderate the learner’s learning path, conduct role plays, offer encouragement, or debrief a test. We can use well-designed test questions diagnostically to moderate the learner’s path, reinforce learning, and help the learner gain new insights; and, together with appropriate feedback, they can even encourage and motivate the learner. Online assessments can play an economic role as well, by saving employers time, money, and effort. [Greenstein 2010]

Before we go on, let me set the scope of this article. The goal of workplace learning is job and organizational performance. Usually such performance involves some levels of knowledge and skill that we can’t teach or measure on a computer. We often use online learning (or Web-based training) for lower-level cognitive skills, but we’ll see how you can use it at higher levels of Bloom’s taxonomy, too. Other types of learning such as authentic assessment must and do take place in the workplace, and we may measure these in other ways. But although some of the principles mentioned here apply equally to paper and pencil tests, the thrust of this article is on using online tests, or automated testing as some would call it. [Brooks 1997]. We will also examine how to enhance instruction by what we do after the learner takes a test, and we’ll look at some authoring and learning management system capabilities and limitations.

Tests can measure knowledge at the outset of a course or program (pre-tests), within it (embedded), and at the end (post-tests). Tests can be summative (scored) or formative (used to adjust the learning path or approach). Properly set up and combined with the right tone of feedback or remediation, a test can encourage the learner and build self-confidence. We frequently embed self-check questions in our online courses that look and feel like games, and our employees love them! So much so that we even use them in some instructor-led group settings to add some variety and fun. I believe there is a key difference between tests used in a school setting and those used in the workplace. In school, the student’s goal is a high test-score, yet the instructor often wants to force a bell curve. Some would likely accuse an instructor whose students all get A’s of teaching the test. Conversely, in the workplace the goal is mastery – by everyone. Why? Because we want employees to know how to do their job. They may not all achieve 100%, but we hope to skew the curve as far towards 100% as feasible – learner ability, not intentionally difficult test design, being the variable.

Test types

Now let’s look at the pros and cons of using the various test types. Throughout this discussion I’ll use the word “course” to signify any designated content no matter how structured or unstructured.

Pre-tests

We give pre-tests before the learner embarks on a course. Here are some reasons commonly cited for using pre-tests [Berry, 2008, and Berman, 2001]:

• To find out what the learner(s) already know about the subject. This may be useful to:

  • Establish a baseline to judge the effectiveness of the training or, at least, the amount of new knowledge the learners gained

  • Divide a group into learning groups (pertains more to instructor-led, but there is a parallel for WBT as we’ll see later)

  • Help the learner herself know which material to zero in on. [Stoner and Smith, 2007]

• Provide a preview of the terminology and content as well as the depth of coverage.

• Motivate the learner by virtue of the preview.

• Set the learner’s expectations, especially if you plan to use similar or identical questions on the post-test.

The above reasons notwithstanding, in my opinion there is a single compelling time to use pre-tests, and that is when you expect a significant amount of prior knowledge among the learners and you will adjust the learning path to accommodate that knowledge.

One big downside of pre-tests is that they are, after all, a test. A significant percentage of learners already suffer test anxiety, so why exacerbate that fear. That can hardly be motivating.

I think there are more personal, congenial ways to motivate or give a preview than through a test. Borrow from fiction and establish suspense at the outset. Here is an example of an opening line you could use in a sexual harassment prevention course:

What if you overheard your neighbor at work making a lewd comment to another co-worker. Should you keep it to yourself, or tell someone? If you tell someone, are you putting yourself in jeopardy of retaliation?

Post-tests

The main reason for using a post-test is simply to measure the learning outcomes. Can the learner demonstrate mastery of the subject matter as defined by the course objectives? The post-test must evolve from solid and thorough task analysis in the early stages of instructional design. In fact, many instructional designers develop and finalize the learning objectives and then go straight to designing the post-test. Only then do they develop the actual instructional material.

I think a key to a successful post-test is what you do with the results. For one thing, the test review itself, even online, should serve as the capstone to that chunk of learning. And you should use item analyses to assess the effectiveness of the instruction and the test.

Embedded tests

If pre-tests are used for tailoring the learning path and post-tests are used solely to measure knowledge, then what is the role of embedded tests? They can do the same things the other two test types can do, and more. Embedded tests can:

• Perform formative assessment, or diagnostics, throughout the course, tailoring the delivery to the learner’s existing knowledge on a topic-by-topic basis instead all at once up front.

• Perform summative assessment, but at intervals throughout the course instead of all at the end. An embedded test can provide an alternative to a final exam. While discussing compliance training at a conference workshop a few years ago, most of the participants agreed it was sometimes more advisable to force learners to get each embedded question correct before they could move on than it was to use a scored final exam. The rationale? In effect, every learner has to get 100% in order to get credit for the course.

• Provide practice for and/or a preview of the final test. Such a preview can reduce anxiety for the final test and can thus build the learner’s confidence, especially with the use of conversational, reassuring feedback.

• Encourage and motivate. You can use short embedded tests along the way to reassure the learner in building-block fashion that he or she is acquiring and applying the knowledge at an acceptable level.

• Provide a scenario-based learning experience.

  • In a course on sexual harassment prevention, we used multiple-choice questions in each scenario to simulate a conversation. [Dickinson, 2009] It wasn’t as open ended as a live roleplay, but it engaged the learner more than a straight presentation would have. It also enabled us to address the “gray area.” For example, an employee has options on whom to approach with a harassment complaint – the instigator, their manager, or HR – and there are pros and cons to each option. So we allowed multiple correct answers (thus letting the learner proceed in the course) while the feedback acknowledged the points to consider with each choice and why one option may be the preferred choice in a given situation.

Tips on overall test design

In the instructional design process we meet with executive sponsors and SMEs, and often use our own experience too, to ascertain what it is the course should cover. We then craft the specific learning objectives. These are critical to everything that comes downstream of this step.

To begin with, it’s tempting to write objectives in terms of actual job tasks. That is a great starting point since that’s the ultimate goal of workplace learning. But we actually need to write the real learning objectives in terms of behaviors the learner will demonstrate in the learning environment. Often, the latter is a proxy for the former. We must spend time crafting precise learning objectives based on what the learner will do in the online course. For example, they will usually select, not edit or describe; identify or arrange, not analyze; sort, not create.

Notice it’s the verb we’re talking about here. We need to select verbs that represent the actual behavior the learner will be exhibiting in the online test.

It doesn’t stop there. Not all objectives are created equal. Whether we care how the learner acquires their knowledge or not, as I posed in the introduction, a test must cover the most important content. And what content is that, since a test can only cover a sample of all the content in the course? A simple rating system can help the most important objectives percolate to the top. [The eLearning Coach, 2009] Note that while we write the objectives at the course level (actions the WBT learner can actually do on the computer), we assign the ratings from the perspective of the job itself. Here is an example using two objectives and three qualities on a scale of 1 to 5, with 5 being high.

 

 

For online tests I would add the quality of application vs. facts, since “…the most important aspect of workplace learning is usually the application of knowledge in the real world.” [The eLearning Coach, 2009]

Here are some general tips for designing tests:

1. Understand the individuals for whom you are developing the test. What is their prior knowledge, including familiarity with technical terms and acronyms?

2. Rate the course objectives to know priorities and cognitive levels the test must address.

3. Identify the kinds of questions you’ll use to elicit the behavior suggested by the course objectives.

4. Determine how you will you sequence the questions. This may be iterative as the test design unfolds.

   1. Logically by content flow

   2. Simple to complex

   3. By question type with introductory instructions for each type

5. Have the questions reviewed by an expert who understands nuances of the job. As with a novelist who captures regional speech patterns in their dialogue, we want both the technical content and the context to ring true to our learners’ ear.

6. Pilot test with members of the target audience to check for precision and clarity of wording

   1. Do the incorrect choices sound plausible?

   2. Is the correct choice unambiguous?

7. Monitor test item performance once you implement it.

   1. Conduct an item analysis to see how learners responded to each question.

   2. If any questions had high miss-rates, assess whether the instruction or the question itself may need improving.

8. Tests must be valid and reliable. Here are some steps to ensure validity. [The eLearning Coach, 2009]

   1. Validity means we are measuring what we think we’re measuring. The best way to ensure this, as we’ve seen, is to first do a thorough task analysis, then create test questions directly from the learning objectives, verb-to-verb.

   2. Using a weighted rating scale for learning objectives, be sure the test covers the most important content. Strive for application questions, not mere selection of facts. Scenario-based items are useful ways to develop application questions.

   3. Continue to monitor learners’ performance over time using item analyses, and adjust the course or test questions as seems prudent.

Above all, students must perceive the test as fair and equitable. Does it address the topics in the test that the learners expected? Do the test questions correctly measure the objectives? The single best way to assure this is to verify verb alignment between the objectives and associated test items.

Here are some other suggestions to instill the feeling of fairness.

As with all other instructional material, pilot test the test to ensure it performs as intended and to identify any misleading terminology. Start with members of the training staff, and then try to enlist a few SMEs. Run the course and test with a few members of the target audience if possible.

As soon as you start running the test “live,” run item analysis reports if your software can do so. [Svinicki, 1999] These reports show, for each question, how many learners got it right and how many got it wrong, and which answer choices they selected. Without this feedback loop you are flying blind. Item analysis reports help you identify the questions that learners missed too frequently and which answers learners are selecting. This data enables you to check if some instruction was lacking or if the question needs rework. [Svinicki, 1999] The ideal item analysis groups the results into the highest scoring third of the group, the middle third, and the lowest third. This method gives you additional insight. If mainly those scoring high are missing the question, then something about the question is probably misleading. If mainly those in the bottom third are missing the question, then it appears to be discriminating as intended. Don’t take too much comfort in that, though, since we’re talking about job training here where we hope everyone does well on the test. It may be a well written test question, but perhaps the associated material needs amplification so more learners “get it.”

If you give the test in a classroom, even an online test, then the instructor may be able to debrief the test in a live group setting. One method for doing this is to first debrief the “high miss” items, finding out why learners chose the answers they did; then let learners ask individually about particular items they missed or want clarified. Finally, if poor questions or misinterpretations warrant, the instructor can consider adjusting learners’ scores. [Hitchcock]

What if you have the test reviewed online and not in a group setting? Look for creative ways to emulate that last learning opportunity. Let’s generate some eLearning Guild chat on this challenge!

What kinds of questions are suitable for online tests?

Questions spring from verbs, verbs spring from objectives, and objectives spring from task analysis and the cognitive levels required by various tasks. So let’s start at the beginning with a modern version of Bloom’s cognitive taxonomy. [Anderson, 2001 and Forehand, 2005]

Looking at Bloom’s updated taxonomy, here is a comparison of cognitive levels and the kinds of behavior that we can, and cannot, evaluate with online tests. Pay particular attention to the verbs in column 3; these suggest actions (questions) that would demonstrate a learner’s knowledge online:

 

 

Additional guidelines for writing online test items

1. Strive for a variety of cognitive levels. [Overbaugh and Schultz]

   1. What type of hammer is this? (Knowledge and Concepts)

   2. How do you use this type of hammer? (Procedures and Performances)

   3. Which type of hammer should you use in this situation? (Problem Solving and Reasoning)

   4. Avoid trick questions. What makes it a trick? Misleading emphasis (decoy), unparallel structure, or artificially making it hard with implausible distracters.

2. Write independent items. Each question should be answerable without any other information.

3. Answer Options

   1. Randomize options unless logical order is required.

   2. Ensure all answer options are mutually exclusive.

   3. Ensure that answer options use parallel constructs, verb tense, etc.

4. Bias Guidelines

   1. Avoid gender-specific references

   2. Avoid personal or place names

   3. Avoid regionalisms

Notes on specific item types

People build careers on the subject of training evaluation. This section summarizes some of the more salient guidelines I have gleaned from personal experience and the literature about specific item types.

True/false

In a word: Don’t. It is hard to write an isolated, discriminating T/F question whose truth we cannot debate in some way. Stronger students, especially, may see multiple perspectives that could make the statement untrue. [Dewey 2010]

Multiple choice

Multiple-choice questions are the workhorses of online tests. They sometimes get a bad rap as being “multiple guess” or superficial, but when designed properly they can assess higher-order cognitive skills.

For asynchronous online learning, essay and short answer tests are somewhat impractical although they are essential for cognitive skills like “describe,” “analyze,” and “argue [for],” which require live or written assessment.

Multiple-choice questions have many advantages. Because the learner can read and answer them rather quickly, they can cover a large sample of the material in a given amount of time. They can assess a wide variety of learning objectives. You can score and analyze them easily, and when given online, they can efficiently test small chunks of material. With careful design they cannot be readily guessed and thus require the learner to truly comprehend the material for both factual recall and application-type questions. They preclude some of the bias that may occur when grading short-answer or essay questions. [Dewey 2010]

Multiple-choice questions are fairest when they ask for a single correct answer. Multiple correct answers, especially if the learner is not told how many to select, are just too difficult for learners in the workplace setting.

Here are suggested tips for writing multiple-choice questions. [Adapted from Svinicki 1999, Bothell 2001, Dewey 2010, and Mandernach 2003]

1. The stem should generally be a complete sentence that asks a question. Use stems with incomplete sentences sparingly if at all.

2. If you would otherwise repeat some words in each answer choice, put those words in the stem.

3. Use plausible distracters

4. Avoid giveaways through grammar or clues from other questions.

5. Use negative statements with great care, e.g., “Which of the following is NOT…” Avoid double negatives altogether.

6. Number of distracters:

   1. Four is most widespread

   2. Five may be used to further reduce the chance of guessing correctly

   3. You may use three with almost the same testing ability as questions with four distracters according to a U.S. Air Force Academy research paper I saw many years ago and couldn’t locate for this article.

7. Choices should generally have these characteristics:

   1. A clearly correct and unambiguous answer. That is not to say it can’t be challenging. (See what I mean about negatives?)

   2. One or two plausible but specifically incorrect choices. If there are predictable wrong answers, e.g., in a math calculation, include them. If a learner misses such a question you have just created a teachable moment where the feedback and, if used, additional remediation can clear up the misunderstanding. And it goes without saying; you should have clearly addressed such predictable misunderstandings in the original instruction.

   3. One clearly wrong choice.

8. Remember through your task analysis and objectives to write questions at all cognitive levels. Example:

   1. (Recall) Which of the following is the harassment hotline phone number?

   2. (Application) If you wanted to stop a fellow employee who continually makes lewd comments to a timid co-worker, which of the following should you first notify?

      1. Your manager

      2. The VP of Human Resources

      3. The harassment hotline

      4. The person making the comments

   3. Keep option lengths similar.

   4. Vary placement of the correct answer.

   5. Be grammatically correct.

   6. Avoid giving clues to the correct answer, either within the question itself, or between questions.

   7. Give clear instructions.

   8. Each question should cover a single, clearly defined problem, and you should state the main idea as part of the question, not in the answer choices.

   9. Avoid using “All the above” or “None of the above” structures

   10. Don’t use a multiple-choice question if another question type is more appropriate for the behavior. For example, if it’s to solve a math problem, use short-answer (calculate) vs. multiple-choice (select).

Finally, I want to illustrate how the combination of thorough task analysis and test design can help you create multiple-choice questions that test at higher levels of Bloom’s taxonomy – commonly referred to as “application.”

When developing online courses for our compliance-training program, I found I had to probe a little to get beneath the surface, that is, to reach higher Bloom’s levels. As I sought insights from managers in various departments, they readily recited what I would call the “black and white” aspect of various compliance rules. For example, in our business there are federal laws against making unsolicited telephone calls, or “cold calls,” to potential Medicare customers. Everyone knows that, and everyone can recite the rule. Even when I asked for examples of what a cold call would look like in various parts of the business, I tended to get straightforward, low risk examples. For example, all our employees know they can’t just pick up the phone book and start dialing.

A refresher course that only covered the superficial knowledge would be b-o-r-i-n-g. So I have learned to ask questions like, “Are there any frequent dilemmas or gray areas that your people deal with?” or “… situations where the right answer isn’t clear?” Bingo. Helping learners gain insight for handling such situations not only makes the learning more relevant and valuable, it also enables us to craft questions at the higher cognitive levels. So, for example, instead of merely asking the learner to select the correct rule from a list, we can ask them to select the correct behavior in a sensitive situation. They’re still selecting from among choices on a multiple-choice question, but now they’re not merely picking a sterile definition or fact, they’re applying their knowledge in a realistic situation.

Short answer or completion

1. If using incomplete statement items, omit only significant words.

2. Be sure to leave enough clues so a student who knows the material can supply the correct answer.

3. It is best if the fill-in blanks occur at the end of the statement.

4. Avoid verbatim quotes from the instructional material. Paraphrase.

5. Decide how important correct spelling is and program the answer-checking algorithms accordingly.

Matching

1. Within an item, use homogeneous premises (list on left side) and homogeneous responses (list on right side). Otherwise, responses that clearly don’t fit reduce the number of plausible choices.

2. Use clear directions; what is the basis for matching?

3. Ensure all the choices fit on one screen.

4. The list of responses should be relatively short.

5. You can use responses multiple times. Check to see if your authoring system will permit uneven lists. If so, use an uneven set of premises and responses so the last pair or two aren’t giveaways.

Drag and drop

1. Use drag and drop to identify components, to place steps of a process in proper sequence, etc.

2. Drag-and-drop questions are useful for objectives like “compare and contrast,” “sort,” “arrange,” etc.

3. Drag–and-drop questions involve different thinking than multiple choice questions plus some physical action, so they may engage the learner a bit more and, at a minimum, offer some variety.

4. One caution: determine the reliability of your authoring system’s ability to correctly interpret correct placement on the different browsers and computers in your environment.

E-mail

• For small groups, and where necessary based on the learning objectives, you can have students e-mail their short answer or essay question responses to the instructor in addition to completing the online questions.

Feedback and remediation

Since an instructor may not be present when learners take an online test, I think we should put extra care into how we design the test, how we introduce it to the learner, and how the learner gets feedback. Take the misinterpretations that sometimes occur with e-mail, add the emotion that surrounds a test, and it is all the more important to establish and maintain a supportive tone throughout the test experience, perhaps more so because it is workplace learning, i.e., fellow employees.

Feedback by test type

Pre-tests

Given the purpose of a pre-test (diagnostic), feedback for each question may succinctly tell the learner if they answered correctly or not along with a brief explanation. In the spirit of keeping the tone supportive, I suggest the learner be told what “decisions” are being made about their learning path and why, and that they be given the option of seeing even those sections in which they tested out.

Post-tests

Assuming you are scoring summative tests, the feedback should clearly explain if the learner was correct or not. For correct answers, sometimes a short explanation is included. I like to include a brief rationale statement on questions involving anticipated misconceptions to reinforce the underlying principle.

For wrong answers, I prefer to give context-specific remediation that explains why the correct answer is correct along with a brief explanation of why that particular choice is incorrect. You can perhaps offer branching to a more robust explanation for those learners who want it.

Remember to be encouraging and motivating as well as technically correct in this, and all communications, within the test.

Embedded tests

Follow the guidelines above if the embedded test is a pseudo pre-test or post-test. As mentioned earlier, embedded questions provide a great opportunity to address the “gray area” with contextual feedback that acknowledges the pros and cons of the real-world application, and to emphasize the preferred choice. If you do not score these embedded questions, and especially if you don’t weight the choices, then, in my opinion, they provide a wonderful learning venue without the emotion involving partial credit scoring.

Course authoring and learning management software

We have used a few different tools in our course development efforts. Some of them offer basic libraries of standard question types with modest ability to customize the way they work. Our most robust authoring system enables more programming, so we use it when we need that flexibility.

Explore your tools so you know their capabilities. Here are a few issues we have run across, beyond the basic issue of screen design, audio, and graphics. I don’t mean these to be selection criteria for choosing a tool — rather, capabilities and limitations to be aware of when designing your tests.

• In matching questions, can you have a different number of premises and responses? Can you use responses for multiple premises?

• Can you branch to a non-interactive slide based on the answer the learner selects, i.e., for more thorough remediation?

• Can you group questions by objective and ensure the learner successfully passes each objective?

• Will the test select a designated number of random questions from objective groups?

• Can you prevent the learner from progressing in the course until they get selected questions correct?

• Can you force the learner to review missed items after a post-test?

• Can you run item analysis reports? If so, will the report show results by the top third, middle third, and bottom third of the class?

• Can you run item analysis reports only when the test questions are contiguous, or also when spread throughout a course?

Conclusion

My opening question was something like, “Who cares how they learn it, just so they passed the test.” The issue of whether you can leave workplace learners to their own discovery methods is a question to address at another time. Regardless of the learner’s path, and regardless of whether the test is spread throughout a course or given all at the end, we will provide the most valid and respectful learning environment if our tests are fair and thorough and if they continue adding to the learner’s growth even as they measure their knowledge.

References

Anderson, L. W., & Krathwohl, D. R. (Eds.). 2001. A taxonomy for learning, teaching and assessing: A revision of Bloom's Taxonomy of educational objectives: Complete edition, New York: Longman.

Berman, Elena. 2001 Using pretests. Downloaded from http://aer.arizona.edu/Teaching/docs/UsingPretests.PDF

Berry, Thomas. 2008. “Pre-test assessment,” American Journal of Business Education, Third Quarter 2008, Vol. I, No. 1, downloaded from http://www.cluteinstitute-onlinejournals.com/pdfs/1051.pdf

Bothell, Timothy W, Ph.D. 2001. 14 rules for writing multiple-choice questions, Brigham Young University, 2001 Annual University Conference.

Brandon, Bill. 2004. “Applying instructional systems processes to constructivist learning environments,” Learning Solutions Magazine, June 29, 2004.

Brandon, Bill. 2004. “How do people learn? Some new ideas for e-Learning Designers,” Learning Solutions Magazine, June 1, 2004.

Brooks, David. 2010. Private e-mail concerning the example of the new employee who got a sizable pay raise from her company after passing her tests at the one year point.

Brooks, David, Nolan, Diane, and Gallagher, Susan. 2003. “Automated testing.” Journal of Science Education and Technology, Vol. 12, No. 2, June 2003

Burton, Steven, Sudweeks, Richard, Merril, Paul, and Wood, Bud. 1991. How to prepare better multiple-choice test items: Guidelines for university faculty. Brigham Young University Testing Services and the Department of Instructional Science.

Dewey, Russell A. 2010. “Writing multiple choice items which require comprehension,” Psych Web, downloaded from http://www.phywww.com/selfquiz/aboutq.htm.

Dickinson, Mike. 2009. “Fiction writer tricks: A tale of two courses.” Learning Solutions Magazine. Feb. 17, 2009.

Forehand, M. 2005. “Bloom's taxonomy: Original and revised...” In M. Orey (Ed.), Emerging perspectives on learning, teaching, and technology. Retrieved November 17, 2010 from http://projects.coe.uga.edu/epltt/

Greenstein, Louis. 2010, Web of deceit. Press release from PreVisor Talent Measurement.

Hitchcock, Dale. 2005. “Post evaluation feedback to the adult learner.” ICF Consulting, Synergy, Inc. Downloaded from http://www.icfi.com/publications/register/download-register.asp?pubid=198

Mandernach, B.J. 2003. Effective multiple-choice items. Retrieved Nov. 8, 2010, from Park University Faculty Development Quick Tips.

Overbaugh, Richard, and Schultz, Lynn. 2009. Bloom’s taxonomy. Downloaded from http://www.medschool.vcu.edu/graduate/pgmdir_res/documents/bloomtaxonomy.pdf

Stoner, Mark, and Smith, Phil. 2007. “Assessing student learning: How do we know what they know?” EOP Learning Communications, downloaded from http://www.ctl.csus.edu/resources.htm.

Svinicki, M. 1999. “Test construction: some practical ideas,”. In Teachers and students: A sourcebook for UT-Austin faculty, Section 8, Testing and grading. Austin, TX: Center for Teaching Effectiveness, University of Texas at Austin. Downloaded from http://www.utexas.edu/academic/ctl/gsi/coursedesign/test_construction.pdf

The eLearning Coach. circa 2009. Are your online tests valid? http://theelearningcoach.com/elearning_design/isd/are-your-online-tests-valid/

The eLearningCoach. circa 2009. Are your online tests reliable? http://theelearningcoach.com/elearning_design/are-your-online-tests-reliable/

Tobin, Daniel R. 2000. All learning is self-directed, American Society for Training and Development, Alexandria, VA.

November 22, 2010