A variety of tools are available to you for practicing and learning about item writing and test development and building your course tests. This chapter reviews some of these tools, with an emphasis on Proola, a web application developed, in part, to accompany this book. The Proola resources introduced here will allow you to apply and extend your learning about effective item writing through sharing and collaboration with peers.
As an instructor, you’ve probably used learning-management systems (LMS) in one way or another to manage data or information on students in a course you teach. An LMS is software that helps you organize and share course content with students and collect data from students in the form of assessments, assignments, and other submitted work. An LMS also provides tools for communication and collaboration between you and students and among students. LMSs differ from other computer-based instructional resources in their scope. LMSs are intended to be a comprehensive solution addressing all aspects of the teaching and learning processes, whereas other resources may only address certain components of these processes. Some popular examples of LMSs include Blackboard, Moodle, and Canvas by Instructure.
LMSs provide a number of resources to support the development of course tests. Features are available for creating and storing a variety of item types, administering items to students, and evaluating item quality with metrics such as item difficulty and discrimination (more on these in Chapter 7). Features are also available for sharing items with other instructors and accessing item banks from textbook publishers (more on this in what follows). LMSs provide only limited support for improving the quality of your own items or evaluating available items.
As you probably know, course textbooks in higher education commonly provide access to item banks, sets of example test items that are intended to assess student KSAs related to the content of the textbook. These item banks may be printed within an instructor’s copy of the textbook or given in a separate publication as a supplement to the textbook itself. Many are now provided electronically, with some being integrated into LMSs through add-ons from textbook publishers. In the next chapter, we review numerous items from sources such as these.
The quality of items in textbooks or LMS item banks should not be taken for granted. Instead, these resources should be used as starting points when creating your own course test. Published example items can demonstrate both effective and ineffective ways of assessing the KSAs in your instructional learning objectives. Most often, these items will be flawed in at least one way and will need to be modified to meet the item-writing guidelines in Chapter 4 and to address the content and cognitive tasks established in your test blueprint based on your learning objectives. Reusing example items without editing and adaption can result in tests measuring unbalanced or irrelevant content, which can lead to invalid score interpretations for your students.
Valuable local resources for improving your item-writing skills and the quality of your test items are often provided by a teaching and learning center at your college or university. Examples with links are provided in Chapter 2. Proola was developed to supplement these resources by providing educators with an online learning community and shared space for practicing writing items.
Proola was originally developed as a resource for educational and psychological measurement courses, where graduate students were required to write and comment on the test items of their peers. The site proola.org provides a simple interface for creating MC and SR questions and then tracking conversations about these questions as they are revised and improved. Proola is continuously being developed to support different audiences of item writers and test developers, including K–12 teachers and college and university instructors. Access is free but requires registration at proola.org/signup.
After signing up for an account, you’ll have access to the item bank and a user dashboard, where you can store and organize your learning objectives, test blueprints, items, and tests. The item bank contains all of the items that have been submitted for review and peer feedback. Note that items from your specific field may not yet be available.
Proola differs from other item-banking and assessment-sharing software in two main ways. First, it is primarily intended to support learning about assessment. Proola does not currently support administering and collecting student responses on assessments. Instead, it aims to be a sandbox for writing items while building item-writing skills. Items created in Proola can easily be exported to PDF or to an LMS for administration. Second, Proola functions as a community of educators and measurement specialists in which knowledge and quality assessments are built collaboratively via peer review.
The item bank at proola.org/items gives you access to hundreds of items written by students and educators with various backgrounds and levels of experience in assessment and measurement. Searching the bank, you may not find items that address your own instructional objectives. However, you will find items from a variety of subjects that demonstrate common mistakes in the item-writing process. You’ll also find items that adhere to the guidelines and incorporate novel contexts to assess higher cognitive tasks. Items of different qualities can serve as examples for you as you brainstorm effective methods for assessing your own course content.
As you browse the bank, you’ll notice that users can comment on others’ items. You should read through others’ comments before providing comments of your own (more on commenting in what follows). You’ll also notice that you can save others’ items. Once saved, they’ll show up in your dashboard for easy access.
As we noted, Proola is built around a simple model of peer review. Before an item is designated as “Approved” in the item bank, it must have been submitted and available in the bank for commenting, and it must have received a formal review, with a positive result, from a Proola administrator. Once you have created your learning objectives, the review process for items involves four main steps:
Peer review will be provided by a Proola administrator, and results will show up within the conversation below the latest draft of your item. If revisions are recommended, you’ll have the opportunity to revise and resubmit. Any edits will then be made to a new version of the item, where previous versions are viewable but not editable. The revision process can be repeated as needed until the fourth version of the item. Once approved, your item can be printed and exported by other users, and you begin to build recognition with the community as an effective item writer.
Here we provide some pointers on authoring your own learning objectives and corresponding test items in Proola. Before you begin, you should browse the objectives and items of other users for examples of what to do and what not to do. That said, don’t be afraid to make mistakes, as you can always edit and delete what you don’t like up until you submit your work for peer review.
Item development begins with the development of a clear and concise instructional learning objective (see Chapter 2 for guidelines). In Proola, you can create and store objectives in your user dashboard and submit them for peer review prior to beginning item writing. Here are some guidelines on writing effective objectives.
Once your objectives have been revised as needed, you can start working on items that address them. The item-writing process is simple. Proola requires that you provide a brief descriptive title for your item, along with a target grade level and subject area. Then your item consists of a stem and at least one option. Follow the item-writing guidelines given in Chapter 4 while keeping in mind the following points.
Reviewing and commenting on the work of others will give you a new perspective on the item-development process. By participating in peer review, you’ll learn new item-writing strategies, such as using applications to assess higher depth of knowledge and cognitive tasks. You’ll also see approaches that should be avoided, such as making an item difficult via trickery (intentionally writing an item to catch the distracted student). Here are a few principles of effective commenting.