Assessment and Review

After completing a first draft of your resource, you will likely want to gather some feedback from others. This will help you determine what is working well in your resource and identify areas for improvement. It is valuable to get feedback from a wide range of stakeholders, since each person will bring their own perspective and may notice things that others do not. This feedback may be focused on content, such as missing topics or areas where more explanation is needed, structure and organization of information, or on user experience, such as the look and feel of your resource, how well it works on various types of devices, etc. It can be helpful to let people know what type(s) of feedback you’re looking for specifically before they review your resource, so they know what to look for as they interact with it.

Student Assessment

The ACCA program includes a requirement to conduct some basic assessment from your students. This generally involves the use of a simple survey to gather the feedback on your students’ impressions of the resource. Survey questions tend to focus on how often the student used the resource, ease of navigation, whether or not the resource represented a diversity of people and experiences, whether the content was clear, relevant, and aligned with course lectures, tests, and quizzes. It’s also valuable to ask students for their impressions of how much the resource contributed to their learning.

To help you meet this program requirement, you can use the Sample Student Assessment Form template and modify, add, or remove questions to meet your specific project needs.

In addition to completing the ACCA assessment, you may want to dig deeper into assessing your resource’s impact on your students. Possible research angles include:

• How has the OER impacted student learning outcomes? How has OER usage impacted student engagement with the learning materials? Your assessment could entail pre- and post-testing of students using the OER and those using the traditional textbook and see how those compare.
• How has OER usage impacted student success? Your assessment could entail gathering and comparing grades from students in OER and non-OER classes. Alternatively, you could compare DFW (drop, fail, withdrawal) rates for OER and non-OER classes.
• What do students think of the OER? How does their perception compare to commercial textbooks? Your assessment could entail a survey asking students about their perceptions of the resource.

The OER Assessment Toolkit from the University of Texas at Austin provides an in-depth look at different course measures and how to craft an assessment instrument. It also contains examples of qualitative and quantitative assessments of resources in various courses. This may be a good starting place to design a more comprehensive study, and your OER Librarian is also available to help!

A quick note about research: approval from the Institutional Review Board (IRB) is not necessary for the purposes of meeting the ACCA assessment requirement, but would be necessary if you want to share your findings more widely, for example in a publication or conference presentation. You can visit the Office of the Vice President for Research and Innovation’s page about VCU’s Human Research Protection Program/Institutional Review Board to learn more about the approval process.

Faculty Review

In addition to gathering feedback from students, you may also be interested in getting feedback from other faculty. Although some OER publishing programs do have formal peer review processes, many do not, including ACCA. However, that does not prohibit authors from seeking feedback, either formally or informally, from peers in your discipline! To find reviewers, you can leverage your department, your professional networks, or the broader open education community.

Similar to requesting feedback from students, it is helpful to provide some guidance on what types of feedback you’re looking for, specifically. There are many existing rubrics and frameworks for reviewing OER, and you can either adopt one or customize your own to meet your project’s specific needs.

Rubrics and Frameworks for Review

One good starting place are the review criteria used by the Open Textbook Library, which were adapted from a rubric originally developed by BCcampus. The Open Textbook Library supports faculty reviews of all of their open textbooks, which are shared publicly on the resource listing pages. Their review criteria are a helpful framework for assessing the quality of a resource:

• Comprehensiveness
• Content Accuracy
• Relevance/Longevity
• Clarity
• Consistency
• Modularity
• Organization, Structure, and Flow
• Interface
• Grammatical Errors
• Cultural Relevance

Another excellent resource is the MERLOT Peer Review Report Form – V 17.6. MERLOT is an OER publisher with a formal peer review process, for which this form is used. However, it is licensed under CC BY-NC-SA, so it can be reused and remixed to fit your project’s needs. That rubric has guiding questions for ranking in the following areas:

• Quality of Content
• Potential Effectiveness as a Teaching Tool
• Ease of Use

Lastly, the Achieve OER Rubrics are another great resource to adopt or to draw on to create a review rubric. These rubrics are used by OER Commons, a major OER repository, to allow reviewers to evaluate resources on their platform. You can also view this short video from Achieve, which gives an overview of the rubrics, how to apply them, and how they are scored.