Introduction

As textbook costs have risen, faculty, librarians, and administrators have turned to free and open textbook materials. Open Educational Resources (OER) and Zero Textbook Cost (ZTC) courses have become mainstream ways of reducing some of the cost burdens in individual classes. OER materials are openly published remixable materials. ZTC materials draw from openly published materials, library books, library materials, and library-accessed articles. While most OER materials and books often include Creative Commons licenses allowing printing (especially helpful for accessibility), most ZTC materials do not carry an open to remixing license. Both create space for faculty and instructors to mix and match resources within their curricular design to meet learner needs. As pointed out by West and Victor (2011), this also means students in an OER or ZTC course need to know about the resources, learn how to access the resources, and have the technology to support accessing the resources as they work through courses. Essentially, West and Victor point out that communicating textbook adoption decisions when no commercial textbook needs to be purchased can immediately raise terminology and student-practices issues, affecting students’ perception of a course. When students are already familiar with the idea that a new semester comes with a bookstore trip and a high fee, free textbooks do not necessarily equate to positive perceptions of a course. Instead, potential hurdles point to a need to understand student perceptions of OER/ZTC materials in higher education courses.

Student perceptions of OER/ZTC materials are one important factor in considering wider scale adoption of OER by faculty. Faculty awareness of OER has continued to see growth. In 2015–16 only 34% of faculty reported they were aware of OER. In 2021–22 awareness of OER has grown to 57% of faculty (Seaman & Seaman, 2022). The increased awareness of OER has correlated with an increase of OER use in courses. Only 5% of faculty reported using OER in 2015–16, but that has grown to 22% of faculty in 2021–22 (Seaman & Seaman, 2022). As use of OER continues to grow it will be beneficial to broadly measure student perceptions and satisfaction with the materials over a broad range of courses. This will help measure the interactions of student satisfaction with other critical factors like student cost, learning outcomes, student agency, and student learning outcomes, that are important for successful OER adoption initiatives.

To understand student satisfaction with OER/ZTC materials, this article presents a validated measurement tool focused on student perceptions and using OER/ZTC materials when adopted within a course. Scholarship and approaches to understanding student perceptions of free course materials has a rich history beginning with the openly shared COUP (Cost, Outcome, Use, Perceptions) tool (Bliss et al., 2013). In his work synthesizing existing research on Open Educational Resources adoption, Hilton notes in both 2016 and 2018 there is a lack of shareable validated tools to understand student perceptions. This article and tool seek to fill that gap. The Student Satisfaction tool is provided here and can be used by other institutions to understand student satisfaction (perception and use) in courses adopting OER and/or ZTC materials (see Pfannenstiel et al., 2020 for student data analysis). The shareable nature of this validated tool also allows institutions to compare student perception data, and understand what factors may be institution-, state-, or region-specific. This will also allow adopters and institutions to understand which factors of student perception exist across OER/ZTC adoptions nationally and internationally. The validated measurement tool allows for a validated understanding of perception, usable data to inform course and campus adoptions of OER/ZTC materials, and functional cross-institutional comparison of student perceptions in cross-institutional OER/ZTC adoption efforts.

Measurement Tools

When this measurement tool was developed, the researchers could not find a validated measurement tool for understanding student perceptions and using OER materials. While much theory-based research has been developed since Bateman, Lane and Moon called for theory-based, generalizable research in 2012, few validated measurement tools have been shared for cross-semester, cross-disciplinary, and/or cross-institutional comparison of OER/ZTC adoption, despite many adoption initiative programs.

Bliss et al. (2013) developed the COUP framework as one of the first tools to provide perception measures, focusing on the impact of cost, outcomes, use, and perceptions of open digital textbooks. Bliss et al. narrowed the impact to these elements within a framework based on the ability to separate; the impact on reducing cost can be assessed on its own, impact on use can be evaluated on its own and across a breadth of elements meaningful to specific adoption practices, impact on students perceptions can be assessed on its own, and impact on outcomes can be evaluated on its own. For each element of the framework, specific adoption practices may lead to narrowing or expanding definitions suitable for the institutional situation. Similarly, each piece may be clearly defined and built into a measurement tool to be used across various student groups for a broader understanding of impact. Finally, Bliss et al. also emphasizes that elements within the framework can impact each other; the reduced cost can impact learning outcomes if a student works less or at fewer jobs because the cost of attendance is reduced, etc. Building upon the work of Bliss, Colvard et al. (2018) explored the impact of OER on student success metrics like D/F/W rate and overall grades, important metrics to be sure, but these data can be better understood when aligned with student perceptions of the resources.

This work created the foundation for our measurement tool; however, we focused specifically on perception and used it while designing for validation. The Bliss tool includes essential questions but is limited by the various measurement formats throughout the survey. Developing and validating a tool based on the ideas from the Bliss tool, we focused on use and perception from the COUP framework, which will help with cross-semester, cross-disciplinary, and cross-institutional comparisons of student use and perceptions of OER/ZTC materials.

Drawing from this understanding of the COUP framework, we developed and validated a measurement tool to understand both self-reported student perception and the use of OER/ZTC materials. We are calling the connection of these two framework elements student satisfaction with OER/ZTC materials.

As pointed out in both Hilton studies (synthesis in 2018, review in 2016), tens of thousands of students have been asked about their perceptions of OER/ZTC materials, and more needs to be done to validate the measurement tools to compare perceptions across institutions. Additionally, as noted by Hilton (2016), many of these perception studies do not also address self-reported use of assigned OER/ZTC materials – what we call satisfaction. Students may overwhelmingly report positive perceptions of free materials, but understanding those perceptions alongside their self-report use of the materials is important if the field is to use perception data in support of future adoptions and initiatives.

Jaggars et al. (2018) introduce a survey instrument focused on three elements of student perception of OER/ZTC. Jaggars et al. discuss the reliability and predictive validity of their tool focused on student self-reported comparisons between OER/ZTC materials and commercial textbooks. Brandle et al. (2019) focus on student use of OER/ZTC materials, seeking to understand and compare student perceptions of OER across institutions within their system. Moving beyond much of the work synthesized by Hilton, Brandle et al. surveyed students from across the CUNY system campuses on where and how they used OER materials for their courses. Todorinova and Wilkinson (2019) explore OER incentive programs as academic librarians, administering a student perception study as part of program improvement assessment. Similar to many other metrics, their study found students are concerned about rising textbook costs, and generally favor OER in courses. Working in similar space, our goal for this validated survey is for programs like Todorinova and Wilkinson’s to have an easily accessible student perception metric, that produces data an institution can compare to existing data sets. The comparability offered through a validated measure can help textbook affordability program assessment and growth across the nation.

Building on the tools of these early studies and drawing from Hilton’s findings about perception and use, this study develops a validated measurement tool meshing student use and student perceptions. These existing studies and surveys build an important foundation for understanding student use and perceptions. The goal is widespread use of this tool to build a body of student satisfaction data for cross-institutional comparison, to support student learning and success, and to support future adoption initiatives.

Methods

Results

Discussion

The Student Satisfaction with ZTC Materials Tool provides a validated measure for assessing student experiences with the use of ZTC materials. The tool has been used to assess a wide variety of courses as part of a ZTC initiative. This tool allows practitioners to gain a better understanding of student experiences with ZTC materials as compared with commercial textbook materials. Understanding student interaction and use of ZTC materials are crucial to understanding the impacts of ZTC and OER initiatives that go beyond the cost saving impact students experience. Analysis of student responses to this survey has shown that students’ past experiences with commercial textbooks influence their perceptions of ZTC materials (Albert et al., 2021) which could influence the learning impacts of ZTC and OER initiatives.

Measurements of the impact of OER on overall student learning have seen mixed results. A meta-analysis by Clinton and Khan (2019) showed no differences in student learning performance for OER versus commercial textbooks but did see that withdrawal rates were lower for courses that used OER. Much of the power of OER to impact student learning in the classroom is tied to the “Access Hypothesis” that suggests the impact from OER will be experienced by students who did not previously have access to course materials (Grimaldi et al., 2019). The “access hypothesis” provides an explanation as to why measuring overall learning gains from OER adoption will prove difficult since many of the students in the course would have had access to materials no matter if a commercial or OER textbook was used. It is only then the students who gained access by adopting OER that will see potential benefit.

Our Student Satisfaction Tool allows for the ability to see interactions between student satisfaction in OER/ZTC courses and other attributes like typical textbook spending and typical interactions with course materials. In previously published work (Pfannenstiel et al., 2020 and Albert et al. 2021), we found that students who typically purchase and have access to course materials report higher satisfaction with OER/ZTC materials and rate the materials as being more beneficial for learning. These findings suggest that providing access to materials for students who do not typically have access to materials is not enough since they report lower satisfaction than peers who typically access materials. This may explain why the expected impacts from the access hypothesis are not always realized (Smith et al., 2020). In measuring impact from OER adoption initiatives not only do we need to account for the fact that only a fraction of students are gaining access but we also need to consider that the students who gain access to materials have built structures to navigate courses without materials. We won’t fully see the potential of OER adoption without also providing support for students who newly gained access to course materials and would benefit from structures that help them navigate a new course experience with access to materials. The broadly applicable ZSS allows for measuring these types of impacts in OER adoption initiatives since it looks at combined student perception and use.

This general tool allows for a wide scale assessment of ZTC student experiences across the college curriculum from a wide variety of disciplines. The validated scale allows adopters of OER and ZTC to measure student perceptions of the course materials, which could lead to faculty ultimately choosing different materials or changing the course structure and instructional support for how students interact with the materials. Faculty changing from the use of commercial textbook materials to ZTC materials have more ownership over the presentation and integration of materials which makes measuring student perceptions of materials crucial as the materials are easier to modify due to the open licenses associated with them. This validated measurement tool facilitates the process of revising and improving a course over time. The more intentional choices faculty have over course materials and presentations could also prime faculty to see the importance of integrating instructional design (Hart, 2020).

Use of a validated instrument for measuring student perceptions of ZTC material implementation in courses and with broad adoption initiatives at a wide variety of institutions will allow for better measurement of the salient features of ZTC adoption. This could lead to improved professional development supporting faculty in adopting ZTC that ultimately leading to improved student perceptions and interactions with the materials. The enhanced understanding will allow for improved ZTC adoption initiatives and improved student experiences with ZTC.

Conclusions

The validated ZSS is available (See appendix A) via a CC BY-NC-ND license is a valuable tool for individual faculty to better understand student perceptions of their OER/ZTC adoptions and for broad scale OER/ZTC adoption initiatives seeking to understand student experiences with materials across a wide variety of courses. The goal of this validation is to share an easily adoptable measurement tool for faculty to deploy to understand student perceptions in their classrooms. Additionally, the hope is OER incentive programs will use this tool as part of program assessment, sharing their data and results. With OER adoption increasing from 5% to 22% (Seaman and Seaman, 2022), understanding student perceptions of the adopted material is critical to adopters and institutions nationwide. A validated measurement tool allows for important cross-institutional comparison of data. Continual improvement of student experiences with course materials is made easier with ZTC and OER materials. The ZTC Student Satisfaction Survey instrument provides a validated benchmark that faculty and program coordinators can use to facilitate future adjustments in supporting the adoption of ZTC materials.