Introduction

Successful adoption of Open Educational Resources (OER) in higher education arguably depends on the effectiveness of the materials. If an argument is made that OER is at least as effective as traditional non-free closed (proprietary) educational resources (CER), then why not choose the free, more adaptable and accessible OER? There may be other factors to consider, such as development of the materials and potential administrative roadblocks. But regardless of these factors, effective OER should be considered a viable substitute to CER.

In this paper, we report on the results of a study designed to compare free OER to the proprietary CER used in Calculus at our university. Four sections of first-semester Calculus were taught face-to-face during the Fall 2017 semester, two using an open textbook and web-based homework system, and two using a proprietary textbook and homework system. Performance measures were used to compare student achievement between the two groups.

Using statistical analyses, we develop multilinear models for predicting student success, and we demonstrate various significant effects measured to within a probability of 5%. First, the OER students accessed their web-based homework earlier than the CER students, which contributed to higher scores on their first exam, on average. However, this effect was not sustaining, as the CER students performed better later in the semester. Second, student preparation prior to class was an indication of better performance, and this effect is sustaining, affecting all performance measures throughout the course. Third, students in the CER group performed better on their homework, which directly correlated with final exam scores. To understand the reasons for this difference, we consider student comments concerning the quality of experience using the OER and CER homework systems.

Literature Review

In his 2016 survey paper, J. Hilton (2016) investigated the state of OER by summarizing the literature on effectiveness and perceptions. At that time, the landscape included a total of just 16 published articles, of which nine focused on the efficacy of OER textbooks as a replacement for CER textbooks. In his address at the 2018 open education conference, Hilton (2018) emphasized that many more studies are needed to fill the gap.

Development of OER courses typically involves replacing traditional textbooks with open textbooks and online resources. Several studies have shown that such a transformation does not significantly affect student performance and or progression, such as Allen, Guzman-Alvarez, Smith, Gamage, Molinaro and Larsen (2015), Weller, de los Arcos, Farrow, Pitt and McAndrew (2015), Hilton, Gaudet, Clark, Robinson and Wiley (2013) and the papers reviewed in Hilton (2016). Some studies indicated an improvement in performance and retention using OER, such as the open psychology implementation in Hilton and Laman (2012), and the secondary level science study in Robinson, Fischer, Wiley and Hilton (2014) based on end-of-the-year state standardized tests. In Hilton and Laman (2012), their study showed a withdrawal rate of 7.1% for the OER group compared to 14% for the traditional group.

In determining the effectiveness of OER, one approach is to define and attempt to validate statistical hypotheses. In Weller et al. (2015), eleven such hypotheses are considered, concerning: Performance, Openness compared to other online resources, Access (equitability), Retention, Reflection (among faculty teaching), Finance (cost-savings) and a few others. However, their investigation specifically concerns perceptions, not efficacy. Note also that their hypothesis on “access” is different from the “access hypothesis” stated in Waters, Mallick, Grimaldi and Baraniuk (2018) which hypothesizes that barriers in adopting the traditional materials led to poorer performance than with OER materials. Among the more analytical studies on efficacy is Allen et al. (2015). In it, they conduct a statistical approach to hypothesis testing by demonstrating non-inferiority of their OER. In general, all these studies indicate OER is at least as good as CER.

The above studies consider the textbook as the sole resource. However, in mathematics, web-based homework is often the dominant resource. Nguyen, Hsieh and Allen (2006) and Williams (2012), as well as many others cited in these papers, investigated the effectiveness of online homework versus traditional homework in mathematics and statistics. These studies indicate that instant grading and feedback is an advantage of online homework. However, we note that Williams (2012) showed that the traditional homework group scored higher on homework and final grades than the online group. They attribute this in part to the lack of partial credit and feedback with their online homework system.

A widely used open online homework system is WeBWorK. It is open source software that can be downloaded and installed for free and contains an open problem library of over 40,000 math and science problems. Good references on using WeBWorK include the reports (Carpenter & Camp, 2008; Denny & Yackel, 2005; and Gage & Pizer, 1999), and papers on effectiveness WeBWorK include Hauk, Powers, Safer and Segalla (2014) and Swanbom, Moller, Evans and Reeves (2016). This homework system plays a primary role in our study that follows.

Filling the Gap

The goal of this study is to provide new insights on the effectiveness of OER compared to CER. The first aspect of our work that is different than the existing literature is that the textbook is not the main, or only, resource. In our work, web-based homework plays a much more prominent role. Indeed, students indicated that the textbook was the least important resource for them in learning Calculus. The second aspect of our work is the tightly controlled side-by-side comparison between the traditional proprietary CER materials and replacement OER materials. The same instructor taught all classes. A third aspect of our work is the quantitative nature of our study. We provide a statistical analysis (in particular multilinear regression models) that predicts student outcomes as well as marginal gains. Finally, we provide validation to certain hypotheses conjectured in the literature concerning benefits of early access on performance using OER.

Methodology

This study was conducted during the Fall 2017 semester, involving four classes of Calculus taught in a face-to-face format, each meeting four hours a week in lecture and one hour in recitation. Two of the classes used the textbook and web-based homework normally used in our department: Thomas’ Calculus and MyMathLab. These two classes comprise the CER group. The two other classes, the OER group, used an open source textbook and web-based homework system installed at our university: OpenStax Calculus and WeBWorK. Table 1 compares features of the two homework systems used in this study. The biggest differences between the two homework systems are cost, problem selection, and help resources.

The following procedures were followed to reduce variability between the two groups:¹

• Same instructor for all classes (the author of this paper)
• Same number of students (70 in each group at the beginning of the semester)
• Students had no prior knowledge to opt-in or not
• Same lecture notes
• Same exam reviews
• Similar exams
• Similar homework
• Similar student demographics

To determine the effectiveness of the course materials, student data was collected throughout the semester, including a pretest, homework login (first date web-based homework was accessed), homework scores, and exam scores. The data was analyzed using statistical tests, leading to the results and conclusions of our study.

Cost Savings

One advantage for students using OER is the money they save. The traditional course materials used at our institution include a textbook and the web-based homework system MyMathLab. These come as a bundle for $288.40 including a hard copy of the book, or $106.90 with an electronic book (based on 2017 prices). Hence, the total cost for 70 students in the CER group ranges from $7483 to $20,188 total. Based on information provided by our bookstore, the vast majority of the students opt for the online textbook. Hence, the cost would be closer to $10,000 than $20,000 for these 70 students using CER, which equates to the amount saved by those students using free OER.

Statistical Analysis

In this section we analyze our data to determine significant effects and differences between the CER and OER groups, and to create multilinear models that can be used to predict outcomes on exams based on the covariates in our study. To carry out the analysis we use the statistical libraries in the programming languages R, Matlab and LibreOffice Calc.

In our first comparison, the performance means for the CER and OER groups are analyzed. The results are in Table 2.

The last column lists p-values, which are the probability that the Null hypothesis “means are the same” is valid based on the T-test (we also used a rank-sum test on the medians with similar results). A p-value of less than 0.05 (5%) indicates significant differences. The first item of the table compares the average scores of the pretest students took on the first day of class. The mean scores of 8.40 and 8.43 out of 20 are nearly identical, with p-value of 96.5%, indicating the ability of the two student populations at the beginning of the semester are nearly identical. Henceforth, we can assume that differences in other performance measures are due to differences in instruction methods, not prior experiences. The items in the table that indicate significant differences are the “login day” (which is the first date students accessed the web-based homework), “exam 1” and “homework”. Of these, the difference in first login day is highly significant, with p-value 3e-5.

Next, a multilinear regression analysis was conducted to determine the significance of the covariates on student performance in our study. The covariates are variables that may affect performance measures but not directly used in calculating performance. For example, the four exams and homework averages are variables but not covariates of the course average. However, other factors, such as students’ pre-test scores may indirectly affect their course average, hence are covariates. We aim to determine which effects are significant and to derive linear models for performance measures based on these covariates.

The covariates considered in our work include the Pre-Test score (P), whether a student is in the OER group (G=1) or the CER group (G=0), the first day accessing the online homework system (A ranges from -3 to 28 in our study), and the final homework average (H). We look at the significance of these effects on the performance measures (exam scores and final average) according to equations. The results of this analysis are in Table 3.

In Table 3, the significant effects include only those covariates with probabilities less than 5% (i.e., .05). Those with probabilities higher than 5% are crossed off. After removing those covariates, a linear regression is run again with only those significant covariates. From this, we arrive at the multilinear models in Table 4.

For example, the equation

Exam 4 = 32.434 + 1.580*P + 0.397*H

shows that a student who earned 10 points out of 20 on their pre-test and had a homework average of 80, would be expected to get a final exam score of

Exam 4 = 32.434 + 1.580*10 + 0.397*80 = 79.99.

Another way to understand this formula is by marginal gains. For each gain of 1 point on the pre-test, the student is expected to have a gain of 1.58 points on their final, and for each gain of 5 points on homework, they would have about two more points on their final. In this way, one can compute marginal increases. For our linear models, these marginal gains correspond to partial derivatives.

Early Access Effects

Students using OER typically have access to course materials on the first day of class at no cost and with little effort, while students using CER may delay purchasing materials due to costs or other considerations. In Waters et al. (2018) it was hypothesized that this early access to open textbooks would enhance student performance. To our knowledge there is no research to validate their hypothesis, we can demonstrate the access hypothesis with regard to web-based homework. In a typical semester at our university, students using the proprietary homework system delay purchasing the needed access code, or have difficulty logging on using the access code. This delays when they can start using the homework.

The access day is illustrated in Figure 1. At the end of the first week, 91% of the OER students had logged in to their online homework system, compared to 65% of the CER students. This difference was shown to be significant in Table 2. Moreover, from the multilinear regression analysis in Table 3, we have the fit:

Exam 1 = 26.109 + 3.526*P - 0.840*A

This shows that for every day a student delays logging onto their homework system would lower their score on the first exam by .84 points.

Alternatively, we can express this result in terms of whether students are in the OER or CER group:

Exam 1 = 17.197 + 3.665*P + 8.989*G.

Here, the marginal gain in exam performance due to student preparation (as demonstrated on their pretest) is 3.665 points higher, while the students in the OER group (G=1) gained 8.989 points more than in the CER group (G=0) on their first exam due to the early access. The gains in first exam scores for the OER over the CER group is illustrated in the linear regression fits in Figure 2.

Preparation Effects

Student preparation was measured by a pretest given on the first day of class, before instruction. The linear models in Table 4 show that student performance on all exams depends significantly on their pretest. Hence, we consider the effect of preparation on student performance as sustaining, lasting throughout the semester.

Homework Effects

In our study, the web-based homework was assigned weekly, and the homework average was recorded before the Exam 4 (the final exam). By Table 4, the final exam depends significantly on the pre-test and homework:

Exam 4 = 32.434 + 1.580*P + 0.397*H

Every point (out of 20) on the pre-test correlates to a gain of 1.58 points on the final exam, and every homework point (out of 100) corresponds to a gain of .397 in the final exam. By this model, there are two paths to success: by pre-class preparation and by homework performance.

Another model for the final exam from Table 4 is:

Exam 4 = 64.255 + 1.923*P – 8.199*G

Students in the CER group (G=0) do better on the final exam on average than students in the OER group (G=1), which is consistent with the higher homework average in the CER group. In understanding this difference in performance using these two homework systems, we consider student comments. Students using the proprietary homework system appreciated the online help, similar problems generated, and reference to the textbook, while students felt the open system did not provide enough online support. As a result, some students had a better overall experience using the proprietary system.

Conclusion

In this paper, we investigated the effectiveness of OER in teaching College Calculus using face-to-face instruction. We demonstrated significant differences in performance between the OER and CER groups, and we developed multilinear models to predict performance based on the covariates: pre-test, homework access data, group (CER or OER), and homework average. Based on the pretest, we showed that the preparation prior to instruction was similar between the two groups, and had a significant effect on performance throughout the semester. We also showed that the students in the OER group likely scored better on the first exam due to the earlier access to the web-based homework, verifying a kind of “Access Hypothesis.” On the other hand, the overall homework average and final exam scores were higher in the CER group, which, based on student comments, may be attributed to the level of help resources and overall experience with the homework systems.

There were a few limitations to our study. While four sections with 140 total students were enough to determine significant effects, a large-scale implementation involving more students and faculty may provide further insights. A second limitation to the study was in effectively selecting homework problems in the open system consistent with the proprietary system, which possibly affected student performance. Finally, while this is a quantitative study measuring student success based on exam scores, we do not claim our study directly addresses student learning.

There are several directions to consider in follow-up studies. In particular, in a forthcoming paper, we are investigating student perceptions and attitudes of OER. Another area of investigation is the impact of OER on remediation. We believe that reaching out to students prior before class begins would help improve student performance. We also suggest that careful choice of the homework problems and consideration of help resources available with the open homework system can affect the overall experience, and close the gap between student performance using the OER and CER homework systems.