Two free fully online courses were offered by Peoples-uni on its Open Online Courses site, both as self-paced courses available any time and as courses run over four weeks with tutor-led discussions. We tested the hypothesis that there are no measurable differences in outcomes between the two delivery methods. Similar numbers attended both versions of each course; students came from multiple countries and backgrounds. Numbers of discussion forum posts were greater in tutor-led than self-paced courses. Measured outcomes of certificates of completion, quiz completion and marks gained were very similar and not statistically significantly different between the tutor-led and the self-paced versions of either course. In light of little discernible difference in outcome between self-paced learning compared with courses including tutor-led discussions, the utility of the time cost to tutors is in question. The findings may be relevant to others designing online courses, including MOOCs.
There has been an explosion in the offering of online education in recent years, accelerated by the number of Massive Open Online Courses (MOOCs) available. Peoples-uni Open Online Courses (http://ooc.peoples-uni.org) have been running since 2008 and have included those which are available at any time for self-paced learning, as well as a number which have also been offered to a timetable with tutor facilitated discussions (Heller, Zurynski, Barrett, Oaiya & Madhok, 2017). But what is the role of the tutor in the course discussions? We have had the opportunity to compare and contrast tutor-led and non tutor-led courses in the context of two Open Online Courses where courses were offered with tutor facilitation of online discussions and the same courses offered for self-paced learning without facilitated discussions.
The People’s Open Access Education Initiative, Peoples-uni (http://peoples-uni.org) aims to build Public Health capacity mainly in low- to middle-income countries through online learning offered at a very low cost. In addition to its courses for academic credit towards a master’s award (Heller, Strobl & Madhok, 2019), a separate site, also using a Moodle platform, provides free Open Online Courses http://ooc.peoples-uni.org (Heller et al., 2017). While these are not intended for a massive audience, they have similarities with MOOCs in that they are free and fully online. Students enrol themselves, and in general the courses are self-paced with access to online resources, metadata to guide the students through the resources, the opportunity to post reflections on a discussion forum, and a quiz. Criteria for the award of a certificate of completion vary between courses, but all include the completion of a multiple-choice quiz (MCQ). Some of these courses are run to a timetable with expert tutor facilitation of discussions, and we report on two courses that were offered both with and without tutor facilitated discussions through the Peoples-uni Open Online Courses site.
An Exercise and Health course was launched in October 2015 and an Injury Prevention course in April 2017. They were available for self-enrolment for self-paced learning, and low key publicity was conducted through various social media outlets and previous Peoples-uni students and graduates. They joined approximately 20 other courses on the Peoples-uni Open Online Courses site http://ooc.peoples-uni.org hosted on the Moodle platform. Students enrolled themselves up to the time that the tutor-led versions started. The same two courses were copied and updated and offered for new participants over a 4-week period in November 2018. Expert tutors facilitated the online discussions, which had existed in the previous versions but without tutor involvement. Both courses aimed to explore issues in a global context.
The Exercise and Health course provided a context for a series of 18 presentations prepared by an international group of content experts, coordinated by Exercise Works! (http://exercise-works.org/). Originally developed as PowerPoint presentations, they were each converted to e-books using commercial software. They were placed in an educational context with learning objectives, metadata and other links to relevant open access resources including those from the World Health Organisation. The resources were placed in four major sections, which each also included a discussion forum for participants to reflect on pre-prepared questions made available on the course front page. A multiple choice quiz tested knowledge of a number of aspects covered in the course, with detailed feedback, no restriction on the number of attempts, and a pass mark of 8 correct out of 10 questions. An automated certificate was awarded with the criteria of having posted to each of the four discussion forums and having a pass mark in the quiz. Other than the restricted timetable and use of tutor discussion facilitation, the only differences between the self-paced and tutor-led courses was that the presentations were updated to the 2018 versions, and a video produced by one of the tutors added to the resources. The tutors for the tutor-led version of the course were from four different countries and were all experts in the field with educational experience, and most had been involved in the preparation of the presentations on which the course was based.
The Injury Prevention course was originally developed as a course for academic credit through the Peoples-uni master’s level programme (http://peoples-uni.org) (Heller et al., 2019), developed by an international group of volunteer tutors with special interest in injury prevention and Public Health. It was converted for the Open Online Courses site, and had a similar structure to the Exercise and Health course described above. Differences were the absence of e-book presentations, rather a set of resources as links to Open Educational Resources and more extensive metadata in each of the four sections. The quiz only required a grade, rather than a pass, with one mark assigned for each of 11 questions. The certificate, in addition to a grade in the quiz, required the download of the resources in each section rather than posting to the discussion forums. Small differences between the self-paced and tutor-led courses involved the addition of one video and updating some of the links to resources. The tutors were among those who had developed the course originally, came from three countries, and each had experience as online tutors for Peoples-uni.
Each of the courses was offered free, and advertised through social media and to previous students of Peoples-uni.
We tested the hypothesis there are no measurable differences in outcomes between online courses offered with tutor facilitation of online discussions and the same courses offered for self-paced learning without facilitated discussions.
There is considerable debate about whether online (face-to-screen) learning achieves the same educational outcomes as face-to-face education, with a large review concluding “...online learning has been modestly more effective, on average, than the traditional face-to-face instruction with which it has been compared” (US Department of Education, 2009). Others find no difference (Cavanaugh & Jacquemin, 2015), or dismiss these findings (Protopsaltis & Baum, 2019). However, there are a number of limitations in the methods of comparison between online and face-to-face courses (Stack, 2015) and most of the published comparisons relate to largely undergraduate, for-credit courses. Although the results offer the providers of online education some reassurance that acceptable outcomes are possible with this method of delivery, the debate is not particularly relevant to MOOCs, which are usually offered in addition to traditional university courses. There is no face-to-face alternative to MOOCs due to the large numbers, the geographical spread of the students, and the lack of academic credit –although some providers are now offering micro-credits.
It has been suggested that there are two types of MOOC, xMOOCs, more similar to traditional university courses (but without face-to-face teacher-student interactions), and cMOOCS where collectivists of teachers and learners work together and network to explore and develop content (Bates, 2014). The connectivist approach to learning, defined by Siemens (2005) underpins cMOOCs and reflects the use of technology and connection to move learning into the digital age. Most of the xMOOCs have a standard format and contain pre-recorded video lectures produced by tutors who also facilitate online discussion forums (Zhan et al., 2015). Both xMOOCs and cMOOCs include tutor facilitation of learning.
The type of online learning we describe, in common with both forms of MOOC, fits into the self-directed learning component of adult learning theory (Manning, 2007; Knowles, 1975). Social networks formed by communities of learners formed through the formal or informal discussion forums may assist each other and contribute to learning outcomes (Grandzol & Grandzol, 2006; Fidalgo-Blanco, Sein-Echaluce, García-Peñalvo & Esteban-Escaño, 2014; Sharif & Magrill, 2015). Various third-party organisations now offer structured social interaction and support for MOOC-based learning, both online and offline (Creelman & Witthaus, 2018). But the role of the tutor in the course discussions has not been explored in the context of MOOC-based learning - this is central to our paper, where we pose the question about the value of tutors in not-for-credit online education such as those offered as Open Online Courses.
The review of online learning in more traditional higher education, which we have quoted above, has suggested “Organizations providing or promoting online learning generally recommend the use of instructors or other adults as online moderators, but research support for the effects of this practice on student learning is mixed” (US Department of Education, 2009). In addition, a review of the literature relevant to higher education in the USA summarises “...surveys find that prospective and actual online students clearly demand a more interactive educational experience, which includes regular and direct contact and communication with instructors...” (Protopsaltis & Baum, 2019). An earlier review indicates “...that the quality of interpersonal interaction within a course relates positively and significantly to student grades” (Jaggars & Xu, 2013). A number of reports have suggested that participation in online forums in MOOCs does lead to improved outcomes (Coetzee, Fox, Hearst, & Hartmann, 2014; He, Ma, Zhou, & Wu, 2018). Wise and Cui (2018) found that students who contributed to forums had higher passing rates than those who did not, although among the learners who passed the course there were no differences in course grades between contributors and non-contributors to the forums. Chiu and Hew (2018) found that it was actually the viewing of the messages rather than posting contributions to the forums that increased quiz scores. Onah, Sinclair and Boyatt (2015) compared tutor supported and peer supported forums, and found higher completion rates among students who chose the tutor supported mode. However the authors found that the peer supported forums had only occasional tutor involvement, while the additional facilities offered by the tutors were poorly used. Therefore the authors suggested that the outcomes may reflect learner motivation and commitment rather than tutor involvement. The theme of student motivation is one to which we will return in the Discussion. Continuing the theme of the impact of tutor involvement, Cho and Tobias (2016) report no difference in time spent in the virtual learning environment, course satisfaction or student achievement in an undergraduate online course, between courses offered without discussions, with discussion but no tutor involvement, and discussions with tutor involvement. The authors comment on the paucity of experimental data on the effect of discussion forums on student outcome.
The literature thus seems unclear in identifying real outcome benefits of tutor facilitated discussions, and lacking in experimental data comparing outcomes between online learning courses which offer tutor moderated discussion forums with those which do not offer tutor moderation, setting the scene for this paper. Although the courses we describe do not fit into the ‘massive’ category, and might fit the category defined as SPOCs (self Paced Open Courses) (Davidson, 2013), they do have much in common with MOOCs in that they are offered fully online to a wide geographical audience and include some form of assessment. The analysis we present in this paper may thus be of interest and relevance to a wide number of providers of online education.
The study design was a ‘natural experiment’ providing an opportunity to make comparisons between courses offered with and without tutor involvement, with the added strength of duplication in two subject areas.
Data were abstracted in December 2018, from the self-paced Exercise and Health course from its launch in October 2015, the self-paced Injury Prevention course from its launch in April 2017and the same two courses offered for new participants with tutor facilitation of discussion forums over a 4-week period in November 2018.
The outcomes chosen for exploration were the number of certificates gained, the number of quizzes completed and the grades gained in the quizzes, with the hypothesis that courses offered with tutor facilitation would lead to a higher proportion of certificates gained, and more quizzes completed, and with higher grades in the quizzes than when the same courses were offered without tutor facilitated discussions. The number of students posting to discussion forums and number of posts to the forums were also examined. The criteria for a certificate were clearly identified in the course material, and did not differ between the two delivery methods although they differed between the exercise and injury courses with the former requiring contributions to each of the forums and a pass in the quiz, while the latter required having accessed the resources in each section and a grade in the quiz. We did not assess student satisfaction with the course or other forms of feedback.
Analysis of the results used the configurable reports function of Moodle to describe the demographics of the participants of the four course offerings. Contributions to the discussion forums were counted through the course completion function of Moodle as well as by manual counts, and quiz results and certificates awarded were counted through the relevant functions on the course site. Counts and percentages were described, and chi square tests conducted to compare the stated hypotheses.
Table 1 shows that more students enrolled in the Exercise and Health than the Injury Prevention course, however numbers in the self-paced and the tutor-led delivery versions of the courses were similar. Gender patterns were similar between both modes of delivery, although the gender balance differed between the two content areas (relatively more females in the exercise courses). Students represented a wide range of ages and professional backgrounds, and came from between 21 and 57 countries in the various courses, and across a number of geographical regions. The students in the tutor-led courses were younger and more likely to be from Africa than those enrolled in the self-paced versions.
|Injury self-paced (N=38)||Injury tutor-led (N=51)||Exercise self-paced (N=392)||Exercise tutor-led (N=345)|
|F||13 (34%)||20 (39%)||233 (59%)||178 (52%)|
|M||24 (63%)||31 (61%)||140 (36%)||146 (42%)|
|Year of Birth|
|<1970||6 (16%)||3 (6%)||102 (26%)||46 (13%)|
|1970-1979||8 (21%)||12 (24%)||84 (21%)||66 (19%)|
|1980-1989||19 (50%)||15 (29%)||94 (24%)||112 (32%)|
|1990+||4 (11%)||20 (39%)||102 (26%)||114 (33%)|
|Medical practitioner - clinical||2 (5%)||16 (31%)||45 (11%)||63 (18%)|
|Medical practitioner - non clinical||4 (11%)||8 (2%)||11 (3%)|
|Nurse/Midwife||2 (5%)||2 (4%)||31 (8%)||8 (2%)|
|Other||12 (32%)||6 (12%)||66 (17%)||39 (11%)|
|Other health professional||16 (42%)||11 (22%)||135 (34%)||153 (44%)|
|Student||2 (5%)||14 (27%)||93 (24%)||66 (19%)|
|UK/Ireland||3 (8%)||6 (12%)||224 (57%)||73 (21%)|
|Australia/NZ/USA/Canada||9 (24%)||1 (2%)||32 (8%)||38 (11%)|
|Europe||3 (8%)||2 (4%)||47 (12%)||48 (14%)|
|Africa||5 (14%)||19 (37%)||24 (6%)||73 (21%)|
|Latin America||2 (6%)||4 (8%)||4 (1%)||42 (12%)|
|Indian subcontinent||8 (22%)||12 (24%)||6 (2%)||15 (4%)|
|Other||6 (17%)||1 (2%)||2 (1%)||7 (2%)|
Table 2 explores student participation in the forums. The number of posts made by students was higher during the tutor-led courses compared with the self-paced versions, as was the number of discussion threads with more than one post over the time that the courses ran in the tutor-led versions.
|Injury self-paced (N=38)||Injury tutor-led (N=51)||Exercise self-paced (N=392)||Exercise tutor-led (N=345)|
|Student posts to discussion/reflection forums|
|N students who posted||N posts||N students who posted||N posts||N students who posted||N posts||N students who posted||N posts|
|Forum 1||4 (11%)||4||10 (20%)||14||48 (12%)||48||54 (16%)||99|
|Forum 2||2 (5%)||2||9 (18%)||10||42 (11%)||42||49 (14%)||73|
|Forum 3||2 (5%)||2||8 (16%)||9||42 (11%)||42||45 (13%)||60|
|Forum 4||2 (5%)||2||7 (14%)||7||42 (11%)||42||44 (13%)||51|
|Posted to all forums||2 (5%)||5 (10%)||42 (11%)||43 (12%)|
|Posted to any forum||4 (11%)||10||11 (22%)||40||48 (12%)||174||54 (16%)||283|
|N discussion threads with more than one student response||0||5||1||46|
Table 3 explores course outcomes, and shows that in both content areas there were minimal and not statistically significant differences between the self-paced and tutor-led versions. This applied to the outcomes of completing the quiz, the grades obtained, and gaining a certificate of completion.
|Outcome criterion||Theme||Group||N (% of each group total)||Chi square||p-value|
|Completed quiz||Injury||Self-paced||9 (24%)||0.2155||0.64|
|Completed quiz||Exercise||Self-paced||64 (16%)||0.4814||0.49|
|Quiz score 8+/11||Injury||Self-paced||4 (11%)||0.0335||0.85|
|Quiz score 8+/10||Exercise||Self-paced||55 (14%)||0.0843||0.77|
|Certificates awarded*||Injury||Self-paced||6 (16%)||0.2153||0.64|
|Certificates awarded**||Exercise||Self-paced||44 (11%)||0.0012||0.97|
There was a trend towards higher quiz scores among younger participants in both versions of the exercise course.
It appears that tutor led discussions have increased the number of discussion posts, as well as the numbers of discussions with interaction between student and tutor reflected by the number of threads with more than one post. However, there were no differences in the course outcomes in terms of quizzes completed, marks gained, or course completion certificates awarded between the versions of the courses with and without online tutors to lead the discussions. Although there were some differences between the two content areas in terms of the outcomes measured, the results of no difference between delivery method with and without tutors are consistent between the two content areas.
None of the courses attracted MOOC type numbers –the courses were not delivered by a recognised MOOC provider and were not linked to any identifiable professional organisation. Numbers were much larger for the exercise courses, where the social media footprint for advertising was larger due to the established recognition and position of Exercise Works! in the social media space. Although not by design, the numbers in both versions of each content area were similar, allowing the comparisons we report.
The students in the exercise courses were more likely to be female (59% and 52% in the exercise courses compared with 33% and 39% in the injury courses) possibly a refection of the interest in exercise among physiotherapists a predominantly female profession. Students in the exercise courses were less likely to come from the global south than those in the injury courses (12% and 38% in the exercise courses came from Africa, Latin America or the Indian subcontinent compared with 39% and 69% in the injury courses), probably a result of the social media advertising footprint. However, both sets of courses attracted students from a wide range of professional groups in the health field, and from a wide geographical area. Younger students were more likely to complete the quiz with a high mark, and since the tutor-led groups were younger than the self-paced group we might have expected higher marks in the tutor-led groups.
Discussion forums were available in each of the four course iterations, with invitations to reflect on a question related to the content in the resources provided in each section. Certification in the exercise courses depended on having posted to the forum in each section, although this was not a requirement in the injury courses. Although we saw an increase in the proportion of students who posted to the tutor-led version on the injury course, this difference between tutor-led and self-paced version was smaller in the exercise course. It may be that obtaining a certificate was a motivation to post to the exercise forums. In both content areas, there were more discussion posts in the tutor-led than the self-paced versions, and a considerable increase in the threads with more than one student post indicating that discussions took place rather than just a posting of reflections. In the tutor-led versions, numbers of posts decreased over the time that the courses ran, consistent with our previous experience.
In July/August 2016, Physiopedia, in association with Exercise Works!, offered a 6-week MOOC ‘Physiotherapy, Exercise and Physical Activity’ (PEPA MOOC), using many of the same presentations as in the Exercise and Health courses we describe (Lowe, 2017). Online discussion forums were facilitated by expert tutors, as in the tutor-led course we describe. There were 8482 participants (86% physiotherapists/physical therapists) from 157 countries. The participants contributed 10799 discussion posts in total, with 561 (7%) of total participants contributing each week. 1050 quiz attempts were made, with 859 passes (12% and 10% of total participants respectively). Completing an evaluation was one of the criteria for a certificate, and 559 (7%) did so, although the number of certificates awarded was not included in the report. Although the course was more intensive than the ones we describe and the structure somewhat different, as was the audience and number of participants, the proportion of students achieving measurable outcomes were very similar to, or lower than, the proportions in both versions of the exercise course we report here.
We have previously reported completion (certification) rates of 15% in the early experience of the Peoples-uni Open Online Courses (Heller et al., 2017), and that results from MOOCs in different subject areas confirm similar, or lower, levels of course completion than these. The certification rates in this paper, are thus consistent with the experience of other online courses outside the for-credit education system.
Our findings are consistent with other experiments, such as those by Cho and Tobias (2016) and Onah et al. (2015) in finding no difference in student outcome as a result of tutor-led discussions in online education. Therefore Onah et al. (2015) suggested that student motivation might be more important in predicting success. Similarly, Brooker, Corrin, de Barba, Lodge and Kennedy (2018) reported that in a professional development MOOC (which would be consistent with our situation), professional development motivation contributed to the final grade, and their review of the literature emphasised the importance of motivation in student participation and performance. Although we did not measure motivation, our results lead us to conclude that motivation to complete the course is likely to be a confounding factor, influencing both commitment to participation in discussion forums and the outcomes of grades and course completion.
In light of little discernible difference in outcome between tutor-led and non tutor-led courses, the utility of the time cost to tutors is in question. The advantages of having courses available both for enrolment at any time and also for later study after the end of a timetabled course, rather than only for the period over which discussions are timetabled, adds another possible benefit of self-paced learning. Despite most MOOCs being offered to a timetable with tutors, self-paced MOOCs are also available, in 2015 it was stated that 20% of all MOOCs were in this form (Shah, 2015).
The structure of our online courses is similar to others in the xMOOC category, although we do not rely on the more usual tutor generated video presentations (Zhan et al., 2015). There does not seem to have been a vigorous debate about the merits of videos compared with other methods of information presentation in the literature. Peoples-uni offers its own certification, and does not articulate with any accredited educational or professional provider to provide academic or professional credit. Thus we should not extrapolate our findings beyond short online courses offered without academic credit.
MOOCs are increasingly available in health and medical areas, and appear to facilitate effective communication among international communities and disseminate knowledge across health specialities and across geographies (Goldberg & Crocombe, 2017). The ability we describe to reach a wide range of professional groups, as well as geographical settings including those in low- to middle-income countries is encouraging; this is especially as the reach of the internet is rapidly expanding
“In developed countries, slow and steady growth has increased the percentage of the world’s population using the Internet, from 51.3 per cent in 2005 to 80.9 per cent in 2018. In developing countries, more-sustained growth has shown an increase from 7.7 per cent in 2005 to 45.3 per cent at the end of 2018” (International Telecommunication Union News, 2018).
The injury course is also available for academic credit as part of a master’s programme, although of course there are major differences in delivery, mandatory discussion forums and assessment. We are encouraged to offer more of our academic for-credit courses on our Open Online Courses site, helping to meet the Peoples-uni major mission of capacity building in low- to middle-income countries. Learning the lessons from this evaluation, we will focus on self-paced delivery rather than delivery to a timetable with tutor led discussion forums.
The educational outcomes we report, although easily measurable, may not reflect the outcomes that matter in terms of meeting educational objectives (Otto, Bollmann, Becker & Sander, 2018). As stated in the introduction to the course material, the goal of the Exercise and Health course was described as
“An inter-disciplinary educational resource to help healthcare professionals understand the size of the problem of physical inactivity in populations globally, the role of physical inactivity in disease causation and the benefits of exercise in treatment and prevention, and to encourage us to perform and evaluate interventions to increase physical activity in our settings.”
Similarly, the Injury Prevention course goal was
“…designed to learn how to collect information on the burden of injury in your setting, understand the causes and risk factors for injury, and develop and evaluate intervention programs relevant to your setting. This will be underpinned by the principles and characteristics of a public health approach to prevention”.
While the outcome measures we have used can assess knowledge and commitment, the use of MCQs provides only a limited assessment and depends on the quality of the questions. Commitment was assessed by course completion, but any follow-up actions resulting from the courses were not assessed. Nevertheless, our limited study objective of comparing outcomes between self-paced and tutor-led delivery mechanisms is not compromised by the use of the measurable outcomes we have chosen.
In addition, numbers in the courses are relatively small and may not be relevant to MOOCs with their different structures and size. The power to examine differences between the two Injury courses is very low, but the numbers in the exercise courses provide the power to examine a moderate effect size. However, the consistency in findings between the two different course content areas adds to the strength of the conclusions reached.
Our findings suggest that it is possible to achieve similar outcomes through self-paced learning compared with courses which include tutor-led discussions in online courses such as those we describe. This is consistent with other experimental data indicating little effect of tutor moderated discussion on student outcome, and leads us to suggest that student motivation may be a confounding factor, influencing both participation in discussion forums and student outcome. Although we are careful not to extrapolate too far beyond the professional development type of short online course and the outcomes that we describe, we feel that these results may be relevant to others designing online courses, including MOOCs. We encourage further experiments to explore the effect of various components of online courses on student outcome.
While Peoples-uni is not a major provider of not-for-credit online courses, and mainly supplies a niche market at present, the lessons learned from this evaluation will allow us to continue to develop, host and offer online courses to a wide geographical audience. We are encouraged to continue to offer courses on our Open Online Courses site, helping to meet the Peoples-uni major mission of capacity building in low- to middle-income countries. Learning the lessons from this evaluation, we will focus on self-paced delivery rather than delivery to a timetable with tutor led discussion forums.
Bates, T. (2014, October 13). Comparing xMOOCs and cMOOCs: philosophy and practice [Blog post]. Online Learning and Distance Education Resources. Retrieved from https://www.tonybates.ca/2014/10/13/comparing-xmoocs-and-cmoocs-philosophy-and-practice/
Brooker, A., Corrin, L., de Barba, P., Lodge, J., & Kennedy, G. (2018). A tale of two MOOCs: How student motivation and participation predict learning outcomes in different MOOCs. Australasian Journal of Educational Technology, 34(1). https://doi.org/10.14742/ajet.3237
Cavanaugh, J. K. & Jacquemin, S. J. (2015). A Large Sample Comparison of Grade Based Student Learning Outcomes in Online vs. Face-to-Face Courses. Journal of Asynchronous Learning Networks, 19(2). Retrieved from https://www.learntechlib.org/p/157858/
Chiu, T., K. F. & Hew, T. K., F. (2018). Factors influencing peer learning and performance in MOOC asynchronous online discussion forum. Australasian Journal of Educational Technology, 34(4). https://doi.org/10.14742/ajet.3240
Cho, M-H, & Tobias, S. (2016). Should Instructors Require Discussion in Online Courses? Effects of Online Discussion on Community of Inquiry, Learner Time, Satisfaction, and Achievement. International Review of Research in Open and Distributed Learning, 17(2). https://doi.org/10.19173/irrodl.v17i2.2342
Coetzee, D., Fox, A., Hearst, M. A., & Hartmann, B. (2014). Should Your MOOC Forum Use a Reputation System? Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 1176-1187). https://doi.org/10.1145/2531602.2531657
Creelman, A., & Witthaus, G. (2018) Facilitated MOOC support - closed bubbles in an open sea. In D. Jansen; L. Konings (Eds.). The 2018 OpenupEd Trend Report on MOOCs (pp. 31-34). Maastricht, NL: EADTU. Retrieved from https://tinyurl.com/2018OpenupEdtrendreport
Davidson, C. (2013, August 21). MOOC, SPOC, DOCC, Massive Online Face2Face Open . . . (Uh Oh!): Age of the Acronym [Blog post]. Hastac. Retrieved from https://www.hastac.org/blogs/cathy-davidson/2013/08/21/mooc-spoc-docc-massive-online-face2face-open-uh-oh-age-acronym
Fidalgo-Blanco, Á., Sein-Echaluce, M. L., García-Peñalvo, F. J., & Esteban-Escaño, J. (2014). Improving the MOOC learning outcomes throughout informal learning activities. In F. J. García-Peñalvo (Ed.). Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM 2014) (Salamanca, Spain, October 1-3, 2014). (pp. 611-617). New York, NY, USA: ACM. https://doi.org/10.1145/2669711.2669963
Goldberg, L. & Crocombe, L. (2017). Advances in medical education and practice: role of massive open online courses. Advances in Medical Education and Practice, 8, 603-609. https://doi.org/10.2147/AMEP.S115321
Grandzol, C. J., & Grandzol PhD, J. R. (2006). Best Practices for Online Business Education. The International Review of Research in Open and Distributed Learning, 7(1). https://doi.org/10.19173/irrodl.v7i1.246
He, C., Ma, P, Zhou, L., & Wu, J. (2018). Is Participating in MOOC Forums Important for Students? A Data-driven Study from the Perspective of the Supernetwork. Journal of Data and Information Science 3(2), 62–77. https://doi.org/10.2478/jdis-2018-0009
Heller, R. F., Strobl, J., & Madhok, R. (2019). Online Education for Public Health Capacity Building in Low- to Middle-Income Countries. The International Review of Research in Open and Distributed Learning, 20(1). https://doi.org/10.19173/irrodl.v20i1.3927
Heller, R. F., Zurynski, R., Barrett, A., Oaiya, O., & Madhok, R. (2017). Open Online Courses in Public Health: experience from Peoples-uni. F1000Research 6:170. https://doi.org/10.12688/ f1000research.10728.2.
International Telecommunication Union News (2018). New ITU statistics show more than half the world is now using the Internet. ITU, the United Nations’ specialized agency for ICT. Retrieved from https://news.itu.int/itu-statistics-leaving-no-one-offline/
Jaggars, S., S. & Xu, D. (2013). Predicting Online Student Outcomes From a Measure of Course Quality. CCRC Working Paper No. 57. New York: Community College Research Center Teachers College, Columbia University.
Lowe, R. (2017). Physiotherapy, Exercise and Physical Activity Massive Open Online Course (PEPA MOOC) Evaluation Report. Physiospot, Feb 2017. Retrieved from https://www.physiospot.com/physiopedia/report-from-the-physiotherapy-exercise-and-physical-activity-mooc/
Manning, G. (2007). Self-Directed Learning: A Key Component of Adult Learning Theory. Journal of the Washington Institute of China Studies, 2(2), 104. Retrieved from https://www.bpastudies.org/bpastudies/article/view/38/78
Onah, D., F., O., Sinclair, J., E., & Boyatt, R. (2015). Forum Posting Habits and Attainment in a Dual-Mode MOOC. International Journal for Cross-Disciplinary Subjects in Education (IJCDSE), Special Issue 5(2). https://doi.org/10.20533/ijcdse.2042.6364.2015.0336
Otto, D., Bollmann, A., Becker, S., & Sander, K. (2018). It’s the learning, stupid! Discussing the role of learning outcomes in MOOCs, Open Learning: The Journal of Open, Distance and e-Learning, 33(3), 203-220. https://doi.org/10.1080/02680513.2018.1486183
Protopsaltis, S. & Baum, S. (2019). Does online education live up to its promise? A look at the evidence and implications for federal policy. George Mason University. Retrieved from https://mason.gmu.edu/~sprotops/OnlineEd.pdf
Siemens, G. (2005). Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology & Distance Learning, 2(1). Retrieved from http://www.itdl.org/journal/jan_05/article01.htm
Shah, D. (2015, December 28). MOOCs in 2015: Breaking Down the Numbers. EdSurge. Retrieved from https://www.edsurge.com/news/2015-12-28-moocs-in-2015-breaking-down-the-numbers
Sharif, A & Magrill, B. (2015) Discussion Forums in MOOCs. International Journal of Learning, Teaching and Educational Research, 12(1), 119-132. Retrieved from https://www.ijlter.org/index.php/ijlter/article/view/368
Stack, S. (2015). Learning Outcomes in an online vs traditional course. International Journal for the Scholarship of Teaching and Learning, 9(1). https://doi.org/10.20429/ijsotl.2015.090105
U.S. Department of Education (2009). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Washington, D.C.: Office of Planning, Evaluation, and Policy Development.
Wise, A., F. & Cui, Y. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs: content is key. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 330-339). Retrieved from https://dl.acm.org/citation.cfm?id=3170403&dl=ACM&coll=DL
Zhan, Z., Fong, P., S., W., Mei, H., Chang, X, Liang, T., Ma, Z. (2015). Sustainability Education in Massive Open Online Courses: A Content Analysis Approach. Sustainability, 7, 2274-2300. https://doi.org/10.3390/su7032274