Establishing a MOOC quality assurance framework – a case study

 

Establishing a MOOC quality assurance framework – a case study

Antonella Poce symbol, Francesca Amenduni symbol, Maria Rosaria Re symbol & Carlo De Medio

University of Roma Tre (Italy)

antonella.poce@uniroma3.it, francesca.amenduni@uniroma3.it,
mariarosaria.re@uniroma3.it & carlo.demedio@uniroma3.it

Abstract

The rapidly growing number of learning materials and repositories makes the issue of how to find the most relevant and best quality resources to be integrated in teaching and learning offers. Thus, effective quality assessment tools are more and more needed. In the present paper, a case-study focusing on quality assurance in a Virtual Mobility (VM) international project is presented. VM stands for ICT supported activities, organized at higher institutional level, that makes possible or facilitate international, collaborative experiences in a context of teaching and/or learning. Different approaches were combined to ensure the quality of a specific MOOC and the OERs created to promote VM. Three main macro-indicators were identified for OERs evaluation: 1. Quality, 2. Appropriateness, and 3. Technical aspects. Each project partner was invited to search, select and peer-assess OERs related to the skills necessary to be engaged in VM. First results of the peer-review activity and future directions to ensure OpenVM OERs and MOOC quality are presented.

Keywords: Virtual mobility, OER, MOOCs, quality assurance framework.

Reception date: 25 July 2019 • Acceptance date: 7 October 2019

DOI: https://doi.org/10.5944/openpraxis.11.4.1019

Introduction

Open Education is understood as a mode of carrying out education using digital technologies to provide alternative and less restrictive access routes to formal and non-formal education (Brown, 2008). This perspective is broad enough to enable a comprehensive view, thus encompassing for instance Open Educational Resources (OERs), Massive Open Online Courses (MOOCs), and recognition of open learning (Stracke, & Tan, 2018). According to OECD definition (2007), Open Educational Resources (OER) are “digital learning resources offered online freely (without cost) and openly (without licensing barriers) to teachers, educators, students, and independent learners in order to be used, shared, combined, adapted, and expanded in teaching, learning and research”. OERs are not only course components, but they can be entire courses, a museum collection, an open access journal or a reference work. Over time, the term has come to cover also content management software and content development tools. Finally, OERs include implementation resources such as standards and licensing tools for publishing digital resources, which allow users to adapt resources in accordance with their cultural, curricular and pedagogical requirements. Having said that, the term ‘OER’ is not synonymous with online learning, eLearning or mobile learning. Many OERs are also printable. What makes “Open” an Educational Resource is the feature of

“free availability on the public internet, permitting any users to read, download, copy, distribute, print, search, or link to the full texts of these articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose, without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself…“ (Chan et al., 2002).

OERs are free and they can be adapted and remixed, thus they can enhance collaboration and networking, fostering a sharing culture and respect for various cultures and believes (Tappeiner, DiSanto, & Lyons, 2019). According to Tuomi (2006), openness includes social and technical features: the social domain concerns the freedom to use, contribute and share the resources. Constraints to the social domain can be the copyright, the price of access or accessibility. Regarding the copyright challenge, the Creative Commons licence is the best-known and most often used open licence at present and offers a number of sharing options. Openness means also accessibility and it can depend on individual capabilities; for example, course contents may be freely available in a language the user does not understand, or the user may have a disability that precludes the individuals using the content.

A systematic approach to OERs quality assessment is particularly important to make decisions about which existing resources to include in a learning path. The rapidly growing number of learning materials and repositories makes the issue of how to find the most relevant and best quality resources. In addition, overlapping and competing standards, size of the search pool, quality of metadata are issues that different initiatives in the field of Open Education have tried to solve (Dietze et al., 2013; St. Lifer, 2018). Thus, there is urgency for effective search, discovery, and quality assessment tools. Quality can be defined as “[…] appropriately meeting the stakeholders’ objectives and needs which is the result of a transparent, participatory negotiation process within an organization” (Pawlowski, 2007). In the context of OERs, quality can for example mean that a teacher finds a suitable resource for his/her teaching. There are several alternative ways of approaching quality management in Open Education. Quality assurance can be a centrally or decentralized process, and the process may be open or closed (OECD, 2007; Jansen, Rosewell, & Kear, 2017). A common tool for the evaluation of the OERs is social ranking, which can be described as a form of crowd-sourced peer-review (Camilleri, Ehlers, & Pawlowski, 2014). The present paper will describe and critical discuss OERs and MOOC quality approach adopted in the Erasmus + project “OpenVM: Opening Education for Developing, Assessing and Recognising Virtual Mobility Skills in Higher Education”1.

Ensure OERs quality in the Open Virtual Mobility Project

Virtual mobility (VM) stands for ICT supported activities, organized at higher institutional level, that support or facilitate international, collaborative experiences in a context of teaching and/or learning (Tur, Urbina, Firssova, Rajagopal, & Buchem, 2018). Virtual Mobility (VM) has a great potential to contribute to the internationalization, innovation and inclusion in higher education. The barriers to physical mobility of educators and students, such as high costs, socio-economic, political and health-related issues, can be dramatically reduced by adding the virtual component to mobility and, making mobility accessible to everyone (EuroPACE, 2010). The OpenVM project is a Erasmus+ strategic partnership dedicated to create accessible opportunities for achievement of virtual mobility skills and to ensure higher uptake of virtual mobility in higher education in Europe. Despite numerous virtual mobility initiatives and projects in the past years, the uptake of virtual mobility in higher education is still low and the possibilities remain unknown to a large number of educators and students in Europe. Higher education teachers and students but also internationalization officers and other institutional stakeholders, need the skills, confidence and readiness to start, implement and develop virtual mobility actions. Open Virtual Mobility (OpenVM) has a great potential to contribute to the internationalization, innovation and inclusion in higher education.

The project lasts three years (2017-2020) and it aims at supporting higher education teachers and students in developing, assessing and recognising the skills needed for design, implement and participate in virtual mobility activities in line with Open Education principles.

The key outcome of the openVM project is the Virtual Mobility Learning Hub2 for achievement, assessment and recognition of virtual mobility skills as a central reference point. The VM Learning Hub will apply innovative tools and methods (such as open credentials, evidence-based assessment and matching algorithms for learning groups) and provide a set of open educational resources (OER), a massive online learning course (MOOC) and guidelines to support the design, implementation and participation in virtual mobility in higher education.

There is widespread skepticism of the quality of MOOCs and the learning methodologies used and there is evidence that supports this skeptical view (Margaryan, Bianco, & Littlejohn, 2015; Lowenthal & Hodges, 2015). Thus, the Quality Assurance Framework (QAF) is one of the pivotal aspects for the success of the Open Virtual Mobility (Atenas, & Havemann, 2014). Levels of quality can be defined from the most general aspects to the more specific ones (Fig. 1). Although each quality level interacts with each other, in this paper we will focus on two of the five levels presented in the figure 1: quality of the courses (in our case, MOOCs) and individual OERs quality.

OP-11-1019-F1.jpg
Figure 1: Levels of quality in Open Education – Adapted from Camilleri, Ehlers, & Pawlowski (2014).

In the context of the the OpenVM project, a Massive Open Online Course named OpenVM MOOC has been developed in order to promote students’ and teachers’ skills necessary to be involved in VM. The OpenVM is structured in eight miniMOOCs, corresponding to the eight key skills and related content necessary to be engaged in Virtual Mobility (Firssova & Rajagopal, 2018): 1. Intercultural Skills; 2. Collaborative learning; 3. Autonomy-driven learning; 4. Networked Learning; 5. Media and digital literacy; 6. Active self-regulated learning; 7. Open mindedness; 8. Knowledge of Virtual Mobility and Open Education. These knowledge and skills were identified by applying a group concept mapping methodology and involving 49 experts in the field of virtual mobility and/or open education, with experience in higher education as university professors or education management and support (Firssova & Rajagopal, 2018).

Three levels are then proposed for each miniMOOCs:

  1. foundation level: focused on knowledge acquisition;
  2. intermediate level: focused on knowledge application in a collaborative learning environment;
  3. advanced level: focused on self-reflection and meta-reflection;

Each miniMOOC has a pre-assessment activity: participants are required to fill in a quiz and, according to the score they obtained, they can be directed to the foundation level, intermediate level or advanced level. Each combination between the level and the miniMOOC is defined a subMOOC. Thus, the OpenVM MOOC is composed by 24 subMOOC, 8 miniMOOCs for 3 levels (Figure 2). Each subMOOC has different forms of assessment. In the foundation level and in the intermediate level there are mainly quizzes (e.g. multiple choices, true or false and drag and drop exercises), whilst in the advanced level there are also e-portfolio and peer-assessment activities. At the end of each subMOOC, participants obtain a badge that certifies the skills acquired in that specific subMOOC.

All the miniMOOC will contain approximately 9 Open Educational Resources (3 for the basic level, 3 for the intermediate level and 3 for the advanced level). In the OpenVM MOOC, OERs are considered the study material that participants could read, listen to, download and re-use for their personal purposes. OERs include slides, supplementary audio files, URLs to other resources, online articles and video lectures.

OP-11-1019-F2.jpg
Figure 2: The OpenVM MOOC structure.

Different approaches were combined to ensure the quality of the OpenVM MOOC and OpenVM OERs. At a more general level, quality assurance of the OpenVM MOOC is addressed through an iterative cycle of design, creation, implementation, and assessment, following the Design Based Research model – DBR (Barab & Squire, 2004). Salinas (2012) remarks that the DBR model has had an important uptake in Technology Enhanced Learning research as it is aimed at creating knowledge on the design, implementation and evaluation of the educational experience. It aspires to explore problems in real contexts requiring a solution in a particular context (de Benito & Salinas, 2016). Moreover, the DBR model has been argued to be suitable for the study of innovation, for which the contrast with the theoretical background and action observation in successive iterations is the strategy for knowledge creation (Brown, 1992; de Benito & Salinas, 2016; Shavelson, Phillips, Towne & Feuer, 2003). In the QA for the OpenVM Erasmus+ project, diverse tasks and instruments have been included following the DBR model. For each component (e-assessment, OERs, and MOOCs) the following phases (Piedra, Chicaiza, López, & Caro, 2015) are included:

  1. Assessment by partner (internal);
  2. Assessment by external experts;
  3. Assessment by pilot users;
  4. User testing assessment;
  5. Learning analytics.

Within this paper, we will present the results of the first phase, the assessment carried out by partners regarding OERs and future perspective on the other phases.

In the quality assurance of the OpenVM OERs, elements of the traditional peer-review with social rating were combined (Camilleri, Ehlers, & Pawlowski, 2014). In the OpenVM project, project partners can change role between reviewer and producer depending on the project phase, and this make the quality review process closer to a social rating practice. Partners were also provided with a rubric (Table 1) to assess OERs selected and produced by peers as in the traditional peer-review. Three macro-indicators have been identified for the OERs evaluation (Poce, Agrusti & Re 2015) to assess OERs to be included in the Open VM MOOC:

  1. Quality;
  2. Appropriateness;
  3. Technical aspects.

 

Table 1: OER rubrics adapted from ACHIEVE, 2011
Quality Appropriateness Technical aspects
  • Creator knowledgeable (Who is the creator and what kind of expertise and experience do they have?)
  • Creator authenticity (Are you reasonably certain that it is actually the work of the person claiming to be the author?)
  • Creator bias (What is the intended purpose? (Think educate/inform, sell something, entertain, change minds/behavior, even propaganda/hate speech)
  • Organization affiliation (What is the “hosting” organization and what kind of reputation do they have?)
  • Organization quality control (Does the hosting organization conduct any sort of quality control?)
  • Peer reviewed (Has it been through peer review?)
  • Material(s) currency (How recent or up-to-date is its content?)
  • Type of assessment (T/F; multiple choices; filling in the blanks; matching; open ended)
  • Clearness of structure and content
  • 8 badges topics (Intercultural skills, Collaborative learning, Autonomy-driven learning, Networked learning, Media and digital learning, Active Self-regulated learning, Open mindedness, VM knowledge)
  • Difficulty level (from Beginner: resource written with simple language, providing a general definition of the skill OR video that provides a general definition of the skill. Intermediate: resources written with a plain language that connect the skills to possible applications OR video that explain how that skills can be applied in certain situations. Advanced: resource written with complex or academic language that refers to real-undefined issues OR video that describe complexity and interconnection between the skill and other skills, ethical questions and so on.
  • Licensing status (What is its copyright and licensing status and how does that impact what you can do with it?)
  • Human accessibility (Is it accessible to people with disabilities?)
  • Remix or Edit (If you want to remix it, is the source file available, and in a format that you can edit?)
  • Technical accessibility (Is it accessible to people using different devices (multi-channel)?)
  • Technical Quality (in terms of graphis, sound, text layout)
  • Numbers of items in the e-assessment
Overall evaluation: (0 = not usable; 1 = limited; 2 = good; 3= superior).

Each macro-indicator was operationalised through sub-indicators (Table 1). By combining the answers on different sub-indicators, it is possible to provide a general overall evaluation of the OER (0=not usable; 1=limited; 2=good; 3=superior). For example, a resource can be considered weak if it is not recent neither peer-reviewed and/or accessible to people with disabilities. On the other hand a resource is considered superior if it covers one of the MOOC’s topics, if it is updated and its contents are clear organized and accessible to different kinds of target. The table was mainly inspired by a separate rubric for the evaluation of OERs created by ACHIEVE.org, a nonprofit education organization created in 1996 by a bipartisan group of governors and business leaders, fully recognized by international companies and institutions.3

OERs assessment and selection in the OpenVM Project

University Roma Tre is responsible for the Intellectual Output 6 of the OpenVM project, which includes organizing the process of OERs design, assessment and selection. Once the OERs assessment rubric presented in Table 1 was created, project partners were required to provide OERs in different formats (mainly texts and videos) and partners’ languages, following the quality guidelines of the OERs assessment rubric. OERs contents had to be connected to the eight skills necessary to be engaged effectively in virtual mobility. Each skill was assigned according to every partners’ specific background and expertise. In order to support OERs identification, Roma Tre team proposed different types of OERs repositories on the web. Not only repositories created by formal educational institutions, such as universities, but also other informal and no formal institutions databases (e.g. TedX video repository) were suggested to be used on purpose.

The process was organized as follows:

  1. Each partner had to identify at least 9 OERs (3 for the foundation level, 3 for the intermediate level, 3 for the advanced level) related to one of the eight skills of the OpenVM MOOC. Each partner was responsible to identify OERs within a certain area in order to cover all MOOCs’ contents. Partners had to download the OERs in a spreadsheet created on Google Sheets. The use of Google Sheets allowed partners to comment, insert feedback, and propose alternative contents.
  2. OERs selected were peer-assessed by another partner of the project. Peer-assessors could add comments, feedback, and propose alternative OERs. This way, partners had the opportunity to discuss suitability or non-suitability of the OERs selected to be included in the OpenVM MOOC.
  3. In the last phase, during a face-to-face workshop organised in February 2019 in Heerlen, partners worked in small groups of two or three people. Each group was invited to organize the OERs selected and assessed into a template for a miniMOOC design provided by the Roma Tre team.

The process was thought to guarantee each partner’s participation in the selection and assessment of the OERs and, eventually, in the OpenVM MOOC design.

First results of the OERs selection and assessment

Since the peer-assessment process is in progress, first results related only to six skills area are presented in Table 2.

 

Table 2: A description of the OERs selected and assessed
Skills to promote in the VM MOOC Level Number of OERs for each level Date of the creation of the resources (in average) Quality overall peer assessment
Autonomy driven learning Foundation 4 2007 (from 2001 to 2018) 80% good; 20% superior.
Intermediate 4
Advanced 3
Mainly for teachers 6
Self-regulated learning Foundation 3 2014 (from 2010 to 2017) 80% good; 20% superior.
Intermediate 3
Advanced 3
Collaborative learning Foundation 3 2013 (from 2010 to 2017) 20% not usable; 20% limited; 20% good; 20% superior;
Intermediate 3
Advanced 3
Networked learning Foundation 3 2013 (from 2008 to 2017) 40% limited; 40% good; 20 % superior;
Intermediate 3
Advanced 3
Open-mindedness Foundation 2 2011 (from 2004 to 2018) 20% limited; 30% good; 50% superior
Intermediate 3
Advanced 5
Intercultural skills Foundation 5 2014 (from 2008 to 2018) 20% limited; 30% good; 50% superior
Intermediate 5
Advanced 5

This process was useful to exclude resources with poor overall quality. Only resources that obtained a “good” or “superior” overall assessment were included into the miniMOOCs. In case of positive assessment, partners had also the opportunity to include scoring descriptions, commenting briefly, like in the following extract (E1):

E1 “This resource is a good way to start a discussion about similar processes and the implications in the educational situation!” (Peer assessor of the Intercultural skills OERs)

On the other hand, when resources received a negative evaluation, the use of the Google spreadsheet allowed partners to discuss further and find a shared final decision (Figure 3).

OP-11-1019-F3.jpg
Figure 3: Screenshot – Google Spreadsheet used to discuss OERs assessment.

Partners had approximately three months to complete this work, from November 2018 to February 2019. Once partners achieved a final agreement related to the course contents, learning objectives and assessment methods, then OERs, e-assessment and instructions were uploaded on the learning hub.

Conclusion and future steps

In the context of Open Education and Virtual Mobility, quality assessment needs to be taken into account. The present work describes the Quality Assurance framework for OERs within a specific MOOC created in the Erasmus+ Open Virtual Mobility Project. The University of Roma Tre research group firstly has developed a rubric for the OERs quality assessment. Then, each project partner was invited to search and assess OERs related to the eight skills identified by (Firssova & Rajagopal, 2018) necessary to be engaged in Virtual Mobility. Partners had to download the OERs in a spreadsheet created on Google Sheets. The OERs selected were peer-assessed by other partners in a joint project meeting. This way all the partners contributed to build the OpenVM MOOC (Camilleri, Ehlers, & Pawlowski, 2014), combining elements of the traditional peer-review with social rating activities.

Having said that, further work is needed to ensure OERs and MOOCs quality. As quality is not a generic concept, user behavior and comments can indicate the quality of MOOCs and OERs in relation to the learner context. We will carry out a pilot phase from 2019 to 2020 and we will collect different kinds of data regarding user interaction with the miniMOOCs, part of the MOOC under investigation, and the OERs used per each miniMOOC. As a strategic management decision, the OpenVM project will have to consider the role of learning analytics to gather and assess data from the MOOC and all single elements. In addition, different forms of data collection will be combined: user comments, recommendation and ratings. The insights will be used to improve OERs and the OpenVM MOOC quality and design, following an iterative process, as indicated by the Design Based Research approach.

Initial results regarding users’ OERs assessment were collected (Poce, Re, Amenduni, & Valente, 2019). In future phases, OVM MOOC users will be asked to provide their evaluation of the selected OERs.

Acknowledgements

A. Poce coordinated the research presented in this paper. Research group is composed by the authors of the contribution that was edited in the following order: A. Poce (Introduction; Conclusion and future steps) and Discussion), F. Amenduni (Ensure OERs quality in the Open Virtual Mobility Project); M. R. Re (OERs assessment and selection in the OpenVM Project), C. De Medio (First results of the OERs selection and assessment).

References

ACHIEVE (2011). Rubrics for Evaluating Open Education Resource (OER) Objects. Retrieved from: http://achieve.org/files/AchieveOERRubrics_1.pdf
Atenas, J., & Havemann, L. (2014). Questions of quality in repositories of open educational resources: a literature review. Research in Learning Technology, 22, https://doi.org/10.3402/rlt.v22.20889
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The journal of the learning sciences, 13(1), 1–14.
Brown, A.L. (1992). Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141–178.
Brown, J. S. (2008). Opening up education: The collective advancement of education through open technology, open content, and open knowledge. Mit Press.
Camilleri, A.F., Ehlers, U.D., & Pawlowski, J. (2014). State of the art review of quality issues related to open educational resources (OER). Luxembourg: Publications Office of the European Union.
Chan, L., Cuplinskas, D., Eisen, M., Friend, F., Genova, Y., Guédon, J. C., … & La Manna, M. (2002). Budapest open access initiative. Retrieved by http://www.budapestopenaccessinitiative.org
De Benito Crosetti, B., & Salinas, J.M. (2016). La investigación basada en diseño en Tecnología Educativa. Revista Interuniversitaria de Investigación en Tecnología Educativa, 0. Retrieved from http://revistas.um.es/riite/article/view/260631
Dietze, S., Sanchez-Alonso, S., Ebner, H., Qing Yu, H., Giordano, D., Marenzi, I., & Pereira Nunes, B. (2013). Interlinking educational resources and the web of data: A survey of challenges and approaches. Program, 47(1), 60–91 https://doi.org/10.1108/00330331211296312
EuroPACE (2010). Interests – Virtual mobility. Retrieved November 15, 2010, from http://www.europace.org/interest3.php
Firssova, O., & Rajagopal, K. (2018). Open VM Competence Framework. Retrieved from https://www.openvirtualmobility.eu/topics/outputs
Jansen, D., Rosewell, J., & Kear, K. (2017). Quality frameworks for MOOCs. In Open Education: from OERs to MOOCs (pp. 261–281). Berlin, Heidelberg: Springer.
Lowenthal, P., & Hodges, C. (2015). In search of quality: Using Quality Matters to analyze the quality of massive, open, online, courses (MOOCs). The International Review of Research in Open and Distributed Learning, 16(5). https://doi.org/10.19173/irrodl.v16i5.2348
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education 80, 77–83. http://dx.doi.org/10.1016/j.compedu.2014.08.005
OECD (2007). Giving knowledge for free: The emergence of open educational resources. Retrieved from http://www.oecd.org/education/ceri/38654317.pdf
Pawlowski, J.M. (2007). The quality adaptation model: adaptation and adoption of the quality standard ISO/IEC 19796-1 for learning, education, and training. Journal of Educational Technology & Society, 10(2), 3–16.
Piedra, N., Chicaiza, J., López, J., & Caro, E.T. (2015, March). Towards a learning analytics approach for supporting discovery and reuse of OER an approach based on Social Networks Analysis and Linked Open Data. In 2015 IEEE Global Engineering Education Conference (EDUCON) (pp. 978–988). IEEE.
Poce, A., Agrusti, F., & Re, M.R. (2015) Sviluppo di uno strumento di valutazione delle risorse aperte (OERs). Analisi dei dati raccolti: abitudini nell’uso della tecnologia e di scrittura. CADMO, Giornale Italiano di Pedagogia sperimentale. An International Journal of Educational Research, 23(2), 86–92.
Poce, A., Re, M.R., Amenduni, F., & Valente, M. (2019). MOOC and OERs in a Virtual Mobility Experience. In EMOOCs-WIP (pp. 181–186).
Salinas, J. (2012). La investigación ante los desafíos de los escenarios de aprendizaje futuros. RED, Revista de Educación a Distancia, 32. Retrieved from http://www.um.es/ead/red/32
Shavelson, R.J., Phillips, D.C., Towne, L., & Feuer, M.J. (2003). On the science of education design studies. Educational Researcher, 32(1), 25–28. https://doi.org/10.3102/0013189X032001025
St. Lifer, E. (2018, January 11). Five Ed-Tech Elephants: From OER to equitable access, challenges to the digital revolution persist. School Library Journal, 1, 12. Retrieved from https://www.slj.com/?detailStory=the-five-ed-tech-elephants-in-the-prek-12-classroom-opinion
Stracke, C.M., & Tan, E. (2018). The quality of open online learning and education: Towards a quality reference framework for MOOCs.
Tappeiner, L., DiSanto, J.M., & Lyons, K. (2019). Expanding Access to Education Through Open Educational Resources (OERs). In Developing Educational Technology at an Urban Community College (pp. 135–146). Palgrave Macmillan, Cham.
Tuomi, I. (2006). Open Educational Resources: What they are and why do they Matter. Retrieved from https://www.oecd.org/edu/oer
Tur, G., Urbina, S., Firssova, O., Rajagopal, K., & Buchem, I. (2018). Open Virtual Mobility: A Learning Design 4 SRL. In EDEN Research Workshop 2018.

Footnotes

1https://www.openvirtualmobility.eu/es_ES/

2https://hub.openvirtualmobility.eu

3https://www.achieve.org/contributors

 

Papers are licensed under a Creative Commons Attribution 4.0 International License

Refbacks

  • There are currently no refbacks.


Bookmark and Share