This paper was presented at the ICDE Virtual Global Conference Week 2021: Upskilling and upscaling for quality Open, Flexible and Distance Learning (OFDL) during the week of 25–29 October 2021, and the contribution was preselected for publication in Open Praxis.
Learning and assessments are two interwoven domains which are complementary to each other. In that, the formative assessments serve as monitoring systems for both teachers and students to improve the capacity of teaching and learning respectively, while influencing the instructional practice. The fluid and uncertain environment due to COVID 19 pandemic brought about many challenges related to assessment practices, and hence the conventional paper-based assessments of universities were shifted towards online assessments.
Owing to the travel restriction imposed in Sri Lanka due to the COVID19 pandemic, The Open University of Sri Lanka (OUSL) decided to conduct all its academic activities through online mode, including the assessments, without any deferment. Hitherto this decision, all formative assessments of the Bachelor of Science (B.Sc.) degree programme were conducted in conventional, face-to-face paper-based mode using its network of regional and study centres. Hence, teachers had to redesign and re-structure their formative assessment practices to suit the online mode, amidst many challenges. To facilitate this function, faculty teachers were trained to develop online tests, examination guidelines were formulated, online mock assessments were conducted and students were informed of the new examination guidelines.
The BOU4300 Plant Physiology course is one of the undergraduate level four (second year) courses of the B.Sc. Degree programme offered by the Faculty of Natural Sciences. Originally the formative assessment component of this course comprised of three main assessment components, the Open Book Test, the Laboratory Assessment and the No Book Test. Shifting from its conventional paper-based formative assessment method, the Open Book Test (OBT) was redesigned and restructured to be conducted online, using its online course component in the Moodle Learning Management System (LMS). In doing so, the OBT was re-structured to be offered in three parts. The design of OBTs will be discussed in detail in the next section; ‘Methods’.
The aim of this study was to explore the impact of the newly introduced online formative assessments on students’ learning. In that the following two research questions were investigated:
Conceptual framework of this study is mainly based on Constructive Alignment (CA) of teaching and learning (Biggs, 1996; Biggs, 2003) (Figure 1). CA emphasise that students construct the meaning by themselves through relevant learning activities; and in order a successful learning to happen the teaching-learning activities and assessment task should be aligned with the Intended Learning Outcomes (ILO). Accordingly CA instruct teachers to design appropriate and different assessment tasks that allow students to achieve the intended range of knowledge, skills and competencies in their learning process.
In designing appropriate assessments, ILOs play a pivotal role. ILOs are the statements by which classified educational goals are listed and communicated to the students about the levels of cognitive engagement that are required from them. There are various frameworks that are used to derive ILOs, depending on the hierarchical nature of the ‘Knowledge’ dimension. Two widely used such frameworks are: the SOLO taxonomy (Biggs, 1999) and the Revised Bloom’s taxonomy (Krathwohl, 2002), from which the latter was used for this study, as the course is already designed and developed based on this framework.
The Revised Blooms taxonomy (Krathwohl, 2002) represents two dimensional framework: the ‘Knowledge’ dimension and the ‘Cognitive process’ dimension. The ‘Knowledge’ dimension includes four main knowledge categories: Factual, Conceptual, Procedural and Metacognitive knowledge, while the dimension of ‘Cognitive processing’ includes six major categories: Remember, Understand, Apply, Analyse Evaluate and Create. Anderson and Krathwohl (2001) proposed a Taxonomy Table, having Knowledge and Cognitive process in vertical and horizontal dimensions respectively. This table serves as an analytical tool to position the relevant ILOs with appropriate assessment tasks that ensure the ‘right’ skills and knowledge to be assessed.
Anderson and Krathwohl (2001) state that assessments are conducted for two basic reasons: 1) monitor student learning and make necessary adjustments in instruction, hence defined as formative assessment and 2) to assign a grade for students, which is defined as summative assessments. The emphasis of formative assessment is to help students learn while providing room for students to improve. The primary function of summative assessments is to make judgments of student learning at the end of the learning process. The purpose of this study is to design formative assessment tasks, to support students learn in an online environment.
Sadler (1998) emphasised that the main intention of formative assessments is to provide feedback to students’ performance which will in turn help to improve and accelerate learning. By examining hundreds of studies, Black and Wiliam (1998a, 1998b) state that formative assessments serve as determinants of students’ status of learning, having provisions of feedback on how to improve while correcting misconceptions, through the development of self assessment and peer assessment skills. Based on these concepts, formative assessments frame the basis for ‘assessments for learning’, which happens more than once during the process of learning, which enhance students’ motivation and commitment for learning, rather than at the end or summative assessments (Earl & Katz, 2006; Stiggins et al., 2007). Assessment for learning, is interactive with teachers, aligning instructions with outcomes, identifying learning needs, using assessments as investigating tools, while providing feedback and direction for students (Earl & Katz, 2006).
Moving further from assessments for learning, ‘assessment as learning’ allows student to take more responsibility of their own learning, through metacognition. When students actively engaged in learning through self-reflection, they make sense of information related to their prior knowledge and use it in new situation. This is the regulatory process in metacognition (Earl & Katz, 2006). Teachers’ role in assessment as learning is to provide regular and challenging opportunities to practice, so that students can become confident, competent self-assessors. To accomplish self-assessment and independent learning, descriptive feedback is very essential which creates conditions for metacognition, self-reflection and review of ideas.
Metacognition is one of the key components of the knowledge dimensions in Bloom’s taxonomy that are rarely reached in formal education (Blaschke & Hase, 2016). Moving away from the formal educational practices of pedagogy and andragogy, the ‘heutagogy’ expands upon the human agency in the learning process where metacognition is a key component. Heutagogy is a form of self-determined learning (Hase & Kenyon, 2000), where learners serve as “the major agent in their own learning, which occurs as a result of personal experiences” (Hase & Kenyon, 2007, p. 112). Learner-centeredness, learner autonomy are the key features of self determined learning, and in that teachers play a passive role involving only in facilitating function. The basic principles of heutagogic design are the capability, self-reflection and metacognition, double-loop learning, and nonlinear learning and teaching (Blaschke, 2012). Course design elements such as learning journals, action research and formative/summative assessments with feedback can be incorporated to support self reflective practice and metacognition (Blaschke, 2012) and thereby enhance self determined learning. In this study too, authors tried to supplement self determined learning (metacogntive knowledge in Bloom’s taxonomy) through formative assessments, while keeping the CA as the main frame of designing assessments.
As explained above, the outcome-based approach proposed by the constructive alignment of teaching and learning (Biggs, 1996; 1999; 2003) became the focus in re-designing of the online OBTs. In that the assessment tasks were aligned with the intended learning outcomes (ILO) of the course, which were already designed based on revised Bloom’s taxonomy (Krathwohl, 2002). While aligning the ILOs with assessment tasks, it was decided to conduct the OBT in three parts in capturing different ILOs. This structure also became useful in providing a unique adaptive approach for students, by gradually introducing the online assessment environment, as students are also not much familiar in facing online assessments. Feedback for all three OBTs was provided only in the online course.
The approach of this OBT was problem-based, and more specifically to investigate a national issue following the recent government decision to ban agro-chemicals in Sri Lanka from 2021. Students were asked to explore and analyse the issues and propose views and solutions to the problem, as an undergraduate student who is studying plant physiology. This is with the intention of allowing them to make connections between what is learned and real-world issues and thereby provoke metacognitive knowledge. Students were required to access the question paper from the Moodle course at a given time, complete the answers at home and submit online within five days in a typed-written or hand-written form, using the Moodle assignment drop box. Being a home based assessment and providing a longer window to submit the answer, OBT1 relied less on stable internet connection, less rigid restrictions imposed on assessment conditions, and provided ample opportunities and time for students to build the answer.
Marking and providing feedback to students were carried out online in the LMS, using an analytical rubric, having criteria to assess performance on: cognitive skills in defining the issue, comparative analysis, utilization of knowledge, arriving at solutions, as well as detection of plagiarism. Marked feedback rubric was made available to students after marking.
In an overall, OBT1 situated in between a conventional paper based and online mode, facilitating the gradual transition into online assessments, where the next two OBTs were conducted fully online.
The OBT2 was implemented entirely online using the Moodle Quizzes, in testing the factual, conceptual and procedural knowledge, while adopting examination related restrictions. Each student was given a randomly assigned set of questions from a question bank of 70 questions, comprised of multiple choice, multiple response, matching and missing words in Moodle quizzes. Since students were taking the assessments from home, they were supervised through Zoom video conferencing and their identity was checked before the examination. A password to enter into the test was announced five minutes before the test. Almost all examination administration procedures were followed during this online supervision. Students were given a specific time period to answer the test with one attempt, and the test was set to auto-submit at the end of the time duration. All questions were included with immediate feedback, which were to be viewed by the students after the test is closed.
This test purposely set to conduct after the OBT1, with the intension that students may refresh and revise their knowledge on subject content by going through various resources to build the answer for the OBT1.
This was based on analysing results of three internationally published journal articles in plant physiology, provoking higher order cognitive skills related to conceptual and procedural knowledge. Moodle Quiz component was utilised here again having three main questions only to be answered within a given time period, including three multiple choice questions and two short answer questions in each main question. Short answer questions were mainly to elicit students’ justification, explanation and analysis (Figure 2). This test was also supervised online in the same manner as in OBT2. Feedback to each question was also embedded providing awareness of their analysis, recognizing their mistakes/misconceptions and mastering the subject content.
After careful consideration of ILOs and assessment components, the ILOs that correspond to each OBT was mapped (Table 1), and the analysis of ILOs in terms of Taxonomy Table (Krathwohl, 2002) based on OBTs was outlined (Table 2), with the intention of providing an integrated learning experience for students.
|INTENDED LEARNING OUTCOME (ILO)||OBT1||OBT2||OBT3|
|ILO1||Explain the pathway of water and nutrient uptake and transport in plants.||√||√|
|ILO2||Explain the processes of photosynthesis in plants||√||√|
|ILO3||Illustrate the effect of enzymes on plant metabolism in a given situation.||√||√|
|ILO4||Assess how the changes in the external environment affect the water status, nutrient acquisition, and productivity of plants.||√||√|
|ILO5||Discuss how the outcomes of plant physiological research are applied in solving real world problems.||√||√|
|THE KNOWLEDGE DIMENSION||THE COGNITIVE PROCESS DIMENSION|
|1. REMEMBER||2. UNDERSTAND||3. APPLY||4. ANALYSE||5. EVALUATE||6. CREATE|
|Factual knowledge||ILO1, ILO2
|Conceptual knowledge||ILO1, ILO2,
Students’ experiences on online assessments were collected by means of quantitative and qualitative research methods. Hence a mixed approach of convergent parallel design method was used to collect and interpret quantitative and qualitative data (Creswell & Plano Clark, 2011). Therefore, the quantitative and qualitative data were collected more or less concurrently. Quantitative data were collected by means of two questionnaires, 1) getting perceptions on students’ learning through online assessments 2) getting general feedback about online assessments. The first questionnaire was developed adapting the Assessment Experience Questionnaire by Gibbs (2010). The two questionnaires comprised of close ended questions and three open ended questions. Both questionnaires were administered as Google forms, after finishing all three assessments. Interviews were conducted through Telephone/Zoom video conferencing to gather qualitative information. Triangulation of research methods was used to validate the results.
Sample - The quantitative study was conducted with all students registered for the course (240) and 20 students were interviewed, in order to gather qualitative data. Frequencies were computed for quantitative data and content analysis was carried out with qualitative data. Two data sets were analysed separately.
Students’ perceptions on the online assessments were gathered as soon as the test was completed and 180 students out of 240 provided their responses.
More than 70% of students stated that all three OBTs induced deep learning in the field of plant physiology. Students specifically mapped the different levels of learning and indicated that OBT1 and OBT3 focused more on testing critical thinking, problem solving and making judgments, while the OBT2 was more on recalling facts and understanding (Figure 3).
A majority of students (85%) agreed that the OBT1 prompted them to study various sources of information outside the course material to build their answer. This was also affirmed during interviews. Students said that they spent more time in gathering information through web resources/social networks/webinars conducted on this topic by other universities/Television discussions with experts/Newspapers/contacting people involved in agriculture, and also relating their own experience and knowledge to prepare the answer for OBT1. Students requested an extra ‘Zoom’ Day school to discuss more about their learning, and collaboratively review their viewpoints with regard to OBT1 and OBT3.
Students spoke more about OBT1 and OBT3 during the interviews and students expressed their reflections regarding the OBT1 as follows:
“This assessment reminded me of why we should study! Actually it was to apply our knowledge in practice and not only to pass exams. Even when I practically engage in activities at my home environment, for example, like when I treat my home plants, I always memorise what I have learned in plant physiology”. (SF1)
“OBT1 and OBT3 were very useful. Because these two tests made me feel that BSc students are not book worms”. (SF10)
“OBT1 on banning of agro-chemicals pushed me to explore more on this issue even after the assessment, and I am still continuing my exploration, by participating in webinars conducted by universities, browsing through the web, and updated information in mass media”. (SF3)
Overall, these results throw lights on self-determined learning (or heutagogy) as defined by Hase and Kenyon (2000). The characteristics of heutagogy: learner-centeredness, connectivity, self- reflection, double loop learning, and nonlinear learning of students (Blaschke & Hase, 2016) became quite apparent as students proactively engaged in exploring information using technology, connecting with people, sharing information as well as reflecting on their learning experiences. Requesting an additional Zoom discussion class specifically discloses their willingness to review their new learning experience and viewpoints, clarify issues, need for guidance, discover others experiences and co-construct knowledge with peers and tutor. Students’ reflections also typify how this assessment (OBT1) has influenced their values and belief systems, thus throwing light on double loop learning. Having continuing the knowledge exploration even after the assessment, driven by the assessments, shed glimpse towards ‘learner agency’, shifting the ownership of learning towards learner autonomy. Hence, these results endorse that the OBT1 has created more of a learning experience for students rather than measuring their attainment, as stated in the heutagogic approach of Hase and Kenyon (2000).
The Figure 4 displays the results obtained on five point Likert scale, where they were asked, whether the questions given in OBT3 spearheaded students to critically analyse their knowledge on plant physiology. In that about 60% of students have agreed that OBT3 has induced their critical thinking skills.
Accordingly, the OBT3 has tapped students’ ability to think critically is a crucial factor for scientific inquiry. This in turn, is a creativity or application of competency, which is a crucial cognitive skill needed for the development of young scientists. In achieving these abilities, research articles made available in advance as learning resources in the LMS, may also have paved the way for enhanced preparations. This also became apparent during the interviews too.
Almost all students who were interviewed spoke much about OBT3 and have become fascinated about it. Some of their thoughts about the OBT3 and consequences of this assessment were revealed as follows:
“OBT3 was intriguing even though I got low marks. It allowed me to utilize my knowledge on subject matter as a whole to analyse certain situations, specially the questions regarding the carbon enrichment graphs.” (SM8)
“OBT3 alerted me on worldwide plant physiology research. I checked the research papers posted in the LMS and got myself prepared before doing the OBT 3. Now, I am eagerly waiting to start and engage in laboratory practicals.” (SF16)
“Although I could not score well at the OBT3, having exposed to international research, stimulated me of the importance of studying the theories and concepts thoroughly before beginning the laboratory practicals”. (SM6)
In general, these interview results indicate that the OBT3 has pushed students into a challenging situation, and have aroused their motivation to engage in the laboratory components that are yet to come. These results also indicative of students’ understanding of the connectedness and their preparedness on theories and concepts, to engage with laboratory experiments, which will be helpful in analysing and interpreting of scientific knowledge.
The OBT3 was designed with questions that require skills of the cognitive domain such as ‘apply’ and ‘analysis’ level, and based on practical applications of three internationally published plant physiology research results. This OBT was also conducted before doing the laboratory component of this course, with the intention of showing students the importance of the basic procedural and conceptual knowledge in analysing research results, and also to make them involve in a challenging situation. The strategy used here is to expose the problem first, reversing the traditional educational practice, and use it to motivate students, as stated by White (2002). Hence, the objectives of having the OBT3 before the laboratory component became quite positive through the results of this study.
Students’ motivation and preparedness for laboratory practical is crucial to maximise the potential of practical work (Croker et al., 2010). According to Knutson et al. (2010) incorporation of research experience in undergraduate coursework, improve students’ engagement and motivation in learning science. In our study, it became apparent that incorporating research-focused experience in formative assessments has become meaningful and inspirational before starting the laboratory learning.
With regard to the feedback rubric offered for OBT1, except for 8% of students, all others agreed that it led them to self-assess their answers and to take corrective measures accordingly. Interestingly, more than 75% of students have stated that they paid careful attention to post- assessment feedback and tried understanding them. A similar proportion of students stated that this feedback prompted them to revisit the study material and take corrective measures, showing the characteristics of deep learning (Figure 5). Eighty percent (80%) of students indicated that they feel better prepared for the summative assessments after the OBT.
Accordingly, different types of feedback integrated with assessments, have become useful in building self-efficacy in students. Mainly the analytic rubric used for OBT1 has provided them a clear picture of why they got that score and have made them understand clearly about each level of performance, in an open ended assessment task. Especially, as these students are learning at a distance, and are in a solitary learning situation at that moment due to COVID 19 outbreak, feedback has encouraged them to review and revise their learning. Hence, students taking corrective measures through feedback may have led them in achieving the intended learning outcomes through remediation and assistance. On the whole, feedback provided in all three OBTs was perceived as positive learning tools by the students, and have provided them a powerful reinforcement, while adapting to a new learning and assessment situation.
Questionnaire results indicated that the presence of supervisor through the Zoom (OBT2 & OBT3) had given them the feeling of being in a supervised environment that upholds the integrity and authenticity of examinations (Figure 6).
Moreover, 80% of students stated that conducting assessments early in the morning have provided a conducive environment at home and has become so convenient for students who are employed. About 90% of students said they like to do formative assessments online in future. During the interview one student expressed her views regarding the online examination as follows:
“It was conducted in a student friendly way and was made with ease that almost every student could actively participate in it. Thank you so much for all the efforts that are being put by all the relevant course coordinators”. (SM8)
However, during the interviews, students who used mobile phones, expressed their concerns regarding connectivity problems; difficulty in viewing the graphs/images in a small screen; requirement of more time to understand the questions in small screens; and had preferred to sit for the online test using computer labs at OUSL regional centres in future. Interview results too indicated that disturbances at home environments are not very feasible for assessments.
“The network issue and the surrounding disturbances at home were the only hardships faced so far”. (SF7)
These results indicate that although online tests were welcomed by students, there should be favourable technical infrastructure that provides conducive environments to conduct online assessments, especially when conducting time bound online examinations. It is apparent through research (Hayashi et al. 2020; Gunawardhana, 2020; Aseslla, Dias & Palihawadana, 2020,) that Information Communication Technology reliability and outreach in Sri Lanka varies geographically. Therefore the key dimensions that need to be addressed for successful online learning are the stable high-speed Internet access, availability of technology and technical equipment. Hence, the expansion of existing computer facilities at OUSL regional centres and providing students with laptop computers would create a more conducive environment in conducting online learning and assessments more widely.
Careful designing of online formative assessment has induced deep learning in students, serving them as the major agent in their own learning. Analysis of student responses clearly indicated that the design of the assessment enhanced student engagement with the subject and prompted to think beyond it. Although, designing of the assessments was more biased towards the constructive alignment than the heutagogic principles or self determined learning, the outcome of the assessment tasks clearly indicated the presence of self-determined learning. This scenario also throw lights on the complementary effects of different learning approaches to achieve higher order learning skills, even in large classes. It further reinforces the concept of convergence of learning approaches in achieving the higher order cognitive skills. The feedback that was given after each assessment had served as a driving force in developing students’ capacities in learning through reflection. The intermission availed to students after each part of the OBT had paved the way for them to take corrective measures and perform better in the subsequent part. The scope of the subject matter in all three OBTs was the same, but the design of tests paved the way for the students to gradually explore their higher order learning skills. The metacognition developed through different phases of assessment have helped students to self evaluate their performance, reflect on what has been learned, build new personal insights and engage in self determined learning. Our future studies too will further look into the achievement of the core principles of self- determined learning.
Moreover, examiners need to pay special attention in setting up their questions so that they are legible across various devices including those with small screens used by students. Technically these questions need to be tested for the browser compatibility. Given that online tests are more amenable and offer certain advantages over the paper-based test, the authors are recommending that universities must pay attention to develop necessary infrastructure facilities and student accessibility of ICT for assessments. Many academics in the South Asian region are still developing their skills in online assessments and they mostly tend to set up online assessments with the paper-based mind-set. This preliminary approach often compromises the intended learning outcomes, exam integrity and authenticity. Both the substandard infrastructure and ill-trained human resources in online assessment systems may portray online exams as a failure and it is a real threat developing online exams as a formidable tool for learning.
Authors of this paper acknowledge Mr. Madusanka Wickramarathne for designing the figure of Constructive Alignment (Figure 1) used in this paper and Ms. Piyali Dias for the help given in conducting the interviews.
The authors have no competing interests to declare.
Aseslla, A. P. N., Dias, U. G. M. D., & Palihawadana, P. H. (2020). Effectiveness of Online Assessments in the Perspective of NSBM Green University Town, Sri Lanka. Proceedings of the International Conference on Business Innovation (ICOBI), Colombo, Sri Lanka.
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347–364. DOI: https://doi.org/10.1007/BF00138871
Biggs, J. (1999). What the student does: Teaching for enhanced learning. Higher education research & development, 18(1), 57–75. DOI: https://doi.org/10.1080/0729436990180105
Black, P., & Wiliam, D. (1998a). Assessment and Classroom Learning. Assessment in Education, 5(1), 7–71. DOI: https://doi.org/10.1080/0969595980050102
Blaschke, L., & Hase, S. (2016). Heutagogy: A Holistic Framework for Creating Twenty-First- Century Self-determined Learners. The Future of Ubiquitous Learning, 25. DOI: https://doi.org/10.1007/978-3-662-47724-3_2
Blaschke, L. M. (2012). Heutagogy and lifelong learning: A review of heutagogical practice and self-determined learning. The International Review of Research in Open and Distributed Learning, 13(1), 56–71. DOI: https://doi.org/10.19173/irrodl.v13i1.1076
Croker, K., Andersson, H., Lush, D., Prince, R., & Gomez, S. (2010). Enhancing the student experience of laboratory practicals through digital video guides. Bioscience Education, 16(1), 1–13. DOI: https://doi.org/10.3108/beej.16.2
Earl, L. M., & Katz, M. S. (2006). Rethinking classroom assessment with purpose in mind: assessment for learning, assessment as learning, assessment of learning. Manitoba Education, Citizenship & Youth.
Gunawardhana, L. K. P. D. (2020). Review of E-Learning as a Platform for Distance Learning in Sri Lanka. Education Quarterly Reviews, 3(2), 141–145. DOI: https://doi.org/10.31014/aior.1993.03.02.126
Hase, S., & Kenyon, C. (2000). From andragogy to heutagogy. Ultibase Articles, 5, 1–10. https://www.researchgate.net/publication/301339522_From_andragogy_to_heutagogy
Hase, S., & Kenyon, C. (2007). Heutagogy: A child of complexity theory. Complicity: An International Journal of Complexity and Education, 4(1), 111–119. DOI: https://doi.org/10.29173/cmplct8766
Hayashi, R., Garcia, M., Maddawin, A., & Hewagamage, K. P. (2020). Online Learning in Sri Lanka’s Higher Education Institutions during the COVID-19 Pandemic. ADB Briefs. DOI: https://doi.org/10.22617/BRF200260-2
Knutson, K., Smith, J., Wallert, M. A., & Provost, J. J. (2010). Bringing the excitement and motivation of research to students; Using inquiry and research-based learning in a year-long biochemistry laboratory: Part I-guided inquiry-purification and characterization of a fusion protein: Histidine tag, malate dehydrogenase, and green fluorescent protein. Biochemistry and molecular biology education: a bimonthly publication of the International Union of Biochemistry and Molecular Biology, 38(5), 317–323. DOI: https://doi.org/10.1002/bmb.20400
Krathwohl, D. R. (2002). A Revision of Bloom’s Taxonomy: An Overview. Theory into Practice, 41(4), Autumn. Ohio State University. DOI: https://doi.org/10.1207/s15430421tip4104_2
Ramsden, P. (2003). Learning to teach in higher education. Routledge. DOI: https://doi.org/10.4324/9780203507711
Sadler, D. R. (1998). Formative Assessment: revisiting the territory. Assessment in Education: Principles, Policy & Practice, 5(1), 77–84. DOI: https://doi.org/10.1080/0969595980050104
White, H. B. (2002). The Promise of Problem-based Learning. Biochemistry and Molecular Biology Education, 30(6), 419. DOI: https://doi.org/10.1002/bmb.2002.494030060146