An Evaluation of Online Proctoring Tools

Mohammed Juned Hussein, Javed Yusuf, Arpana Sandhya Deb, Letila Fong, Som Naidu

Abstract


COVID’19 is hastening the adoption of online learning and teaching worldwide, and across all levels of education. While many of the typical learning and teaching transactions such as lecturing and communicating are easily handled by contemporary online learning technologies, others, such as assessment of learning outcomes with closed book examinations are fraught with challenges. Among other issues to do with students and teachers, these challenges have to do with the ability of teachers and educational organizations to ensure academic integrity in the absence of a live proctor when an examination is being taken remotely and from a private location. A number of online proctoring tools are appearing on the market that portend to offer solutions to some of the major challenges. But for the moment, they too remain untried and tested on any large scale. This includes the cost of the service and their technical requirements. This paper reports on one of the first attempts to properly evaluate a selection of these tools and offer recommendations for educational institutions. This investigation, which was carried out at the University of the South Pacific, comprised a four-phased approach, starting with desk research that was followed with pilot testing by a group of experts as well as students. The elimination of a tool in every phase was based on the ‘survival of the fittest’ approach with each phase building upon the milestones and deliverables from the previous phase. This paper presents the results of this investigation and discusses its key findings.


Keywords


COVID’19; online exam; online proctoring; remote teaching; ICT; evaluation; flexible learning

Full Text:

HTML PDF XML

References


Alessio, H. M., Malay, N., Maurer, K., Bailer, A. J., & Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning, 21(1), 146-161. http://dx.doi.org/10.24059/olj.v21i1.885

Barnes, C., & Paris, B. L. (2013). An analysis of academic integrity techniques used in online courses at a southern university. In Northwest Decision Sciences Institute Annual Meeting Proceedings.

Beck, V. (2014). Testing a model to predict online cheating—Much ado about nothing. Active learning in higher education, 15(1), 65-75. https://doi.org/10.1177%2F1469787413514646

Bedford, W., Gregg, J., & Clinton, S. (2009). Implementing technology to prevent online cheating: A case study at a small southern regional university (SSRU). MERLOT Journal of Online Learning and Teaching, 5(2), 230-238.

Berkey, D., & Halfond, J. (2015, July 20). Cheating, Student Authentication and Proctoring in Online Programs. New England Journal of Higher Education. Retrieved from https://nebhe.org/journal/cheating-student-authentication-and-proctoring-in-online-programs/

Brown, V. (2018). Evaluating technology to prevent academic integrity violations in online environments. Online Journal of Distance Learning Administration, 21(1). Retrieved from https://www.westga.edu/~distance/ojdla/spring211/brown211.html

Carstairs, J., & Myors, B. (2009). Internet testing: A natural experiment reveals test score inflation on a high-stakes, unproctored cognitive test. Computers in Human Behavior, 25(3), 738-742. https://doi.org/10.1016/j.chb.2009.01.011

Casey, D. M. (2008). A journey to legitimacy: The historical development of distance education through technology. TechTrends, 52(2), 45. https://doi.org/10.1007/s11528-008-0135-z

CAUDIT/ACODE (2020). CAUDIT/ACODE Forum on e-exams. Retrieved April 14, 2020, from https://www.acode.edu.au/

Cramp, J., Medlin, J. F., Lake, P., & Sharp, C. (2019). Lessons learned from implementing remotely invigilated online exams. Journal of University Teaching & Learning Practice, 16(1), 10. Retrieved from https://ro.uow.edu.au/jutlp/vol16/iss1/10/

Crowdmark (2020). Crowdmark is a collaborative online grading and analytics platform. Retrieved April 14, 2020, from https://crowdmark.com/

Etter, S., Cramer, J. J., & Finn, S. (2006). Origins of academic dishonesty: Ethical orientations and personality factors associated with attitudes about cheating with information technology. Journal of Research on Technology in Education, 39(2), 133-155. https://doi.org/10.1080/15391523.2006.10782477

Foster, D., & Layman, H. (2013). Online proctoring systems compared. Retrieved from https://caveon.com/wp-content/uploads/2013/03/Online-Proctoring-Systems-Compared-Mar-13-2013.pdf

Gautam, M. (2017, December 20). 3 types of online proctoring services and how to select the best for hiring. Hackerearth. Retrieved from https://www.hackerearth.com/blog/talent-assessment/online-proctoring-for-hiring-developer

Grijalva, T. C., Kerkvliet, J., & Nowell, C. (2006). Academic honesty and online courses. College Student Journal, 40(1), 180-185.

Harmon, O. R., & Lambrinos, J. (2008). Are online exams an invitation to cheat? The Journal of Economic Education, 39(2), 116-125. https://doi.org/10.3200/JECE.39.2.116-125

Harmon, O. R., Lambrinos, J., & Buffolino, J. (2010). Assessment design and cheating risk in online instruction. Online Journal of Distance Learning Administration, 13(3). Retrieved from https://www.westga.edu/~distance/ojdla/Fall133/harmon_lambrinos_buffolino133.html

Hollister, K. K., & Berenson, M. L. (2009). Proctored versus unproctored online exams: Studying the impact of exam environment on student performance. Decision Sciences Journal of Innovative Education, 7(1), 271-294. https://doi.org/10.1111/j.1540-4609.2008.00220.x

Jose, S. (2016, December 15). Online proctoring is trending: Here is all you must know. Talview. Retrieved from https://blog.talview.com/a-complete-guide-to-online-remote-proctoring

Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, reactions, and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology, 29(4), 555-572. https://doi.org/10.1007/s10869-014-9343-z

King, C. G., Guyette Jr, R. W., & Piotrowski, C. (2009). Online exams and cheating: An empirical analysis of business students' views. Journal of Educators Online, 6(1), 1-11.

Kinney, N. E. (2001). A guide to design and testing in online psychology courses. Psychology Learning & Teaching, 1(1), 16-20.

Kolski, T., & Weible, J. L. (2019). Do Community College Students Demonstrate Different Behaviors from Four-Year University Students on Virtual Proctored Exams? Community College Journal of Research and Practice, 43(10-11), 690-701. https://doi.org/10.1080/10668926.2019.1600615

Ladyshewsky, R. K. (2015). Post-graduate student performance in ‘supervised in-class’ vs. ‘unsupervised online’ multiple choice tests: implications for cheating and test security. Assessment & Evaluation in Higher Education, 40(7), 883-897. https://doi.org/10.1080/02602938.2014.956683

Lanier, M. M. (2006). Academic integrity and distance learning. Journal of criminal justice education, 17(2), 244-261. https://doi.org/10.1080/10511250600866166

Mitra, S., & Gofman, M. I. (2016). Towards Greater Integrity in Online Exams. In Americas Conference on Information Systems (AMCIS). Association For Information Systems.

Moten Jr, J., Fitterer, A., Brazier, E., Leonard, J., & Brown, A. (2013). Examining online college cyber cheating methods and prevention measures. Electronic Journal of E-learning, 11(2), 139-146.

Prisacari, A. A., & Danielson, J. (2017). Computer-based versus paper-based testing: Investigating testing mode with cognitive load and scratch paper use. Computers in Human Behavior, 77, 1-10. https://doi.org/10.1016/j.chb.2017.07.044

Proctorio (2019). A Comprehensive Learning Integrity Platform. Retrieved from https://www.proctorio.com/

Richardson, R., & North, M. (2013). Strengthening the trust in online courses: a common sense approach. Journal of Computing Sciences in Colleges, 28(5), 266-272.

Rose, C. (2009). Virtual proctoring in distance education: An open-source solution. American Journal of Business Education, 2(2), 81-88.

Schultz, M. C., Schultz, J. T., & Gallogly, J. (2007). The management of testing in distance learning environments. Journal of College Teaching & Learning, 4(9), 19-26. https://doi.org/10.19030/tlc.v4i9.1543

Sietses, L. (2016). White Paper Online Proctoring. Questions and answers about remote proctoring. SURFnet. Retrieved from https://www.surf.nl/files/2019-04/whitepaper-online-proctoring_en.pdf

Tao, J., & Li, Z. (2012). A Case Study on Computerized Take-Home Testing: Benefits and Pitfalls. International Journal of Technology in Teaching & Learning, 8(1), 33-43.

Watson, G., & Sottile, J. (2010). Cheating in the digital age: Do students cheat more in online courses? Online Journal of Distance Learning Administration, 13(1). Retrieved from https://www.westga.edu/~distance/ojdla/spring131/watson131.html

Wellman, G. S., & Marcinkiewicz, H. (2004). Online learning and time-on-task: Impact of proctored vs. un-proctored testing. Journal of Asynchronous Learning Networks, 8(4), 93–104.

Yates, R.W., & Beaudrie, B. (2009). The impact of online assessment on grades in community college distance education mathematics courses. American Journal of Distance Education, 23(2), 62–70. https://doi.org/10.1080/08923640902850601




DOI: http://dx.doi.org/10.5944/openpraxis.12.4.1113

Refbacks

  • There are currently no refbacks.