The usability of augmented reality in open and distance learning systems: A qualitative Delphi study

Hakan Altınpulluk, Mehmet Kesim, Gulsun Kurubacak


The main purpose of this research is to determine the usability of augmented reality in open and distance learning environments in accordance with universal design principles, and to make predictions for the future by gathering expert opinions on this subject using the Delphi technique. The Delphi technique was applied to 14 expert participants for 3 rounds as the primary data collection tool with open ended questions based on the theoretical framework. Structured interview questions were used as a secondary data collection tool and were applied during an academic exchange in China. In the Delphi technique used as the primary data collection tool, 92 themes were evaluated by experts and accepted as usability principles by end of this research. Therefore, 92 themes under 21 titles were presented for the use of augmented reality within the framework of universal design principles in open and distance learning. This research may be the first unique study on the usability of augmented reality not just as the convergence of this technology with open and distance learning environments but also incorporating the learning and communication dimensions of this convergence, thereby contributing to the literature of the field. It is imperative to determine the usability of augmented reality in open and distance learning, along with the “how” of this use in application. In this regard, the findings of this study are significant in shedding light to the enrichment, diversification and increased interaction of open and distance learning environments in accordance with universal design principles, bringing a new perspective to how a different technology convergence may be conducted, providing further accessibility.


Augmented reality; open and distance learning; universal design principles; Delphi

Full Text:



Adler, M., & Ziglio. E. (1996). Gazing into the oracle: The delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers.

Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1-11.

Alsina-Jurnet, I., & Guardia-Ortiz, L. (2015). Augmented reality in online educational contexts: the UOC case study. EDEN 2015 Proceedings. Barcelona, Spain. Retrieved from

Altinpulluk, H., & Eby, G. (2016). Theoretical framework regarding the usability of augmented reality in open and distance learning systems. In EdMedia+ Innovate Learning (pp. 487-492). Association for the Advancement of Computing in Education (AACE).

Altinpulluk, H. (2019). Determining the trends of using augmented reality in education between 2006-2016. Education and Information Technologies, 24(2), 1089-1114.

Andujar, J. M., Mejías, A., & Marquez, M. A. (2011). Augmented reality for the improvement of remote laboratories: an augmented remote laboratory. IEEE Transactions on Education, 54(3), 492-500.

Arici, F., Yildirim, P., Caliklar, Ş., & Yilmaz, R. M. (2019). Research trends in the use of augmented reality in science education: Content and bibliometric mapping analysis. Computers & Education, 142, 103647.

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent advances in augmented reality. Computer Graphics and Applications, 21(6), 34-47.

Bach, B., Sicat, R., Beyer, J., Cordeil, M., & Pfister, H. (2017). The hologram in my hand: How effective is interactive exploration of 3d visualizations in immersive tangible augmented reality?. IEEE transactions on visualization and computer graphics, 24(1), 457-467.

Barfield, W. (2015). Fundamentals of wearable computers and augmented reality. CRC Press.

Bau, O., & Poupyrev, I. (2012). REVEL: tactile feedback technology for augmented reality. ACM Transactions on Graphics (TOG), 31(4), 1-11.

Benko, H., Ofek, E., Zheng, F., & Wilson, A. D. (2015). Fovear: Combining an optically see-through near-eye display with projector-based spatial augmented reality. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (pp. 129-135).

Billinghurst, M., & Duenser, A. (2012). Augmented reality in the classroom. Computer, 45(7), 56-63.

Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding approach on students’ scientific project performance and perceptions. Computers & Education, 125, 226-239.

Chatzidimitris, T., Gavalas, D., & Michael, D. (2016). SoundPacman: Audio augmented reality in location-based games. In 2016 18th Mediterranean Electrotechnical Conference (MELECON) (pp. 1-6). IEEE.

Choi, Y. M. (2019). Applying tangible augmented reality for product usability assessment. Journal of Usability Studies, 14(4). Retrieved from

Cohen, L., Manion, L., & Morrison, K. (2013). Research methods in education (7th Ed.). Routledge.

Connell, B.R., Jones, M., Mace, R., Mueller, J., Mullick, A., Ostroff, E., Sanford, J., Steinfeld, E., Story, M., & Vanderheiden, G. (1997). The principles of universal design. Retrieved from

Craig, A. B. (2013). Understanding augmented reality: concepts and applications. Massachusetts: Morgan-Kaufmann Publishers. Retrieved from

Creswell, J. W. (2013). Qualitative inquiry & research design choosing among five approaches. Sage Publications.

Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15(2), 1-10. Retrieved from

Dede, C. (2005). Planning for neomillennial learning styles. Educause Quarterly, 28(1), 7-12.

Eby, G. (2013). Uzaktan eğitim ortamlarının tasarımı yazılım mühendisliği yaşam döngüsü yaklaşımı. Ankara: Kültür Ajans Publisher.

Essany, M. (2015). B-schools get an a for virtual reality experiments. Retrieved from

Fingas, R. (2016). Tim Cook says AI & augmented reality are core technologies in Apple's future. Retrieved from

Frank, J. A., & Kapila, V. (2017). Mixed-reality learning environments: Integrating mobile interfaces with laboratory test-beds. Computers & Education, 110, 88-104.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.

Gündoğan, M. B. (2012). Uzaktan eğitime ekolojik bir yaklaşım: sürdürülebilir uzaktan bilgisayar mühendisliği eğitimi ekosistemi önerisi (Unpublished doctoral dissertation). Eskişehir, Anadolu University, Turkey.

Härmä, A., Jakka, J., Tikander, M., Karjalainen, M., Lokki, T., Hiipakka, J. & Lorho, G. (2004). Augmented reality audio for mobile and wearable appliances. Journal of the Audio Engineering Society, 52(6), 618-639.

Harr, J. (2015). Oculus rift technology may improve online learning. Retrieved from

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015.

Heller, F., Jevanesan, J., Dietrich, P., & Borchers, J. (2016). Where are we? evaluating the current rendering fidelity of mobile audio augmented reality systems. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 278-282).

Higbee, J. L., & Goff, E. (2008). Pedagogy and student services for institutional transformation: implementing universal design in higher education. Center for Research on Developmental Education and Urban Literacy. Retrieved from

Höllerer, T., & Feiner, S. (2004). Mobile augmented reality. H. Karimi ve A. Hammad (Eds.), Telegeoinformatics: Location-Based Computing and Services (pp. 1-39) . London: Taylor and Francis Books. Retrieved from

Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.

Hung, H. L., Altschuld, J. W., & Lee, Y. F. (2008). Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Evaluation and Program Planning, 31(2), 191-198.

Ibáñez, M. B., Portillo, A. U., Cabada, R. Z., & Barrón, M. L. (2020). Impact of augmented reality technology on academic achievement and motivation of students from public and private Mexican schools. A case study in a middle-school geometry course. Computers & Education, 145, 103734.

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC horizon report: 2016 higher education edition. Austin, Texas: The New Media Consortium. Retrieved from

Jot, J. M., & Lee, K. S. (2016). Augmented reality headphone environment rendering. In Audio Engineering Society Conference: 2016 AES International Conference on Audio for Virtual and Augmented Reality. Audio Engineering Society. Retrieved from

Laviole, J., Thevin, L., Albouys-Perrois, J., & Brock, A. (2018). Nectar: Multi-user spatial augmented reality for everyone: three live demonstrations of educative applications. In Proceedings of the Virtual Reality International Conference-Laval Virtual (pp. 1-6).

Lee, J., Jung, S., Kim, J. W., & Biocca, F. (2019). Applying Spatial Augmented Reality to Anti-Smoking Message: Focusing on Spatial Presence, Negative Emotions, and Threat Appraisal. International Journal of Human–Computer Interaction, 35(9), 751-760.

Maiti, A., Smith, M., Maxwell, A. D., & Kist, A. A. (2018). Augmented reality and natural user interface applications for remote laboratories. In Cyber-Physical Laboratories in Engineering and Science Education (pp. 79-109). Springer, Cham.

Metz, R. (2015). Augmented-reality glasses could help legally blind navigate. MIT Technology Review. Retrieved from

Metz, R. (2016). Augmented reality study projects life-sized people into other rooms. MIT Technology Review.

Moore, M. G., & Kearsley, G. (2011). Distance education: a systems view of online learning. Cengage Learning.

Neuendorf, K. A. (2002). The content analysis guidebook. Sage Publications. Retrieved from

Núñez, M., Quirós, R., Núñez, I., Carda, J. B., Camahort, E., & Mauri, J. L. (2008). Collaborative augmented reality for inorganic chemistry education. WSEAS International Conference Proceedings Mathematics and Computers in Science and Engineering (Vol. 5, pp. 271-277). Retrieved from

Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15-29.

Onime, C., & Abiona, O. (2016). 3D mobile augmented reality interface for laboratory experiments. International Journal of Communications, Network and System Sciences, 9(4), 67-76.

Parviz, B. A. (2009). Augmented reality in a contact lens. Retrieved from

Pejsa, T., Kantor, J., Benko, H., Ofek, E., & Wilson, A. (2016). Room2Room: Enabling life-size telepresence in a projected augmented reality environment. Retrieved from

Pérez-López, D., & Contero, M. (2013). Delivering educational multimedia contents through an augmented reality application: A case study on its impact on knowledge acquisition and retention. TOJET: The Turkish Online Journal of Educational Technology, 12(4). Retrieved from

Perry, T. S. (2020). Augmented reality in a contact lens: it’s the real deal. Retrieved from

Rauschnabel, P. A., Brem, A., & Ivens, B. S. (2015). Who will buy smart glasses? Empirical results of two pre-market-entry studies on the role of personality in individual awareness and intended adoption of Google Glass wearables. Computers in Human Behavior, 49, 635-647.

Rauschnabel, P. A., & Ro, Y. K. (2016). Augmented reality smart glasses: An investigation of technology acceptance drivers. International Journal of Technology Marketing, 11(2), 123-148.

Redondo, B., Cózar-Gutiérrez, R., González-Calero, J. A., & Ruiz, R. S. (2020). Integration of Augmented Reality in the Teaching of English as a Foreign Language in Early Childhood Education. Early Childhood Education Journal, 48(2), 147-155.

Reid, N. (1988). The Delphi technique: Its contribution to the evaluation of professional practice. E, Roger. (Ed.), Professional Competence and Quality Assurance in the Caring Professions. London: Chapman & Hall.

Ro, Y. K., Brem, A., & Rauschnabel, P. A. (2018). Augmented reality smart glasses: Definition, concepts and impact on firm value creation. In Augmented reality and virtual reality (pp. 169-181). Springer, Cham.

Rossi, D. (2018). SAR for Kids: Spatial augmented reality as tool for art education. In International and Interdisciplinary Conference on Digital Environments for Education, Arts and Heritage (pp. 355-363). Springer, Cham.

Ruiz-Ariza, A., Casuso, R. A., Suarez-Manzano, S., & Martínez-López, E. J. (2018). Effect of augmented reality game Pokémon GO on cognitive performance and emotional intelligence in adolescent young. Computers & Education, 116, 49-63.

Sahin, D., & Yilmaz, R. M. (2020). The effect of Augmented Reality Technology on middle school students' achievements and attitudes towards science education. Computers & Education, 144, 103710.

Scavo, G., Wild, F., & Scott, P. (2015). The GhostHands UX: Telementoring with hands-on augmented reality instruction. Retrieved from

Seuring, S., & Müller, M. (2008). Core issues in sustainable supply chain management–a Delphi study. Business Strategy and the Environment, 17(8), 455-466.

Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education, 6, 1-21. Retrieved from

Smith, R. E., & Buchannan, T. (2012). Community collaboration, use of universal design in the classroom. Journal of Postsecondary Education and Disability, 25(3), 259-265. Retrieved from

Sumsion, T. (1998). The Delphi technique: An adaptive research tool. The British Journal of Occupational Therapy, 61(4), 153-156.

Sun, S. W., & Lan, Y. S. (2019). Augmented reality displaying scheme in a smart glass based on relative object positions and orientation sensors. World Wide Web, 22(3), 1221-1239.

Takahashi, D. (2020). Mojo Vision is putting an augmented reality screen on a contact lens. Retrieved from

Tashev, I. J. (2019). Capture, representation, and rendering of 3d audio for virtual and augmented reality. International Journal on Information Technologies & Security, 11(2), 49-62. Retrieved from

Thangaratinam, S., & Redman, C. W. (2005). The delphi technique. The Obstetrician & Gynaecologist, 7(2), 120-125.

The New Economy (2014). Using mobiles to smell: how technology is giving us our senses. Retrieved from

Topa-Çiftçi, G. (2011). Uzaktan eğitimde iptv’nin kullanılabilirliğine ilişkin bir delphi çalışması (Unpublished master’s dissertation). Eskişehir, Anadolu University, Turkey.

Tussyadiah, I. P., Jung, T. H., & Tom Dieck, M. C. (2018). Embodiment of wearable augmented reality technology in tourism experiences. Journal of Travel Research, 57(5), 597-611.

Vargas, H., Farias, G., Sanchez, J., Dormido, S., & Esquembre, F. (2013). Using augmented reality in remote laboratories. International Journal of Computers Communications & Control, 8(4), 622-634. Retrieved from

Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent augmented reality training for motherboard assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.

Williams, P. L., & Webb, C. (1994). The Delphi technique: A methodological discussion. Journal of Advanced Nursing, 19(1), 180-186.

Yıldırım, A., & Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri. Seçkin Publisher, Turkey.

Yin, R. K. (2003). Case study research: Design and methods. Sage Publications.

Yoon, B., Kim, H. I., Lee, G. A., Billinqhurst, M., & Woo, W. (2019). The effect of avatar appearance on social presence in an augmented reality remote collaboration. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 547-556). IEEE.



  • There are currently no refbacks.