The Usability of Augmented Reality in Open and Distance Learning Systems: A Qualitative Delphi Study

 

The Usability of Augmented Reality in Open and Distance Learning Systems: A Qualitative Delphi Study

Hakan Altınpulluk symbol, Mehmet Kesim symbol & Gulsun Kurubacak symbol

Anadolu University (Turkey)

hakanaltinpulluk@anadolu.edu.tr, mkesim@anadolu.edu.tr & gkurubac@anadolu.edu.tr

Abstract

The main purpose of this research is to determine the usability of augmented reality in open and distance learning environments in accordance with universal design principles, and to make predictions for the future by gathering expert opinions on this subject using the Delphi technique. The Delphi technique was applied to 14 expert participants for 3 rounds as the primary data collection tool with open ended questions based on the theoretical framework. Structured interview questions were used as a secondary data collection tool and were applied during an academic exchange in China. In the Delphi technique used as the primary data collection tool, 92 themes were evaluated by experts and accepted as usability principles by end of this research. Therefore, 92 themes under 21 titles were presented for the use of augmented reality within the framework of universal design principles in open and distance learning. This research may be the first unique study on the usability of augmented reality not just as the convergence of this technology with open and distance learning environments but also incorporating the learning and communication dimensions of this convergence, thereby contributing to the literature of the field. It is imperative to determine the usability of augmented reality in open and distance learning, along with the “how” of this use in application. In this regard, the findings of this study are significant in shedding light to the enrichment, diversification and increased interaction of open and distance learning environments in accordance with universal design principles, bringing a new perspective to how a different technology convergence may be conducted, providing further accessibility.

Keywords: Augmented reality, open and distance learning, universal design principles, Delphi

Reception date: 25 July 2019 • Acceptance date: 17 March 2020

DOI: https://dx.doi.org/10.5944/openpraxis.12.2.1017

Introduction

Open and distance learning is something that should not be reduced to the combination of advanced technologies in learning activities (Eby, 2013), yet it is also a field that is directly or indirectly influenced by developments and advances in technology. From correspondence education –the first generation of distance learning– to the advanced computer and internet technologies encompassing smart and flexible learning environments today –the latest generation of distance learning– distance learning applications differentiate in various ways. As such, it is apparent that information and communication technologies had great influence in the development and shaping of this discipline throughout its generations. While the generations of open and distance learning applications differ, the tools and approaches of previous generations do not disappear but rather continue to exist through their enrichment by innovative tools (Gündoğan, 2012). In this interconnected process, developments in information and communication technologies are the greatest determinants on what innovative tools will be.

The school system lasting from the 15th and 16th centuries that require learner attendance, note-taking and examinations, has evolved into a more experimental and interactive state in this century with the emergence of new technologies, allowing more effective learning by learners (Núñez et al., 2008). Traditional activities with low interaction classroom environments and materials are making way for unique, interesting, stimulating, realistic and accessible learning environments that provide the opportunity for more interactive collaborative work. In this regard, institutions providing open and distance learning products and services are aiming to provide more interesting and stimulating learning opportunities to learners (Pérez-López & Contero, 2013). It is of great importance that open and distance learning environments are equipped with new communication technologies and designed to provide learners with highly interactive environments and seamless information (Topa-Çiftçi, 2011). One of the most interesting of these new technologies is augmented reality.

Craig (2013) stated that considering augmented reality to be merely a technological innovation would be unfair in that augmented reality encompasses its own philosophy and artistic value within itself. Augmented reality presents new experiences by layering data on three dimensional spaces, increasing access to information and bringing new opportunities to learning environments (Johnson et al., 2016). Augmented reality is the enrichment of the real world with virtual additions rather than a fully artificial environment; technologies that enable the opportunity to present interactive experience (Höllerer & Feiner, 2004). The following section elaborates on the research problem, purpose, and significance regarding how augmented reality may be used in open and distance education based on a qualitative Delphi study.

Research Problem

Augmented reality has many applications in many fields, from medicine to engineering, military to architecture. One of the fields in which augmented reality is most utilized is learning environments. The proliferation of augmented reality in traditional learning environments may not necessarily be true for open and distance learning environments. In open and distance learning environments, which differ from traditional face to face learning activities in philosophy, design, theory and application (Eby, 2013), the usability of augmented reality must be questioned and exactly how this technology may be applied to and enrich open and distance learning environments must be determined. However, a review of the literature in the field revealed that the studies directly relating augmented reality with open and distance learning are limited and insufficient.

While the usability of augmented reality widely utilized in traditional face to face learning environments for learners at distance locations is debatable, the number of studies regarding the distance access of augmented reality is limited (Alsina-Jurnet & Guardia-Ortiz, 2015; Altinpulluk & Eby, 2016; Harr, 2015; Pejsa et al., 2016; Scavo, Wild, & Scott, 2015; Yoon et al., 2019). This study, which aims to determine the usability of augmented reality in open and distance learning, establishes a theoretical framework in which only the “technology” aspect of open and distance learning environments are explored, while the relationships with “learning” and “communication” are also taken into consideration. In this regard, in the preparation of a theoretical matrix, the horizontal axis is comprised of the “learning”, “technology”, and “communication” components of open and distance learning (Eby, 2013; Moore & Kearsley, 2011). The second approach of the theoretical framework established are universal design principles.

Universal design principles may be used in various disciplines as an approach to optimize the functional capacity of all individuals through increasing awareness. While the roots of these principles lie in architecture, they have been widely adapted for various studies in the field of education and generally serve to establish meaningful learning environments that may be presented for use to all individuals in an equally accessible manner by removing obstacles. The literature review conducted did not reveal any studies in which augmented reality, universal design principles, and open and distance learning were correlated. This study aims to present the usability of augmented reality in open and distance learning environments from the perspective of universal design principles.

Purpose of the Study

The fundamental goal of this study is to determine the usability of augmented reality in open and distance learning environments in accordance with universal design principles. As such, the research aims to collect the opinions of various experts in the field using Delphi, and provide predictions for the future of this field. Based on the goal of the study, the following research goals in accordance with universal design principles have been established:

  • To establish an approach that will aid in the design of interactive, efficient, enriched and innovative open and distance learning experiences using augmented reality,
  • To establish a roadmap that combines augmented reality with the established structure of open and distance learning environments and,
  • To draw the outline and analyze the processes involved in the establishment of the required infrastructure for the future use of augmented reality in open and distance learning environments.

Importance of the Study

This research may be the first unique study on the usability of augmented reality not just as the convergence of this technology with open and distance learning environments but also incorporating the learning and communication dimensions of this convergence, thereby contributing to the literature of the field. It is imperative to determine the usability of augmented reality in open and distance learning, along with the “how” of this use in application. In this regard, the findings of this study are significant in shedding light to the enrichment, diversification and increased interaction of open and distance learning environments in accordance with universal design principles, bringing a new perspective to how a different technology convergence may be conducted, providing further accessibility.

This is the first study in the field determining the sub components required for the effective, efficient and satisfactory use of augmented reality applications designed based on universal design principles by users in open and distance learning environments.

Literature Review: Augmented Reality Studies in Education and Open and Distance Learning

With the development of web based computer technologies, the design of learning environments has become more realistic, unique, entertaining, and intriguing. With great potential for establishing learning environments with these characteristics, augmented reality –with the interactions it provides– is tending to proliferate in traditional learning environments. It may be seen from content analysis and systematic literature review studies that the number of academic studies conducted on the use of augmented reality in education is increasing yearly (Akcayir & Akcayir, 2017; Altinpulluk, 2019; Arici et al., 2019). With many uses in different fields, augmented reality is also being used in various ways in traditional learning processes (Chang & Hwang, 2018; Ibañez et al., 2020; Redondo et al., 2020; Ruiz-Ariza et al., 2018; Sahin & Yilmaz, 2020).

While augmented reality may be a technology intensively used in traditional face-to-face classroom environments, it also has fields of use in open and distance learning environments. When studies using augmented reality in open and distance learning are analyzed, the majority of research was found to be covering augmented reality with concepts such as “e-learning” and “online learning” used augmented reality as a technology applied in traditional, face-to-face classrooms rather than remotely accessible augmented reality based systems. Just as it is used in face-to-face learning environments, augmented reality may be used in environments requiring distance collaboration or shared learning experiences (Billinghurst & Duenser, 2012). Alsina-Jurnet and Guardia Ortiz (2015) state that despite the proliferation of augmented reality applications in traditional learning environments, the usability is not yet at a mature level regarding e-learning. They continue, indicating that more research is needed to better understand when and under what circumstances augmented reality technologies may be integrated into e-learning environments, stating that the number of examples of augmented reality use in a fully online Open University are insufficient. Alsina-Jurnet and Guardia-Ortiz (2015) found that regarding the applicability of augmented reality in fully online-based Open Universities, location-based augmented reality applications are appropriate, while marker-based augmented reality applications may be less effective. Additionally, they suggest that the combined use of game-based learning and discovery-based learning applications in augmented reality for Open University learners may be effective.

There are few studies in the field in which augmented reality is used in the open and distance learning systems of Open Universities through remote access by learners and instructors. The most prominent study is an augmented reality based tele-monitoring application named “Ghost Hands”, conducted by The Open University Knowledge and Environment Institute in England. The remote instructor utilizes a 3D virtual hand model, along with hand motion and audio support to ensure learners successfully complete their own motions and learning processes (Scavo et al., 2015).

Regarding physical presence, Dede (2005) states that while being at that location is necessary, mirroring may provide a copy of remote physical locations in virtual three dimensions. This prediction by Dede came to fruition with the “Room2Room” project by Microsoft Research. This project used Kinect depth of field cameras and digital projectors to capture the image of a remote individual and provided a telepresence experience based on the immersive imaging ensuring the feeling of actually being there (Pejsa et al., 2016). The virtual person silhouettes of people are projected to scale into physical spaces. Each participant sees their virtual counterpart in the correct perspective, can communicate naturally and non-verbally (Metz, 2016). The Room2Room project has the potential to provide an effective learning environment for distance learners if combined with open and distance learning environments.

There are several studies regarding augmented reality based distance laboratory systems (Andujar et al., 2011; Frank & Kapila, 2017; Maiti et al., 2018; Onime & Abiona, 2016; Vargas et al., 2013). Vargas et al. (2013) see AR as an innovative path to enriching visualization in distance laboratories for engineering education. Andujar et al.’s (2011) augmented remote laboratory (ARL) system stands out when research on virtual and distance laboratories and augmented reality is parsed. The ARL system was developed as a new augmented distance laboratory using virtualization. In addition to providing richer sensory experiences to learners compared to traditional laboratories, positive results were obtained regarding learning outcomes. The reduction in sensory and physical interaction of distance laboratories was minimized in augmented reality using stereoscopic vision and virtual models connected to the laboratory. In this study, a comparison and evaluation of ARL with traditional distance laboratories was conducted, emphasizing that ARL was found to be superior as it (1) enriched the feeling of reality, (2) could reuse the same system for different experiments, and (3) provided the opportunity to more easily conduct experiments. Laboratory systems enriched with augmented reality also have the potential to provide experiment and practice in online learning processes, especially in the applied sciences.

Penn State University compared the use of mouse and keyboard with smart glasses and haptic feedback gloves from Oculus Rift through 54 engineering students. Their finding was that haptic feedback support in virtual reality based online environments were more effective. Despite the vast possibilities for online learning, the premise has been that the three dimensionality and haptic feedback interactivities of these environments were limited and insufficient. With its remote access, Oculus Rift allows learners from different countries to connect, enabling collaborative work and joint projects (Harr, 2015).

The simulation-based campus application for geographically distant learners co-developed by Stanford University and MIT enabled students’ use of virtual reality in group projects, discussions, and networking with other individuals (Essany, 2015).

Beyond all these examples, augmented television, social networks with integrated augmented reality applications, augmented teleconference systems, sensor-based applications such as Kinect, and 3D holographic projectors also carry great potential for remotely accessible augmented reality applications. Despite all these studies, no studies on how augmented reality can achieve universal access in open and distance learning environments have been found in the literature. In this context, the theoretical framework of this study was created by using universal design principles.

Universal Design Principles and Theoretical Framework

A group of architects, designers and engineers from the North Carolina State University established a broad scope of design principles that cover environmental organization, production and communication processes, determining the seven principles of universal design. These principles are fundamental approaches to creating environments accessible to everyone regardless of characteristics such as age, skill, or disability. The following are the seven principles of universal design (Connell et al., 1997):

  1. Equitable Use: The design is useful and marketable to people with diverse abilities.
  2. Flexibility in Use: The design accommodates a wide range of individual preferences and abilities.
  3. Simple and Intuitive Use: Use of the design is easy to understand, regardless of the user’s experience, knowledge, language skills, or current concentration level.
  4. Perceptible Information: The design communicates necessary information effectively to the user, regardless of ambient conditions or the user’s sensory abilities.
  5. Tolerance for Error: The design minimizes hazards and the adverse consequences of accidental or unintended actions.
  6. Low Physical Effort: The design can be used efficiently and comfortably and with a minimum of fatigue.
  7. Size and Space for Approach and Use: Appropriate size and space is provided for approach, reach, manipulation, and use regardless of user’s body size, posture, or mobility.

While universal design may be rooted in architecture, these principles have been reinterpreted in learning environments to other concepts such as Human Centered Design, Universal Design for Learning, Universal Design for Instruction and the broader approach of Universal Design for Education (Smith & Buchanan, 2012). These models are closely related and carry complementary characteristics (Higbee & Goff, 2008).

Moore and Kearsley (2011) indicate that open and distance learning comprises of the elements of learning, instruction, communication, and design when approached from a system perspective. In the same study, they indicate “technology” to be the most important determinant of the communication element within the conceptual model and framework of open and distance learning systems. Eby (2013), however, adapted these elements to their own study and determined the elements of open and distance learning to be administration, communication, learning, technology, and evaluation.

For this study, the open and distance learning elements determined by Moore and Kearsley (2011) and Eby (2013) were adapted in accordance with the nature of the study, resulting in “learning”, “communication” and “technology” being determined as the primary elements of open and distance learning. In the usability of advanced technology applications such as augmented reality, it is imperative that not only the technology aspect, but also the communication and learning aspects of open and distance learning environments are taken under consideration. In this regard, the horizontal axis of the theoretical matrix of this study comprises of the (1) Learning, (2) Communication, and (3) Technology elements of open and distance learning environments.

In accordance with the purpose of this study, a cross stitch between the seven principles of universal design and the learning, communication, and technology elements of open and distance learning was established in a theoretical framework (Altinpulluk & Eby, 2016), providing the basis for the research questions and interview questions. The combination of these two approaches establishes the theoretical foundation of this study, determining the research purpose and goals, and lastly provides a basis for the development of the data gathering tool in this study.

The authors of the study prepared the theoretical matrix by taking into consideration both universal design principles, and the elements of open and distance learning. The 21 fields in the matrix were finalized based on universal design principles, all the characteristics of open and distance learning, and the suggestions and corrections of two experts in qualitative research.

Method

Research Model

This study was conducted as a qualitative case study, as it attempts to determine the “why”, “how”, and “in which way” regarding the use of augmented reality in open and distance learning systems in accordance with the theoretical basis of the 7 universal design principles and 3 elements of distance learning.

As such, the justification for the qualitative case study design of the research is as follows:

  • The study deeply explores a problem or subject (Creswell, 2013)
  • The details of an event are defined (Gall, Borg, & Gall, 1996)
  • “How” and “why” are primary concerns, and the study focusses on a current phenomenon (Yin, 2003).

The data gathering for this study was conducted with both local and international experts, with open ended questions emerging from the theoretical matrix. As the primary data gathering tool, the Delphi technique was applied for 3 rounds. Additionally, data from structured interview forms from international participants and observations supported this research.

This study was conducted with a holistic single case (Type 1) design. In this research design;

  • The inconclusiveness of the usability of augmented reality in open and distance learning environments,
  • The unique characteristics of determining usability in accordance with universal design principles,
  • The gap in the literature regarding similar studies resulting in unique value for this research justify the use of a holistic single case design.

The primary data gathering tool for this study was 3 rounds of questionnaires conducted using the Delphi technique.

  • In the first round, a qualitative questionnaire with 21 open ended questions based on the theoretical matrix,
  • In the second and third rounds a questionnaire with a six point grade of importance was applied to the participants.

As a secondary data gathering tool, structured interview questions developed from the first round of Delphi was applied to international experts.

Research Scope and Participants

There are two separate communities of participants within this study. Purposive sampling was used in the first stage of the study to determine participants. The participants in for the primary data gathering Delphi technique have certain characteristics. The following were considered to be important measures in participants:

  1. The knowledge and experience regarding the subject,
  2. The willingness and capacity to participate,
  3. Having sufficient time to participate,
  4. Portraying effective communication skills (Adler & Ziglio, 1996; Skulmoski, Hartman, & Krahn, 2007).

In this regard, the selection of the most competent participants in the fields of “augmented reality”, “universal design”, and “open and distance learning” was a highly important factor in the study. As such, experts living in Turkey with a doctorate degree and at least 5 years of academic experience in primarily augmented reality, but also in open and distance learning and universal design were selected as participants for the study.

The Delphi technique as a primary data gathering tool was applied in three rounds for this study. As a secondary data gathering tool, an interview form with four pre-structured questions was applied to experts during an academic exchange program –Asian Association of Open Universities (AAOU) Inter-university Staff Exchange Fellowship– with the Shanghai Open University in China.

There are varying opinions regarding the number of participants for Delphi studies. Williams and Webb (1994) state that there is no set rule regarding the sample size of participants in the Delphi panels. For example, Reid (1988) states that the panel size may vary from 10 to 1685 participants. Okoli and Pawlowski (2004), however, state that the number of expert participants may vary between 10 and 18. In this study, an initial number of 16 experts were selected for participation in the Delphi rounds.

Prior to conducting the study, the data gathering tool was analyzed and corrected with two experts in qualitative research. These two research experts were also individuals who met the criteria for participation in the list of experts of the study. At this point, these two participants evaluated the 21 open ended questions obtained from the theoretical matrix until they reached a consensus regarding language, ambiguity or necessary corrections for each item. Additionally, a pilot study was conducted with three experts who work at the Open and Distance Education Faculty of Anadolu University who were not participants in the study. The feedback from these experts regarding the 21 open ended questions of the first round of the Delphi study ensured certain corrections which resulted in the final state of the data gathering tool.

The Delphi round 1 open ended questionnaire was distributed to 16 expert participants along with the call to participate and a link to the form via e-mail. 14 of the 16 participants selected completed the questionnaire at the end of round 1, while 2 participants did not respond. Table 1 portrays the participation of all three rounds of the study.

 

Table 1: Delphi participation rates
Delphi rounds No. of selected participants Responding participants Response ratio
First round 16 14 %87,5
Second round 14 14 %100
Third round 14 14 %100

As can be seen from Table 1, following the participation of 14 of the 16 selected experts in the first round of the study, e-mails were not sent to the unresponsive participants for the following rounds. While the response ratio for the first round was 87.5%, the response rate for all subsequent rounds was 100%. Sumsion (1998) suggests that the successful application of a Delphi technique requires a response rate above 70%. Additionally, the researcher must keep track of who responds and who does not (Hasson, Keeney, & McKenna, 2000). In this regard, regular e-mail reminders were sent for this study to keep the response process under control throughout the Delphi rounds.

Yıldırım and Şimşek (2013, p. 52) state that depending on the opportunity to gather data, new situations that emerge during the qualitative research process may re-shape various aspects of the study. In essence, the direction of the research may change when necessary, and new data gathering tools may be developed, and the new data gathering tools may be re-shaped based on newly emerging circumstances. In this study, one of the qualitative researchers was assigned to an unforeseen and unplanned assignment through an academic exchange program to the Shanghai Open University in a China. During the second round of the Delphi data analysis, the researcher conducted the data gathering during an 11-day period. Participants comprised of experts working on augmented reality at the Digital Laboratory of the Open University, and academics from around the world (USA, United Kingdom, China, Bangladesh, Spain, Netherlands) participating in the academic exchange program. Participation in this study was based on voluntary participation.

Thus, the limitation of experts and academics for the Delphi panelists comprising only of participants from Turkey was overcome, and with the inclusion of international expert opinions, new perspectives and opinions enriched the study.

Data Gathering Tools

The Delphi technique gets its name from the “Oracle of Delphi”, a figure from Greek mythology with supernatural gifts who prophecized about the future (Thangaratinam & Redman, 2005). Since this study also requires certain predictions and projections regarding determining the future of use tendencies of “augmented reality” –a relatively new technology– in open and distance learning environments, the Delphi technique was applied. Delphi is a frequently utilized research technique for studies in technological inclinations and predictions, especially in Horizon Reports.

In this study which presents technological predictions, the Delphi technique was utilized because:

  • There is ambiguity or lack of information on the subject in the literature (Hung, Altschuld & Lee, 2008; Skulmoski et al., 2007)
  • It requires studying and revealing currently absent situations (Skulmoski et al., 2007)
  • There is a lack of certainty on the subject and leaves it open to interpretation (Okoli & Pawlowski, 2004)
  • It approaches issues with “what could be” or “what should be” rather than simply “what” (Hsu & Sandford, 2007).

Traditionally, the first round of the Delphi technique is conducted with open ended questions. Open ended questions are important in presenting deep and specialized content on the subject being studied (Custer, Scarcella, & Stewart, 1999). The first round of questions prepared for this study were prepared based on the theoretical matrix of the study in accordance with the purpose and research questions of this study.

A theoretical matrix assists the researcher in defining the dimensions of their research problem, determining the relationships between these dimensions, determining the scope of the data gathering tools, and selecting the themes to utilize during the analysis of data (Yıldırım & Şimşek, 2013). Answers to the 21 open ended questions about the usability of augmented reality in open and distance learning environments, along with predictions are sought within this study. The 21 items in the theoretical matrix were finalized following universal design principles, the elements of open and distance learning, and the suggestions and corrections of two experts in qualitative research.

The 21 questions of the 1st round was presented to participants in a questionnaire form prepared in Google Forms. Following the 1st round, a Likert scale is frequently used to grade panelist opinions based on importance (Thangaratinam & Redman, 2005). In this study, the results of the data analysis from the responses to the first round were utilized in a six point Likert scale that graded between 0 (Unimportant) to 5 (Very Important) in the 2nd and 3rd rounds.

In summary,

The primary data gathering tools were questionnaires conducted in 3 rounds using the Delphi technique and prepared in an online environment.

  • In the 1st round, a qualitative questionnaire comprising of 21 open ended questions prepared from the theoretical matrix was applied.
  • In the 2nd round, the data gathered was analyzed and the findings were used to apply a six point Likert scale developed to determine the degree of importance of this data.
  • In the 3rd round, data from the 2nd round was analyzed and based on the findings, another six point Likert scale was applied to grade the findings based on their importance.

As a secondary data gathering tool, a structured interview form was applied on international expert participants other than the participants of the Delphi rounds.

Due on the preparation of the first round of Delphi questions being prepared in accordance with universal design principles, the researcher developed a different data gathering tool based on the other axis of the theoretical matrix –elements of open and distance learning– based on “Learning”, “Communication”, “Technology”, and “Universal Access”, to gather differing viewpoints. How may augmented reality be used in open and distance learning systems regarding;

  1. Learning?
  2. Communication?
  3. Technology?
  4. Universal Access?

Additionally, to take advantage of the opinions of participants beyond these elements, a section in the form labeled “other opinions (if applicable)” was presented.

Data Analysis

There are various differing viewpoints regarding data analysis in the Delphi technique. Data analysis may be conducted with both qualitative and quantitative methods. In classical Delphi studies, qualitative analyses are used especially in the analysis of the open-ended questions at the beginning of the study (Hsu & Sandford, 2007). The first round of questions in Delphi studies are usually open ended, and analyzed qualitatively to be separated into themes. Based on these analyses, the second round questionnaire questions may be prepared in a more specialized manner. During this stage, grading importance is usually done using quantitative analysis (Thangaratinam & Redman, 2005).

In the first round of this Delphi study, the open ended questions derived from the theoretical matrix of universal design principles and the elements of open and distance learning aimed to gather deeper information regarding the usability of augmented reality in open and distance learning environments. Responses to the open ended questions in the first round are usually evaluated using content analysis (Neuendorf, 2002; Seuring & Müller, 2008). As such, the data from the online form and the dictations of voice recordings were analyzed using content analysis, revealing 103 themes for 21 main subjects in the first round, with two qualitative researchers conducting the coding of the data. During the analysis process of the first round, the reliability between the coding of the two qualitative research experts was determined to be 93%.

During the second round of the Delphi technique, the data gathered in the first round was graded based on their degree of importance. During this grading which was conducted based on the importance degree determined by the panelists, Likert scales are frequently utilized (Thangaratinam & Redman, 2005). To determine the degree of importance regarding the usability themes, a six point Likert scale running between 0 (Unimportant) to 5 (Very Important) was used. Themes with degrees of importance higher than the arithmetic average of 4 were accepted (x ≥4) while themes between 3 and 4 were determined for evaluation in the 3rd round, and themes with an arithmetic average below 3 (x<3) were eliminated.

The data analysis conducted following the 2nd round with these measures did not eliminate any themes. The number of themes with arithmetic averages between 3 and 4 to be re-evaluated for degree of importance in the 3rd round was 27, while the number of themes with arithmetic averages between 4 and 5 reaching a consensus among experts was 76. The results of the second round are portrayed in Table 2.

 

Table 2: 2nd Delphi Round Findings
2nd round results
Eliminated themes 0
Themes to be re-evaluated in round 3 27
Accepted themes 76

Based on the responses provided in the 2nd round, 27 of the 103 usability themes were uploaded to the online form along with their average values in the 3rd round to once again determine their degree of importance. The same six point Likert scale ranging from degrees between 0 (Unimportant) to 5 (Very Important) was used.

For the 3rd round, themes with an arithmetic average of 4 and higher (x ≥4) were accepted while themes with arithmetic averages below 4 (x ≥4) were eliminated.

In the 3rd round, other than one theme, the arithmetic average of all 26 other themes was found to be higher than in the 2nd round. With these measures in place, the data analysis conducted following the 3rd round eliminated 11 themes and accepted 16 themes. Thus, 11 of the total 103 usability themes were eliminated throughout the 3rd round while a consensus was reached regarding 92 themes. It was also agreed that regarding the themes eliminated following round 3, a consensus could not be reached in round 4, so the Delphi rounds were concluded. The final status obtained following round 3 is portrayed in Table 3.

 

Table 3: Total Eliminated and Accepted Themes in Round 3
3rd round results Total
Eliminated themes 11 11
Accepted themes 16 92

The data analysis for the interviews comprised of four structured questions conducted as a secondary data gathering tool was executed similarly to the data analysis of the first round of the Delphi technique, with qualitative content analysis.

Findings

This study, which aims to determine the usability of augmented reality in open and distance learning environments in accordance with universal design principles, took into account seven universal design principles and revealed 92 usability themes under 21 headings. The research questions, Delphi round 1 interview questions, findings and results of the research were all prepared based on the fundamental goal of the study.

Of the 103 themes emerging at the end of the first round, 27 were transferred from the 2nd round to the 3rd round for re-evaluation. Within this round, 11 themes were eliminated, 16 were accepted and a total of 92 themes were evaluated as principles of usability as a result of this study. During the data analysis, data from both the opinions obtained from the Delphi panels from experts in Turkey, and data obtained from structured interview forms obtained from 6 different countries were utilized. The 92 themes that emerged are portrayed in Table 4. This study, which aims to determine the usability of augmented reality in open and distance learning environments in accordance with universal design principles, took into account seven universal design principles and revealed 92 usability themes under 21 headings. The research questions, Delphi round 1 interview questions, findings and results of the research were all prepared based on the fundamental goal of the study.

 

Table 4: General Findings Regarding the Usability of Augmented Reality in Open and Distance Learning Based on Universal Design Principles
General findings of the usability of augmented reality in open and distance learning based on universal design principles
Universal design principles Element of open and distance learning Headings Themes
Equitable use Learning Equal opportunities for everyone in the learning process regardless of individual differences
  • Financial ease through open access, open source code software solution
  • Shaping design while collaborating with distance learners as the end users during all processes
  • Developing suitable augmented reality applications by retaining learner characteristics in a database and recognizing learner characteristics through analysis using advanced statistical techniques
  • Providing equal conditions in the same environment for education requiring augmented reality based application
  • Ensuring equitable learning opportunities to disadvantaged individuals and groups
  • Agreements with institutions for expensive or difficult to access augmented reality hardware and software
Communication “Accessible” environments for all learners independent of time and space
  • Augmented reality applications unique to mobile devices ensuring learning environments independent of space and time
  • GPS and location based mobile augmented reality applications being adaptable to the location of the distance learner
  • Uploading all learning content to the application through augmented reality based virtual distance laboratories
Technology Interaction in the design that draws the attention of learners
  • Augmented reality integration with game based learning environments within the scope of edutainment
  • Aggregation of augmented reality with a gamification approach
  • Adding augmented reality elements to printed materials and books
  • Resolving the stagnation of text based learning with augmented reality
  • Combining barcodes, QR codes and special symbols with course materials
  • Adding augmented reality elements to electronic books
  • Combining the use of digital storytelling with augmented reality
  • Presenting classes in interesting forms such as augmented reality based flash cards, puzzles and brain teasers.
  • Distance learners observing augmented reality based class materials with web cams.
Flexibility in use Learning Preparing courses in accordance with the individual characteristics, skills, and speed of the learn
  • Preparing classes directed at individual learning speeds through modular instructional design
  • Minimizing individual differences through applications adaptable to different living spaces
Communication Providing the opportunity for collaborative work
  • Use of augmented reality based video conference systems
  • Combining open and distance learning systems with current worldwide augmented reality applications in accordance with technological equipment
  • Resolving interpersonal communication problems in teamwork with augmented reality
  • Use of wearable technology able to communicate at distant locations in project groups
  • Augmented reality applications integrated with web technologies enhancing collaborative learning
  • Enriching sharing environments by supporting social networks with augmented reality applications
  • Establishing glocal (global+local) collaborations and partnerships between global technology firms, project offices, NPOs and laboratories conducting studies on augmented reality
  • Conducting content and application production and evaluation in project groups using collaborative approaches and augmented reality
  • Use between distance learner and instructors for courses with potential communication problems
Technology Preparing flexible learning environments that can immediately adapt to different knowledge and skills, providing users with a broad range of choices
  • Supporting augmented reality applications with artificial intelligence
  • Designing environments that can adapt based on personal information by combining augmented reality with learning management systems (LMS)
  • Developing smart augmented reality applications delicately adaptable based on user interaction
Simple and intuitive use Learning Enriching the creative thinking skills and imagination of the learners
  • Enriching imagination by augmented reality making abstract concepts tangible
  • Directed towards learners with low three dimensional thinking and spatial skills
  • Using coloring and three dimensional imaging techniques
Communication Ease of comprehension and use
  • Augmented reality based explanatory notes assuming a guiding role
  • Easier use of open and distance learning environments through augmented reality based instructions and notes
  • Realizing maximum mental activity with the simplest interface design
  • Developing simple and functional augmented reality add-ons
  • Fluent use of applications through augmented reality literacy
  • Orientation through applications such as awareness raising, advertising, and technology introduction regarding augmented reality to open and distance learners
  • Using augmented reality based multimedia functions and notifications integrated with daily life
Technology Motivational for the sustainability of learning
  • Establishing augmented reality environments with current, up to date content
  • Purposeful rather than trendy applications ensuring sustainability
  • Augmented reality presenting interesting, interactive and impressive environments for the sustainability of open and distance learning environments
  • Augmented reality providing motivation and incentive especially for low participation or unsuccessful learners due to it being a new technology
Perceptible information Learning Activating sensory stimuli containing qualities for all senses
  • Contrary to the notion that augmented reality only involves the sense of sight, designing interfaces directed towards the other senses
  • Activating all senses with personalizable augmented reality environments
  • Working with experts in fields such as medicine, psychology and physiology to better understand sensory stimuli
Communication The process of information transfer independent of medium or environment
  • Learners at distant locations connecting to each other
  • The ability for augmented reality applications to work both online and offline
  • Creating designs that are minimally influenced by external factors
  • Equipping university campuses appropriately for augmented reality
  • The necessity for augmented reality applications to work independent of device
Technology Utilizing techniques or interfaces that provide compatibility, encompassing users with sensory limitations/disabilities
  • The use of special augmented reality components for preparing embossments or signage/sign language or similar as precautions against sensory limitations
  • Developing augmented realities adaptable to the type and severity of sensory obstacle
  • Making augmented reality based learning environments compatible based on the feedback from learners with disabilities
  • Designing systems that can automatically sense based on the disability, personal preference, or sensory requirement
Tolerance for error Learning Isolating the learning environment from threats
  • Presenting natural events or experiences which cannot be experienced in the physical world
  • The application of possibly dangerous experiments
  • Accessing inaccessible geographic spaces and locations
  • Determining definitions of danger and degrees of importance, and developing augmented reality applications appropriately
Communication Clearly stating behaviors and design elements that may cause errors and accidents
  • Safer use of open and distance learning environments by designing orientation and on the job training applications with augmented reality
  • Augmented reality based hardware, receptors and sensors perceiving threats in advance
  • Minimizing institutional errors through circular/cyclical design
  • Providing feedback for errors through audio analysis software
  • Developing early warning systems against accidents and errors possible in learning environments
  • Preventing danger before it can happen by conducting designs with virtual markers and warning
Technology Correction and feedback for simple user errors
  • Providing multiple options through augmented reality depending on the source and characteristic of the error
  • Recording erroneous processes in a database during augmented reality activities and providing immediate feedback
Low physical effort Learning Increasing attention levels by providing ease of use
  • Determining subjects proving difficult in learning and presenting them using augmented reality
  • Augmented reality being interesting as a new and different experience
  • Determining subjects with low page viewing durations and presenting them using augmented reality
Communication Effectiveness and efficiency
  • Producing ecological designs that use low energy and resources
  • The complete convergence/integration of augmented reality applications with learning environments
  • Designing a strong backend system with a simple interface
  • The employment and training of professional technical staff and designers in the field of augmented reality
Technology Easily making hard to experience abstract and difficult concepts tangible
  • Supporting simulation applications with augmented reality
  • Using augmented reality based animations and three dimensional videos
  • Using smart glasses in distance access
  • Accessing distant locations virtually using holograms
  • Creating unique learning environments through wearable technologies
  • Presenting distance learning course materials with contact lenses
Size and space for approach and use Learning Ensuring appropriate conditions for the creation of unique learning environments regardless of the individual characteristics (body, communication needs, physical skills, mobility) of the learner
  • Presenting personal authorization to digital content using augmented reality supported digital recognition and identification technologies
  • Structuring augmented reality based designs based on an individual deprived of many senses
Communication Increasing the level of commitment
  • Preparing web based three dimensional virtual environments that use various imaging technologies
  • Integrating augmented reality with web conference based virtual course software
Technology Providing the opportunity for the individual to create their own unique learning environment
  • Free and voluntary access
  • Using semantic technologies from web 3.0 and beyond
  • Personalizable augmented reality applications provide the opportunity to present unique environments
  • The individual may create their own unique learning environment based on augmented reality with a constructivist approach
  • The learner may create their own augmented reality based learning environment by scripting and producing the content themselves

Discussion and Conclusion

Within the scope of this study, which aims to determining the usage possibilities of augmented reality in open and distance learning environments based on universal design principles, a total of 92 usability themes under 21 headings were presented under seven universal design principles. This study was not limited to a certain group of participants for Delphi rounds, but data was also gathered from participants in other countries and observations were made regarding studies on augmented reality. Throughout the Delphi rounds, data triangulation was conducted using online forms with open ended questionnaires, online forms with quantitative based Likert scales, and structured qualitative interview forms presented to foreign experts as a secondary data gathering tool. Space triangulation is used to resolve the limitations of conducting a study within a single culture and society (Cohen, Manion, & Morrison, 2013). In this regard, one of the researchers observed the perceptions regarding augmented reality within a different culture in China, conducted informal interviews with programmers developing augmented reality applications, and also gathered data with structured interview questions presented to a team and participating academics developing and studying augmented reality at a “Digital Laboratory”.

The most fundamental characteristic of the flexibility in use universal design principle refers to offering broad options to users by adapting to different knowledge and skills. In this regard, one of the findings of this heading was the significance of artificial intelligence (AI), smart, and personalizable systems for the development of adaptive systems. The combination of AI and augmented reality applications is gaining prominence not only in open and distance learning, but in all educational environments. The literature in the field also provides examples of smart applications offering augmented reality based education and exercise services (Westerfield et al., 2015). This research finding is also supported by Tim Cook, CEO of Apple. Apple predicts that the most significant core technologies in the near future will arrive through the combination of AI and augmented reality (Fingas, 2016). In short, new augmented reality systems in which AI is used may provide an impressive use opportunity in open and distance learning systems.

Regarding enriching imagination through augmented reality, the foremost finding was the power augmented reality has to actualize abstract concepts. Learners in open and distance learning systems consist of masses from various geographies, various age groups, and various personality traits. The level of three-dimensional thinking skills and spatial intelligence also vary by individual. There is a strong relationship between spatial skills and augmented reality. The difference of spatial augmented reality compared to other definitions lies in how it prevents the hardware from converging with the user through the use of mirrored rays, transparent screens, holograms or video projectors rather than hardware implemented on the head or body, glasses, or mobile devices (Lee et al., 2019). While this approach has it’s strengths and limitations, the development of three dimensional thinking and spatial intelligence by spatial augmented reality is supported by many studies (Benko et al., 2015; Laviole et al., 2018; Rossi, 2018), indicating that augmented reality may also be used by distance learners to enrich creative thinking and imagination in this regard.

There is a misconception that augmented reality is only related to our visual senses. Augmented reality, starting with our sense of hearing, is also capable of encompassing other methods of sensory interaction such as touch (tangible augmented reality), taste, and smell (Azuma et al., 2001). While augmented reality studies covering the enrichment of auditory (Chatzidimitris et al., 2016; Härmä vd., 2004; Heller et al., 2016; Jot & Lee, 2016; Tashev, 2019) and tactile (Bach et al., 2017; Bau & Poupyrev, 2012; Choi, 2019) reality are prevalent, studies on taste and smell also exist (The New Economy, 2014). As such, another finding of this study is that contrary to the notion that augmented reality is only related to sight, interfaces may be designed for other senses as well. While some simulators encompassing all five sensory apparatus have been developed, especially in the context of virtual reality, their adaptation to augmented reality and development in both virtual and actual reality rather than merely virtual reality may be possible.

Another conclusion of this study was the development of augmented reality applications of individuals with disabilities in open and distance learning systems, wherein the augmented reality would adapt depending on the type and severity of the disability. Rather than offer vision based augmented reality applications to a learner with visual impairment, providing aural, oral and olfactory stimulus would be more effective. As such, adaptive designs based on disability type must be conducted. Many studies on augmented reality regarding people with disabilities support this conclusion. VA-ST, an English technology company, designed a set of prototype augmented reality glasses named Smart Specs aiming to improve the vision of people with partial visual impairment, glaucoma, night blindness, and other impairments such as macular degeneration (Metz, 2015). The further development of such applications towards disabilities may be foreseeable. An important aspect of this would be the feedback obtained from learners with disabilities. As the end user, it is important that people with disabilities provide opinions and determine the strengths and weaknesses of augmented reality applications developed for them. This feedback would ensure more robust applications and the addressing of shortcomings. In summary, it is important to work with disabled individuals throughout the whole development process. Another conclusion that was drawn was the design of systems capable of automatically sensing based on the disability, personal preference, or sensory requirements. One example would be the sensing of a visually impaired individual, and their subsequent direction toward augmented reality applications that present stimuli other than visual stimulus. Currently, the use of sensors may be an approach for the determination of the sensory impairment. The data gathered from sensors regarding the sensory impairment type may allow the system to automatically adapt appropriately.

As a branch of wearable technologies, smart glasses are frequently used in the field of augmented reality (Rauschnabel et al., 2015; Rauschnabel & Ro, 2016; Ro et al., 2018; Sun & Lan, 2019) and their influence on learning processes have been previously studied. Many large technology companies have produced and released their own augmented reality and virtual reality glasses, enabling a competitive market. With the interactions they provide, these glasses provide virtual and digital supplements to the physical world of the user and may thereby create unique learning environments. The remote access feature of smart glasses, enabling learners to connect with other learners, instructors, and learning resources in their learning environments suggests that this technology may be used more effectively in the future.

There is a direct correlation between wearable computers and augmented reality technologies (Barfield, 2015; Tussyadiah et al., 2018). In this section of the study, the foundation established was the creation of unique environments regardless of the physical characteristics of the learner, therefore the focus of participants tended to be wearable technologies as the most commonly known subset of augmented reality applications. Wearable technologies allow the creation of unique open and distance learning environments. The wearable computers used by learners may be utilized for various functions such as connecting with each other, remote access, or imaging augmented reality elements. Just as with smart glasses, contact lenses also fall into the same wearable technology category and may be enriched with augmented reality. Studies in this field are continuing (Parviz, 2009; Perry, 2020; Takahashi, 2020) and it is stated that more advanced augmented reality based contact lenses will be designed in the future. While optical health and ethical considerations are continuing debates on this subject, the studies conducted indicate the leniency towards the development of augmented reality based contact lenses.

Recommendations

Despite being extensively used in traditional face to face learning environments, various differing approaches emerged regarding augmented reality and the indeterminate ways it may be used in distant locations. A total of 92 themes presented under 21 headings determined based on 3 dimensions of distance learning and 7 universal design principles are portrayed as a table in the findings section. Based on the findings and conclusions of this study, the following suggestions are presented based on recommendations for “institutions” and “researchers”:

Recommendations for Institutions

  • The usability principles that emerged from this study may be applicable for institutions such as open education faculties, distance education centers, and open universities that provide open and distance learning services.
  • Face to face educational institutions at primary and secondary levels along with universities may conduct further studies regarding the most effective and efficient use of augmented reality in addition to open and distance learning institutions.

Recommendations for Researchers

  • The usability of augmented reality is presented under a broad scope within this study. In this regard, the usability of more specialized applications and environments such as mobile augmented reality, holograms, or wearable technologies may be studied under are more focused scope.
  • Different theoretical approaches to the universal design principles used in this study may be utilized, studying different aspects of the usability of augmented reality in open and distance learning.
  • Studies with a broader scope may be conducted by including elements of open and distance learning beyond “learning”, communication”, and “technology”.
  • There is a lack of research regarding the use of virtual reality in open and distance learning environments, despite being related to but significantly different than augmented reality. Various studies that focus solely on virtual reality may be conducted.
  • Other studies may utilize the Delphi technique with quantitative analyses, rather than the qualitative based Delphi technique utilized in this study.
  • The effectiveness of remotely accessible augmented reality applications may be determined using experimental studies.
  • In depth research over longer durations may be conducted using a design-based research method.
  • Research may be conducted regarding the influence of augmented reality applications in open and distance learning on the academic achievement, interest and motivation levels of learners.
  • The 92 usability themes that emerged from this study may be considered as separate subjects of inquiry in various regards by different researchers.
  • Differing from studies that prioritize the sense of sight in augmented reality, emphasis in future research may be placed on the augmentation of reality for the other senses.
  • Augmented reality applications may be developed specifically for open and distance learners with disabilities and their effectiveness may be determined.
  • The current status of augmented reality applications may be followed in accordance with developments and the pace of advances in technology, allowing for innovative new research.

Acknowledgements

This study was supported by project number 115K627 which was accepted by the TÜBİTAK 3001 Initial Research and Development Support Program.

A portion of this research was supported by project number 1506E479 which was accepted by the Anadolu University Scientific Research Projects Commission.

Authors’ Note

The theoretical framework of this study was presented and published at the EdMedia conference in Vancouver in 2016, and the execution of the study was conducted in accordance with the feedback provided (Altinpulluk & Eby, 2016).

This study is derived from the doctorate thesis titled “Usability of Augmented Reality within the Framework of Universal Design Principles in Open and Distance Learning” conducted at the Anadolu University Institute of Social Sciences, Department of Distance Education.

References

Adler, M., & Ziglio. E. (1996). Gazing into the oracle: The delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers.
Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1–11. https://doi.org/10.1016/j.edurev.2016.11.002
Alsina-Jurnet, I., & Guardia-Ortiz, L. (2015). Augmented reality in online educational contexts: the UOC case study. EDEN 2015 Proceedings. Barcelona, Spain. Retrieved from https://www.researchgate.net/publication/279538875_augmented_reality_in_online_educational_contexts_the_uoc_case_study
Altinpulluk, H., & Eby, G. (2016). Theoretical framework regarding the usability of augmented reality in open and distance learning systems. In EdMedia + Innovate Learning (pp. 487–492). Association for the Advancement of Computing in Education (AACE).
Altinpulluk, H. (2019). Determining the trends of using augmented reality in education between 2006–2016. Education and Information Technologies, 24(2), 1089-1114. https://doi.org/10.1007/s10639-018-9806-3
Andujar, J. M., Mejías, A., & Marquez, M. A. (2011). Augmented reality for the improvement of remote laboratories: an augmented remote laboratory. IEEE Transactions on Education,54(3), 492–500. https://doi.org/10.1109/TE.2010.2085047
Arici, F., Yildirim, P., Caliklar, Ş., & Yilmaz, R. M. (2019). Research trends in the use of augmented reality in science education: Content and bibliometric mapping analysis. Computers & Education, 142, 103647. https://doi.org/10.1016/j.compedu.2019.103647
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent advances in augmented reality. Computer Graphics and Applications, 21(6), 34–47. https://doi.org/10.1109/38.963459
Bach, B., Sicat, R., Beyer, J., Cordeil, M., & Pfister, H. (2017). The hologram in my hand: How effective is interactive exploration of 3d visualizations in immersive tangible augmented reality?. IEEE transactions on visualization and computer graphics, 24(1), 457–467. https://doi.org/10.1109/TVCG.2017.2745941
Barfield, W. (2015). Fundamentals of wearable computers and augmented reality. CRC Press.
Bau, O., & Poupyrev, I. (2012). REVEL: tactile feedback technology for augmented reality. ACM Transactions on Graphics (TOG), 31(4), 1–11. https://doi.org/10.1145/2185520.2185585
Benko, H., Ofek, E., Zheng, F., & Wilson, A. D. (2015). Fovear: Combining an optically see-through near-eye display with projector-based spatial augmented reality. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (pp. 129–135).
Billinghurst, M., & Duenser, A. (2012). Augmented reality in the classroom. Computer, 45(7), 56-63. https://doi.org/10.1109/MC.2012.111
Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding approach on students’ scientific project performance and perceptions.Computers & Education, 125, 226-239. https://doi.org/10.1016/j.compedu.2018.06.007
Chatzidimitris, T., Gavalas, D., & Michael, D. (2016). SoundPacman: Audio augmented reality in location-based games. In 2016 18th Mediterranean Electrotechnical Conference (MELECON) (pp. 1-6). IEEE. https://doi.org/10.1109/melcon.2016.7495414
Choi, Y. M. (2019). Applying tangible augmented reality for product usability assessment. Journal of Usability Studies, 14(4). Retrieved from http://uxpajournal.org/wp-content/uploads/sites/8/pdf/jus_choi_august 2019.pdf
Cohen, L., Manion, L., & Morrison, K. (2013). Research methods in education (7th Ed.). Routledge.
Connell, B.R., Jones, M., Mace, R., Mueller, J., Mullick, A., Ostroff, E., Sanford, J., Steinfeld, E., Story, M., & Vanderheiden, G. (1997). The principles of universal design. Retrieved from http://www.ncsu.edu/ncsu/design/cud/about_ud/udprinciplestext.htm
Craig, A. B. (2013). Understanding augmented reality: concepts and applications. Massachusetts: Morgan-Kaufmann Publishers. Retrieved from https://dl.acm.org/citation.cfm?id=2513619
Creswell, J. W. (2013). Qualitative inquiry & research design choosing among five approaches. Sage Publications.
Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15(2), 1-10. Retrieved from https://files.eric.ed.gov/fulltext/EJ590767.pdf
Dede, C. (2005). Planning for neomillennial learning styles. Educause Quarterly, 28(1), 7-12. https://people.wou.edu/~girodm/middle/Dede_on_neo-millenials.pdf
Eby, G. (2013). Uzaktan eğitim ortamlarının tasarımı yazılım mühendisliği yaşam döngüsü yaklaşımı. Ankara: Kültür Ajans Publisher.
Essany, M. (2015). B-schools get an a for virtual reality experiments. Retrieved from http://mobilemarketingwatch.com/b-schools-get-an-a-for-virtual-reality-experiments-50923/
Fingas, R. (2016). Tim Cook says AI & augmented reality are core technologies in Apple’s future. Retrieved from http://appleinsider.com/articles/16/08/14/tim-cook-says-ai-augmented-reality-are-core-technologies-in-apples-future
Frank, J. A., & Kapila, V. (2017). Mixed-reality learning environments: Integrating mobile interfaces with laboratory test-beds. Computers & Education, 110, 88-104. https://doi.org/10.1016/j.compedu.2017.02.009
Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.
Gündoğan, M. B. (2012). Uzaktan eğitime ekolojik bir yaklaşım: sürdürülebilir uzaktan bilgisayar mühendisliği eğitimi ekosistemi önerisi (Unpublished doctoral dissertation). Eskişehir, Anadolu University, Turkey.
Härmä, A., Jakka, J., Tikander, M., Karjalainen, M., Lokki, T., Hiipakka, J. & Lorho, G. (2004). Augmented reality audio for mobile and wearable appliances. Journal of the Audio Engineering Society,52(6), 618-639.
Harr, J. (2015). Oculus rift technology may improve online learning. Retrieved from http://www.educationnews.org/online-schools/oculus-rift-technology-may-improve-online-learning/
Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x
Heller, F., Jevanesan, J., Dietrich, P., & Borchers, J. (2016). Where are we? evaluating the current rendering fidelity of mobile audio augmented reality systems. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp.278-282). https://doi.org/10.1145/2935334.2935365
Higbee, J. L., & Goff, E. (2008). Pedagogy and student services for institutional transformation: implementing universal design in higher education. Center for Research on Developmental Education and Urban Literacy. Retrieved from https://files.eric.ed.gov/fulltext/ED503835.pdf
Höllerer, T., & Feiner, S. (2004). Mobile augmented reality. H. Karimi ve A. Hammad (Eds.), Telegeoinformatics: Location-Based Computing and Services (pp. 1-39). London: Taylor and Francis Books. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.8006&rep=rep1&type=pdf
Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation,12(10), 1-8. https://doi.org/10.7275/pdz9-th90
Hung, H. L., Altschuld, J. W., & Lee, Y. F. (2008). Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Evaluation and Program Planning, 31(2), 191-198. https://doi.org/10.1016/j.evalprogplan.2008.02.005
Ibáñez, M. B., Portillo, A. U., Cabada, R. Z., & Barrón, M. L. (2020). Impact of augmented reality technology on academic achievement and motivation of students from public and private Mexican schools. A case study in a middle-school geometry course. Computers & Education, 145, 103734. https://doi.org/10.1016/j.compedu.2019.103734
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC horizon report: 2016 higher education edition. Austin, Texas: The New Media Consortium. Retrieved from http://cdn.nmc.org/media/2016-nmc-horizon-report-he-EN.pdf
Jot, J. M., & Lee, K. S. (2016). Augmented reality headphone environment rendering. In Audio Engineering Society Conference: 2016 AES International Conference on Audio for Virtual and Augmented Reality. Audio Engineering Society. Retrieved from http://www.aes.org/e-lib/online/browse.cfm?elib=18506
Laviole, J., Thevin, L., Albouys-Perrois, J., & Brock, A. (2018). Nectar: Multi-user spatial augmented reality for everyone: three live demonstrations of educative applications. In Proceedings of the Virtual Reality International Conference-Laval Virtual (pp. 1-6). https://doi.org/10.1145/3234253.3234317
Lee, J., Jung, S., Kim, J. W., & Biocca, F. (2019). Applying Spatial Augmented Reality to Anti-Smoking Message: Focusing on Spatial Presence, Negative Emotions, and Threat Appraisal.International Journal of Human–Computer Interaction, 35(9), 751-760. https://doi.org/10.1080/10447318.2018.1489581
Maiti, A., Smith, M., Maxwell, A. D., & Kist, A. A. (2018). Augmented reality and natural user interface applications for remote laboratories. InCyber-Physical Laboratories in Engineering and Science Education(pp. 79-109). Springer, Cham. https://doi.org/10.1007/978-3-319-76935-6_4
Metz, R. (2015). Augmented-reality glasses could help legally blind navigate. MIT Technology Review. Retrieved from https://www.technologyreview.com/s/538491/augmented-reality-glasses-could-help-legally-blind-navigate/
Metz, R. (2016). Augmented reality study projects life-sized people into other rooms. MIT Technology Review. https://www.technologyreview.com/s/545466/augmented-reality-study-projects-life-sized-people-into-other-rooms/
Moore, M. G., & Kearsley, G. (2011). Distance education: a systems view of online learning. Cengage Learning.
Neuendorf, K. A. (2002).The content analysis guidebook. Sage Publications. Retrieved from https://academic.csuohio.edu/kneuendorf/SkalskiVitae/SkalskiNeuendorfCajigas17.pdf
Núñez, M., Quirós, R., Núñez, I., Carda, J. B., Camahort, E., & Mauri, J. L. (2008). Collaborative augmented reality for inorganic chemistry education. WSEAS International Conference Proceedings Mathematics and Computers in Science and Engineering (Vol. 5, pp. 271-277). Retrieved from https://pdfs.semanticscholar.org/bf33/32d422c37f785ca6c73a8003cabb2cabd163.pdf
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications.Information & Management, 42(1), 15-29. https://doi.org/10.1016/j.im.2003.11.002
Onime, C., & Abiona, O. (2016). 3D mobile augmented reality interface for laboratory experiments. International Journal of Communications, Network and System Sciences,9(4), 67-76. http://dx.doi.org/10.4236/ijcns.2016.94006
Parviz, B. A. (2009). Augmented reality in a contact lens. Retrieved from http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/0
Pejsa, T., Kantor, J., Benko, H., Ofek, E., & Wilson, A. (2016). Room2Room: Enabling life-size telepresence in a projected augmented reality environment. Retrieved from http://pages.cs.wisc.edu/~tpejsa/files/16-CSCW-Room2Room.pdf
Pérez-López, D., & Contero, M. (2013). Delivering educational multimedia contents through an augmented reality application: A case study on its impact on knowledge acquisition and retention. TOJET: The Turkish Online Journal of Educational Technology, 12(4). Retrieved from https://files.eric.ed.gov/fulltext/EJ1018026.pdf
Perry, T. S. (2020). Augmented reality in a contact lens: it’s the real deal. Retrieved from https://spectrum.ieee.org/view-from-the-valley/consumer-electronics/portable-devices/ar-in-a-contact-lens-its-the-real-deal
Rauschnabel, P. A., Brem, A., & Ivens, B. S. (2015). Who will buy smart glasses? Empirical results of two pre-market-entry studies on the role of personality in individual awareness and intended adoption of Google Glass wearables.Computers in Human Behavior, 49, 635-647. https://psycnet.apa.org/doi/10.1016/j.chb.2015.03.003
Rauschnabel, P. A., & Ro, Y. K. (2016). Augmented reality smart glasses: An investigation of technology acceptance drivers.International Journal of Technology Marketing,11(2), 123-148. http://dx.doi.org/10.1504/IJTMKT.2016.075690
Redondo, B., Cózar-Gutiérrez, R., González-Calero, J. A., & Ruiz, R. S. (2020). Integration of Augmented Reality in the Teaching of English as a Foreign Language in Early Childhood Education. Early Childhood Education Journal, 48(2), 147-155. https://doi.org/10.1007/s10643-019-00999-5
Reid, N. (1988). The Delphi technique: Its contribution to the evaluation of professional practice. E, Roger. (Ed.), Professional Competence and Quality Assurance in the Caring Professions. London: Chapman & Hall.
Ro, Y. K., Brem, A., & Rauschnabel, P. A. (2018). Augmented reality smart glasses: Definition, concepts and impact on firm value creation. In Augmented reality and virtual reality (pp. 169-181). Springer, Cham. https://doi.org/10.1007/978-3-319-64027-3_12
Rossi, D. (2018). SAR for Kids: Spatial augmented reality as tool for art education. In International and Interdisciplinary Conference on Digital Environments for Education, Arts and Heritage (pp. 355-363). Springer, Cham. https://doi.org/10.1007/978-3-030-12240-9_38
Ruiz-Ariza, A., Casuso, R. A., Suarez-Manzano, S., & Martínez-López, E. J. (2018). Effect of augmented reality game Pokémon GO on cognitive performance and emotional intelligence in adolescent young. Computers & Education,116, 49-63. https://doi.org/10.1016/j.compedu.2017.09.002
Sahin, D., & Yilmaz, R. M. (2020). The effect of Augmented Reality Technology on middle school students’ achievements and attitudes towards science education. Computers & Education,144, 103710. https://doi.org/10.1016/j.compedu.2019.103710
Scavo, G., Wild, F., & Scott, P. (2015). The GhostHands UX: Telementoring with hands-on augmented reality instruction. Retrieved from http://www.tellme-ip.eu/sites/default/files/library/AISE19-0236.pdf
Seuring, S., & Müller, M. (2008). Core issues in sustainable supply chain management–a Delphi study.Business Strategy and the Environment,17(8), 455-466. https://doi.org/10.1002/bse.607
Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education, 6, 1-21. Retrieved from https://www.learntechlib.org/p/111405/
Smith, R. E., & Buchannan, T. (2012). Community collaboration, use of universal design in the classroom.Journal of Postsecondary Education and Disability, 25(3), 259-265. Retrieved from https://files.eric.ed.gov/fulltext/EJ994291.pdf
Sumsion, T. (1998). The Delphi technique: An adaptive research tool. The British Journal of Occupational Therapy, 61(4), 153-156. https://doi.org/10.1177%2F030802269806100403
Sun, S. W., & Lan, Y. S. (2019). Augmented reality displaying scheme in a smart glass based on relative object positions and orientation sensors.World Wide Web, 22(3), 1221-1239. https://doi.org/10.1007/s11280-018-0592-z
Takahashi, D. (2020). Mojo Vision is putting an augmented reality screen on a contact lens. Retrieved from https://venturebeat.com/2020/01/16/mojo-vision-is-putting-an-augmented-reality-screen-on-a-contact-lens/
Tashev, I. J. (2019). Capture, representation, and rendering of 3d audio for virtual and augmented reality.International Journal on Information Technologies & Security, 11(2), 49-62. Retrieved from http://ijits-bg.com/contents/IJITS-SP2-2019/2019-SP2-05.pdf
Thangaratinam, S., & Redman, C. W. (2005). The delphi technique.The Obstetrician & Gynaecologist,7(2), 120-125. https://doi.org/10.1576/toag.7.2.120.27071
The New Economy (2014). Using mobiles to smell: how technology is giving us our senses. Retrieved from http://www.theneweconomy.com/technology/using-mobiles-to-smell-how-technology-is-giving-us-our-senses-video
Topa-Çiftçi, G. (2011). Uzaktan eğitimde iptv’nin kullanılabilirliğine ilişkin bir delphi çalışması (Unpublished master’s dissertation). Eskişehir, Anadolu University, Turkey.
Tussyadiah, I. P., Jung, T. H., & Tom Dieck, M. C. (2018). Embodiment of wearable augmented reality technology in tourism experiences. Journal of Travel Research, 57(5), 597-611. https://doi.org/10.1177%2F0047287517709090
Vargas, H., Farias, G., Sanchez, J., Dormido, S., & Esquembre, F. (2013). Using augmented reality in remote laboratories. International Journal of Computers Communications & Control, 8(4), 622-634. Retrieved from http://univagora.ro/jour/index.php/ijccc/article/view/42/pdf_59
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent augmented reality training for motherboard assembly.International Journal of Artificial Intelligence in Education, 25(1), 157-172. https://doi.org/10.1007/s40593-014-0032-x
Williams, P. L., & Webb, C. (1994). The Delphi technique: A methodological discussion. Journal of Advanced Nursing, 19(1), 180-186.
Yıldırım, A., & Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri. Seçkin Publisher, Turkey.
Yin, R. K. (2003). Case study research: Design and methods. Sage Publications.
Yoon, B., Kim, H. I., Lee, G. A., Billinqhurst, M., & Woo, W. (2019). The effect of avatar appearance on social presence in an augmented reality remote collaboration. In2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 547-556). IEEE.

 

Papers are licensed under a Creative Commons Attribution 4.0 International License

Refbacks

  • There are currently no refbacks.