Comparison of the Satisfaction of Students who Use Different Learning Management Systems in Distance Education Processes

This study aims to compare the distance education process and system-related satisfaction of students using different learning management systems (Moodle, Google Classroom). Research samples of this mixed explanatory design comprise a total of 7318 students enrolled in U ş ak University for the 2020–2021 academic year. The sample categories include 3294 students using Moodle and 3332 utilizing Classroom system besides the 692 students who experienced both environments. Research data were collected by a questionnaire, semi-structured interview form and system log records. At the end of the six-week application process, a distance education process satisfaction questionnaire was applied to the students with the experience in both education software and their opinions were obtained. The results regarding the sample including students with experience on both software tools indicate the students using Classroom have higher satisfaction levels in the distance education process in system infrastructure, online lecture system, assessment, and evaluation dimensions than the ones using Moodle. The qualitative data collected from the interview form hints at problems the students experienced in the synchronous-asynchronous, test and assessment systems of the Moodle system which reduced their satisfaction.


INTRODUCTION
Recent technological developments play a certain role in the education sphere as well as in others. Technological innovations in informatics, in particular, directly affect the distance education sector and contribute to the technological development in distance education. The distance education process, starting from printed materials such as newspapers and letters in the 1800s, has now been replaced by digital and particularly web-based technologies.
During the COVID-19 pandemic emerged in the 2020s, education and training standards transformed and the transition from face-to-face education to distance education accelerated. Distance education activities have gained momentum globally, as public and private educational institutions have initiated distance education processes. The distance education practices conducted in this period were branded "emergency distance education" as all the synchronous/asynchronous technologies were utilized to perform distance education activities. Numerous distance education practices had been implemented at each education level from primary to higher.
T.R. Ministry of National Education (MONE) has intensified their mission on primary and secondary education at Education Informatics Network (EBA). Furthermore, the satellite and terrestrial television broadcasts and web platforms were utilized to maintain education without interruptions. The Higher Education Council of the Turkish Republic (YOK) tasked higher education institutions transition to conduct distance learning as all educational activities in formal universities have been transferred to distance education. Universities with a functional system infrastructure have quickly transited their formal education at the associate, undergraduate and graduate levels to distance education, while institutions with inadequate infrastructure started to install distance education system through vigorous efforts.
The 72.6% of foundations universities and 60.6% of state institutions of higher education in Turkey initiated distance education activities in March 2020 using synchronous and asynchronous learning technologies. Foundation and state universities have started to provide approximately 99% of their theoretical courses and approximately 74% of the applied courses suitable for remote teaching were performed through distance education. Moreover, most of these institutions (foundation 77.4% and state 70.8%) have used their genuine learning management systems (YOK, 2020). A learning management system (LMS) is a software tool through which management and organization of educational materials besides monitoring and supervision activities regarding the users registered on the system (student, educator, coordinator, etc.) can be performed (Ozan, 2008). These tools, also known as "LMS" in short, are web-based systems that enable online collaborations, and several educational processes from online course distribution to management and organization of educational resources. On these systems, the progress of the students can be monitored by the instructors or system administrators as well as the parents (Mahnegar, 2012).
Today, there are hundreds of LMS tools (Sasikumar, 2012). There are certain LMS with paid versions as there also free tools. Moreover, some of the LMS should be hosted on the institutions' own servers as some function with a cloud-based server. In YÖK (2020) report, LMS used by universities are analysed. The utilized LMS both by state and foundation universities in Turkey are summarized in Table 1. Table 1 presents the LMS used by universities in Turkey with usage rates. Based on this table, it is found that the most preferred LMS by both foundation and state universities is Moodle which is an open source and free web-based option. Furthermore, it was observed that many learning management systems are used in some universities. YOK (2020) report also indicates that 25.2% of the learning management systems used in Turkish universities are reported to be cloud-based. It is reported that one of the most preferred cloud based LMS is the Classroom and Edmodo. But, in the analyses made by the Council of Higher Education in Turkey, it was seen that Google Classroom was not included detailed in the LMS usage rates report.
Moodle is a learning management system that was designed by a developer named Martin Dougiamas in 1999 and first used at Curtin University in 2001. In 2002, version 1.0 was released as free and open-source code. Moodle, reaching one million users as of 2010, is used by approximately 190 million users on 145000 websites by 2020 (Moodle, 2020).
The Classroom is an LMS, which was first announced by Google in 2014, does not require any setup or infrastructure because it works on the cloud. It is found that the approximate number of users as of 2020 is 100 million (Vynck & Bergen, 2020). Classroom system, which is paid for commercial use, is offered free of charge to public education institutions. Numerous universities that did not have adequate infrastructure utilized the Classroom to initiate the urgent process of distance education. One of these universities was Uşak University.
Uşak University, founded in 2006, is a state university operating in Turkey. As of 2022, it provides education to approximately 27.000 students in 25 academic units. Distance education services have been provided in some departments of the university since 2010. In addition, some courses that are compulsory to be taught in Turkey are given to formal education students as hybrids through distance education (approximately 9.000 students) at Uşak University.
Classroom, which is the cloud-based LMS, was preferred first in Uşak University during the COVID-19 emergency distance education process. Subsequently, Moodle LMS was also utilized. Uşak University uses learning management systems for both synchronous education and homework uploads and exam applications. As the necessity of taking the student views about the distance education processes from the ones with experience on both software tools to examine which LMS is more effective, and their satisfaction with the distance education processes regarding these two systems.
This study aims to compare the students' satisfaction levels with their distance education processes. The satisfaction level regarding the learning management systems used by the students in the distance education process is measured for the technical system infrastructure, live lesson (synchronous-synchronous system) infrastructure and testing -assessment (examination) system satisfaction dimensions with a quantitative approach. Furthermore, the views of students who experienced both LMS systems were examined with a qualitative approach. The main contributions of this paper rest on three pillars. First, the determination of students' satisfaction rates for the two LMS with the crystallization of their strengths and weaknesses through the user experiences. Second, producing a guiding report on LMS selection for institutions and practitioners. Third, comparing process satisfaction in regards to Moodle and Classroom systems.

LEARNING MANAGEMENT SYSTEMS (LMS)
LMS are web-based platforms through which teaching planning, distribution, assessment and management processes can be performed (Teasley, 2009). With these technologies, also known as course management systems or virtual learning environments, operations such as creating and conveying learning contents, monitoring students and evaluating their performance are conducted (Alias & Zainuddin, 2005). According to Teasley (2009), LMS are platforms that facilitate teaching processes, improve students' learning, save time, facilitate course management, and communication between instructors and students. Moreover, it also has features such as content management, testing and assessment, teaching planning, reporting, providing collaboration and conveying announcements to students (Kulshrestha & Kant, 2009).
The success of an LMS is a reflection of students' satisfaction for using that particular system (Freeze et al., 2010;Limayem & Cheung, 2011). Students' adoption of using an LMS and their subsequent usage of it is related to their satisfaction with that system (Yuen et al., 2019). Naveh, et al. (2012) argued that some aspects that affect student satisfaction in LMS are components such as the access to course contents, the timeliness of the contents, ease of navigation and access within LMS, and fast access to the instructor. As Xu and Mahenthiran (2016) stated that distance education student satisfaction on LMS depends on the course organization and content, the ease of assessment and evaluation activities and the user-friendliness of the LMS system. According to Ohliati and Abbas (2019), factors such as the quality of information and infrastructure and the friendliness perceived by students affect their satisfaction with LMS.  Pino et al. (2017) defined satisfaction as the state of meeting students' expectations from the educational institution. Elliott and Healy (2001) and Weerasinghe and Fernando (2017) defined satisfaction as the process of developing an attitude by evaluating students' learning experiences. Moreover, Elliott and Shin (2002) defined satisfaction as the subjective evaluation of learning outcomes and learning experience.

DISTANCE EDUCATION PROCESS SATISFACTION IN LEARNING MANAGEMENT SYSTEMS
Student satisfaction after a distance learning experience on LMS is one of the most important factors affecting their learning performance and achievement. Korkmaz et al. (2015) asserted that there is a positive correlation between students' satisfaction levels in distance education environments and their achievements. Therefore, it can be argued that monitoring and tracking the student satisfaction in a distance learning with LMS is substantial for ensuring an efficient process.
The study conducted by Kuzu and Balaman (2014) obtained the students', with a learning experience on Moodle, views. It was found that the students were satisfied with Moodle, felt comfortable in the environment and were motivated. Moreover, it was revealed that they experienced communication difficulties with their friends and instructors and had problems in accessing the system due to computer and internet problems hence a reduced student satisfaction. Suner (2018) analysed the satisfaction levels of students, with a learning experience on Moodle LMS, regarding the distance education process and found that they had high satisfaction rates. There are numerous attempts, on the impact of using Classroom and Moodle in distance education, in the literature that indicate and increase in students' satisfaction levels after using Classroom (Filho et al., 2019;Shaharanee et al., 2016).
This study analyses the satisfaction levels of students with a learning experience on different LMS regarding distance education processes. In this regard, the aim is to reveal students' satisfaction levels regarding the distance education processes on different LMS, and to assess their current circumstances in detail through evaluating their opinions about these processes.

RESEARCH MODEL
Explanatory mixed design (Creswell & Clark, 2007) was used in this study. The mixed model synthesizes quantitative and qualitative approaches. It is the approach in which qualitative data are collected to complement and support quantitative data (Büyüköztürk et al., 2013). The mixed descriptive design was selected to support the obtained quantitative data with qualitative data, ensure the data integrity, and reach more concrete inferences. The distance education process and system satisfaction student groups using different learning management systems were evaluated and compared.

RESEARCH SAMPLES
Research samples of this study comprise a total of 7318 from 29813 students enrolled in Uşak University for the 2020-2021 academic year and approved to participate in this study with the voluntary online questionnaire and interviews. The sample categories include 3294 students using Moodle, 3332 operated the Classroom system and including the 692 students who experienced both environments.

DATA COLLECTION TOOLS
The distance education process satisfaction questionnaire developed by the author as the quantitative and a semi-structured interview form as the qualitative data collection tool besides LMS system records (log) was used for the sample comprises students who experienced both learning environments.
The questionnaire is a data collection tool including a number of questions to describe the participants' attitude, belief, behaviour or life conditions (Thomas, 1998). As the distance education process evaluation questionnaire is a five-point Likert-type (1: strongly disagree, 5: totally agree) questionnaire with six graded questions. First, aims and questions were 100 Yilmaz Open Praxis DOI: 10.55982/ openpraxis.14.2.152 determined in line with the research problem in the survey development process. As items were prepared to assess the satisfaction levels of the students with a learning experience in different LMS environments through their distance education processes. Three field experts were consulted about the accuracy of the items for the questionnaire's content validity test. The questionnaire items were finalized in line with expert opinions.
The other qualitative data collection method used in the study is interviews. The interview technique was selected to elaborate on the feelings, thoughts and opinions towards the distance education process and to make sophisticated inferences about the quantitative data. Students' views were collected online via a written semi-structured interview form. Three main themes were determined by the author as "opinion", "complaint" and "suggestion" regarding the distance education process and question texts were prepared. Open-ended questions directed to students are as follows: • What do you think about the distance education process?
• Did you have any difficulties and complaints during the distance education process, if yes please specify...
• Do you have any suggestions about the distance education process?
The qualitative data obtained from the interview forms guided the category formation, evaluation and coding by two field experts. Moreover, the inter-coder reliability was measured with Cohen's Kappa statistics and the analysis was also reported.

LEARNING MANAGEMENT SYSTEMS
No customization or changes has been made on default learning management systems to affect their perceptions of students. Any theme installation, plug-in/module installation etc. processes were not carried out. The default versions of learning management systems were used. Moodle version 3.9 is used. The cloud-based default version of Classroom software provided by Google was used.

APPLICATION PROCEDURE
During the application process, 3332 students experienced only Classroom LMS and 692 students utilized both LMS to finalize their education on Classroom for six weeks. Students had a distance learning experience on Classroom for six weeks, attended their daily synchronous (live) lessons, used asynchronous course materials provided by the instructors, and took online exams via the LMS. A distance education process satisfaction questionnaire was applied at the end of the application to 3332 students who experienced only Classroom LMS.
After the first application 3294 students experienced only Moodle LMS as 692 students in the group with experience in both LMS, continued their education on Moodle LMS for six weeks. A distance education process satisfaction questionnaire was applied at the end of the application to 3294 students who experienced only Moodle LMS and to 692 students with experience in both systems. Moreover, the views of students with experience on both environments in distance education processes were collected online via a semi-structured interview form.

DATA ANALYSIS
Descriptive analyses were made primarily of the data obtained from the measurement tools. The percentage, average and frequency tables were formed in this purpose. A t-test analysis was performed to test whether the distance education process assessment questionnaire scores differ for the students using different learning management systems. Survey score ranges are as follows (Tekin, 1991) 4.20-5.00 very satisfied, 3.40-4.19 satisfied, 2.60-3.39 partially satisfied, 1.80-2.59 not satisfied, 1.00-1.79 not satisfied at all. The item score averages of two student groups who experienced only one LMS were analysed by independent samples t-test. For the sample including students with experience in both learning environments dependent samples were analysed with the t-test. SPSS program was used for statistical analysis. It was also examined whether the data were suitable for parametric analysis before the parametric tests. The -2/+2 interval specified by George and Mallery (2010) was considered as an acceptable range for kurtosis and skewness values in the data analysis. Furthermore, effect size values obtained through the tests (eta squared-η2) are also reported. The significance value is designated as p < .05.
Opinion forms obtained from students who experienced both learning environments were analysed with content analysis through qualitative methods. Content analysis is a technique performed to obtain objective results by making valid and systematic inferences from qualitative data (Krippendorff, 2004). Student messages were examined by two field experts and coded into categories under three themes for content analysis. The data obtained are presented in percentage and frequency tables.

FINDINGS DESCRIPTIVE FINDINGS REGARDING ALL UNITS
The distribution of 7318 students participating in the study by the environment they experience is detailed in

DESCRIPTIVE FINDINGS REGARDING DISTANCE EDUCATION PROCESS SATISFACTION OF STUDENTS USING EITHER MOODLE OR CLASSROOM
A distance education process questionnaire was applied to a total of 6626 students who experienced either Classroom or Moodle environments. Descriptive findings of the questionnaire are shown in Table 3.   It can be asserted that only students who have had a learning experience in Classroom LMS are partially satisfied with their distance education processes (X: 3.06). The general averages indicate that the distance education process satisfaction of students using only Moodle (X: 2.63) is slightly lower than the students using only Classroom (X: 3.06). Whether this difference between the averages is significant or not is elaborated in the next sections.

COMPARISON OF STUDENTS' LMS SATISFACTION BY GENDER
Whether the satisfaction levels of the students who had a learning experience either on Classroom or Moodle LMS differ by gender was analyzed by t-test. The statistical analysis results of the t-tests performed are presented in Table 4. Table 4 shows that the distance education process satisfaction of the students who only experienced Classroom shows a significant difference by gender [t (3206.42) = 4.666, p < .05]. The distance education process satisfaction average (X: 3.16) of male students who only experienced the Classroom environment was higher than female students (X: 2.98). It is revealed that there is a significant difference with a moderate impact value across gender-based groups of students using only the Classroom environment (η 2 = 0.065). The analysis presented in Table 4 hints at a finding that the distance education process satisfaction of the students who only used Moodle shows a significant difference by gender [t (2716.24) = 3.447, p < .05]. The distance education process satisfaction average of male students who only experienced the Moodle environment (X: 2.69) was higher than females (X: 2.57). The independent sample t-test indicates that the small effect value of the students using experienced only Moodle environment differs significantly by gender (η 2 = 0.004).
The general averages indicate that the distance education process satisfaction of students using only Moodle (X: 2.63) is slightly lower than the students using only Classroom (X: 3.06). Independent samples t-test was used to examine whether these average differences were significant. The statistical analysis results of the t-test are provided in Table 5. Table 5 points to a situation that students using Classroom have significantly higher distance education process satisfaction than the ones using Moodle. [t (6545.85) = 17.232, p < .05]. The independent sample t-test revealed a significant difference with a moderate effect value among the student groups using Classroom and Moodle environments separately (η 2 = 0.042).

STUDENT GROUP COMPARISONS REGARDING SAMPLES USING EITHER CLASSROOM OR MOODLE BASED ON SURVEY ITEMS ON DISTANCE EDUCATION PROCESS
The distance education process evaluation questionnaire items were also statistically analyzed and compared separately for each group (using either Classroom or Moodle environments). Item-based independent samples t-test analysis results are summarized collectively in Table 6.   The analysis results in Table 6 reveals that there are significant differences in all questionnaire items in favor of Classroom experience.

COMPARISON OF DISTANCE EDUCATION PROCESS SATISFACTION OF STUDENTS USING BOTH MOODLE AND CLASSROOM (PAIRED SAMPLES T-TEST)
This section summarizes the findings regarding 692 students who experienced both LMS. The change in distance education process satisfaction and gender-oriented satisfaction of the student group who experienced both environments was examined by a paired t-test (Table 7).    Table 7 reveals the findings of comparison regarding the distance education process satisfaction of the students who experienced the same environments on item-based and holistic contexts. Regarding students' desire to continue their distance education there is no significant difference between Classroom and Moodle. In other words, students' willingness to continue their distance education process does not change according to the LMS they use within the scope of this study. Moreover, it can be argued that there is a significant difference in favor of students using Classroom LMS for all remaining questionnaire items.

COMPARISON OF DISTANCE EDUCATION PROCESS SATISFACTION OF STUDENTS EXPERIENCED THE SAME ENVIRONMENT BY GENDER (INDEPENDENT SAMPLES T-TEST)
The statistical analysis results of the t-test are provided in Table 8. Table 8 illustrates the analysis report on the comparison of the general distance education process satisfaction levels of male and female students who experienced both Classroom and Moodle environments. Accordingly, the satisfaction average scores of male students who experienced Classroom were significantly higher [t (382.466) = 3.496, p < .05]. Similarly, satisfaction average scores of male students who experienced Moodle LMS were significantly higher than females [t (353.842) = 4.084, p < .05]. As it can be epitomized the satisfaction level of the distance education process of male students who experienced both environments are significantly higher.

STUDENT VIEWS-ORIENTED FINDINGS
692 students who experienced both LMS were asked to submit their views on the distance education process, the difficulties and problems, and their suggestions for the online process using a one-time semi-structured interview form after their learning experiences. A total of 420 messages under three themes in the opinion forms transmitted by 279 students were examined with content analysis. As 119 of the messages conveyed are opinion messages, 162 of them are complaints and 139 are suggestions. The messages transmitted under 3 different themes were transferred to the Excel program and relevant categories were created. There were 7 categories under the opinion, 6 under the complaint and 7 under the suggestion themes. The theme categories (Holsti, 1969) were specified as distinctive and differentiating, holistic, objective, comprehensive, suitable for the research purpose and meaningful. A total of 420 messages under three main themes were coded with a category table guide by two field experts. All messages and categories were transferred to an Excel table, as the experts were informed about the research in the digital environment. The consistency between experts' evaluations was examined using Cohen's Kappa statistics. The Kappa scores obtained were graded with score ranges specified by Landis and Koch (1977) (<0.00 weak, 0.00-0.20 very low, 0.21-0.40 low, 0.41-0.60 good, 0.61-0.80 very good and 0.81-1.00 excellent harmony).
Inter-coder reliability was examined separately for messages sent under three main themes and for all messages. The examination of 119 messages sent under the opinion theme (7 categories) indicate conformity between the two coders κ = .824 (p < .05) excellent harmony level as the ratio for complaint theme (6 categories) regarding 169 messages was κ = .852 (p < .05) excellent harmony level and for suggestiontheme (7 categories) of 139 messages κ = .850 (p < .05) an excellent conformity. Also, intercoder reliability for all messages transmitted indicates excellent harmony with a ratio κ = .845 (p < .05).
The distribution of student views by themes and categories is demonstrated in Table 9.   Table 9 reveals that students were not satisfied with their distance education experience (f:63, 44.68%), and that they had difficulties in the process due to device and internet access problems (f:23, 16.31%). However, 16.31% (f:23) of the students stated that they were satisfied with the distance education process. The students also stated that they could not learn and understand the subjects in distance education (f: 16, 11.34%) and live lessons took too long (f:4, 2.83%).

LMS
The categorical distribution of the messages sent for complaint purposes conveyed indicates that the most frequently mentioned subject was Moodle LMS's online exam system (f:89, 44.45%). The second most mentioned complaint subject was regarding the problems with the LMS operations on the Moodle system (f:57, 25.90%). Students also expressed their complaints regarding the difficulties in live synchronous lessons, previous course video recordings, access problems in asynchronous course areas on the Moodle system. The students did not express a complaint message regarding the Classroom system.
Student suggestions mostly centralized the return to face-to-face education instead of distance education (f:36, 21.56%). Students also suggested homework assignments and software updates in the Moodle LMS instead of online tests (f:34, 20.35%). They also stated that Classroom LMS should be preferred over Moodle (f:33,19.76%), there should be more interaction and lesson duration should be shortened in live synchronous lessons. The portion of students who said that distance education should continue was 9.58% (f:16).

CONCLUSION AND DISCUSSIONS
Distance education process and system satisfaction of student groups using different learning management systems were compared with a mixed model. This section presents a thorough discussion of the research findings within the framework of the relevant literature. Furthermore,  recommendations for researchers, distance education practitioners and institutions are included.
The quantitative findings obtained from the distance education process questionnaire indicates that students using Classroom only had significantly higher overall distance education process satisfaction than the ones used Moodle only. However, it can be argued that students who utilized both LMS were significantly more satisfied with the distance education process they experienced in Classroom. It was revealed that the students were significantly more satisfied with the distance education processes, knowledge and competencies of instructors, the technical infrastructure of the system, the live classroom system (synchronous), the online testing system on Classroom compared to ones using Moodle. It can be asserted that results of this research are in parallel with several studies that revealed students' satisfaction levels are positively affected by the use of Classroom (Filho, et al., 2019;Shaharanee et al., 2016). Çalışkan et al. (2017) stated that student satisfaction in the distance education process is generally high. Accordingly, Goyal and Purohit (2011) reached results indicating that the use of Moodle LMS positively affected students' expectations and satisfaction in their study. Umek et al. (2015) conducted a study on student satisfaction levels for Moodle-based e-learning platforms and revealed that students' satisfaction levels increased. Similarly, Suner (2018) found that students using the Moodle system have high satisfaction levels regarding distance education process. However, this study produced results contrary to some studies in the literature implying the distance education process satisfaction of students using Moodle was significantly low. Damnjanovic et al. (2015) discovered that the software system does not affect student satisfaction. Barman and Karthikeyan (2019) stated that the Classroom environment does not have a complex structure making it user friendly compared to other LMS as Moodle has a more complex structure. It can be argued that the students are more satisfied with the distance education processes on the Classroom system may be related to the user friendliness. However, student views hint that there are substantial amounts of complaints about the Moodle software system as numerous suggestions for improvement.
The student views points to an argument that the majority of the students (44% of total) think that distance education is inefficient and they are generally not satisfied with the processes. The student complaints include difficulties in testing systems (40% of total), as some express their dislike towards Moodle LMS as they experienced difficulties (25% of the students who submitted suggestions). The qualitative data obtained from student opinions indicate problems such as sound interruption, image freeze, and disconnection in the live lesson (synchronous virtual classroom) on Moodle system (14.54% of the students who submitted suggestions). However, students highlighted difficulties in accessing previous lecture video recordings on Moodle and having problems logging into the system. These difficulties experienced by the students are considered as factors that negatively affected their satisfaction with the distance education processes on the Moodle system. The study conducted by Kuzu and Balaman (2014) obtained the students', with a learning experience on Moodle, views. As it was found that the students were satisfied with Moodle, they felt comfortable and motivated in the environment, but this study produced contradictory results implying that students were generally not satisfied with Moodle LMS.
The suggestions submitted by the students regarding the distance education processes include face-to-face education should replace distance education, homework should be assigned instead of online tests, and Moodle software system should be improved. Another noteworthy suggestion made by the students was to use Classroom instead of Moodle in distance education processes (19.76% of the students who submitted suggestions). It can be argued that the students are not satisfied with the Moodle system and they want Classroom to be preferred instead.
Other suggestions include increased teacher-student interaction in synchronous live lessons and shortened time of live lessons. The log records of distance education systems show that the lessons (approximately 30 thousand live lecture videos each) were 46 minutes 31 seconds on average. Several studies in the literature indicate that the average length of live lessons should be 20 minutes (Osipov et al., 2015;Ferriman, 2015). It can be asserted that the students were negatively affected by long live virtual lessons.