The traditional rules for English language teaching and learning in the 21st century have evolved to include every unique innovation of technology that come its way. Language learners have also mastered technological skills in order to succeed in this century as effective learners. Technology has therefore been used as an ideal medium for students to increase exposure and improve their level of proficiency (Li, 2012).
Institutions on the other hand are expected to be innovative in the use of technology. Hence, they carefully select educational technologies that are flexible enough for not only distance or open learning but also to ensure that learners become independent and responsible for learning (Lecercle, 2011). The use of technology has therefore ensured the removal of barriers to learning for learners to study in their preferred time, place of their choice without a direct contact with an instructor (Stewart, 2013). Some examples of such programs include Learning Management System (LMS), English Language Learning Instruction System (ELLIS) and Tell Me More (TMM).
Some reasons, which could account for the use of open learning technologies in language learning is its ease of access for self-study, capacity to increase learners’ motivation, improve learners’ mastery of language skills and consequently make learning student centered through active engagement in the learning process (Guemide & Benachaiba, 2014). Due to these benefits, language-learning software developers such as Tell Me More (TMM) have constantly advertised to learners and institutions to purchase their products in other for them to reach their learning goals with ease. Additionally, these software developers assure users of an improvement in their overall communication skills as fast as possible. Moreover, these software developers guarantee a better language achievement when their products are used according to specific guidelines such as having specific contact hours.
Studies have however been done on some of these computer stand-alone learning programs that claim to have a comprehensive solution to language learning to improve learners’ performance or achievement (Mohsin, 2012; Elimat & AbuSeileek, 2014). As regards Tell Me More (TMM), studies have either focused on the perceived effectiveness of these programs on users’ language ability, usefulness and ease of use. There are, however, a few researches to back the claims that these programs have the capacity to improve learners’ overall language achievement (Yunus et al., 2010; Barrios, 2013; Perez, 2014).
It is against the backdrop of TMM guaranteeing an improvement in overall language achievement when they are used in accordance with specific guidelines that this research was undertaken. This paper therefore reports descriptive and statistical findings of students’ achievement when they used the stand-alone computer assisted language-learning program (CALL), Tell Me More (TMM). This study explored the effect of TMM on the achievement learners of different levels of proficiency. It also investigated whether the proficiency groups differed from each other. Finally, the study examined whether time of use of the program brought an improvement in students’ achievement.
Tell Me More, an asynchronous online learning system, is one of the advanced self-learning tools that may have a comprehensive solution for language learning. The courseware contains lessons that make interaction and second language learning and acquisition possible. Tell Me More covers elements of different topics and context that enable students to practice their listening, speaking, reading and writing skills. TMM online has five different levels of proficiency from beginner to advanced, which correspond to the levels A1 to C1 of the framework Common European of Reference of Languages of the Council of Europe (figure 1).
Tell Me More seeks to tutor learners by exposing them to over 850 hours of learning content, 4,500 exercises and 37 types of activities in six categories: Lesson Workshop, Cultural Workshop, Vocabulary Workshop, Grammar Workshop, Oral Workshop and Written Workshop. Learners choose the level of their preference to define the learning goals and skills they want to improve either linguistic and communication skills (TMM manual).
Tell Me More adopts the role of tutor or instructor and distinctly possess the potential role of giving meaning, controlling the process of learning, giving feedback and evaluating learning. The content of the online learning platform has further been structured around authentic events such as at the airport, weather forecast, a linguistic function and listening to a dialogue on scenarios of communication. This is followed by an activity of interaction (limited by the options offered by the program) and other pronunciation, standard activities of vocabulary and grammar (crossword puzzles, dictation, association exercises sort words, etc.) (TMM manual).
TMM is embedded with functions to detect speech through pronunciation, phrasing, intonation errors and displays a graphical feedback by showing errors after it has been compared with a native model (Blake, 2011). Godwin-Jones (2010) pointed out that the fast rate at which web language programming is developing has allowed online English language application developers such as Tell Me More to incorporate dimensions such as it interactive and audiovisual elements to make current versions sophisticated and meet the demands of the modern times.
However, several studies and reviews CALL revealed complexities in some programs (Alsied & Pathan, 2013; Amaral, & Meurers, 2011). These complexities include the graphics quality, the audio, video and photographic content, its speech recognition and visualization, the user-friendliness and usability of the learning environment of CALL programs.
The program has various aspects; students could select the skill they wish to learn or develop (figure 2). The learners could choose from activities organized around listening, speaking, reading, writing or all of the available skills. Each skill has specific components, which is aimed at improving the language ability of learners.
For example, the listening and speaking parts have been structured around everyday situation or business related situations such as, at the airport, weather, culture, history etc. and a series of oral expressions. The writing and reading parts are organized around linguistic functions such as introducing yourself and samples of reading tasks.
As regards the vocabulary and grammar aspects, TMM incorporate traditional activities such as crossword puzzle, dictations, word association activities, verb conjugation and word ordering exercises etc.
The program provides various functions for students develop their pronunciation with the automatic speech recognition.
All these aspects have been organized according different levels of difficulty (proficiency levels) and topics.
There are a few existing empirical studies on the effectiveness of the stand-alone CALL programs in improving users’ specific language skill or overall language achievement (Al-Qudah, 2012; Mohsin, 2012; Perez, 2014). Many researches on CALL have focused on how computer-learning programs promote interaction and how learners interact with the program specific language ability to the disadvantage of an evaluation of the effect of CALL programs on learners’ overall achievement (AbuSeileek, 2012; Hurkmans & Goos, 2013). As regards Tell Me More, researchers have focused on learners’ perceptions, attitudes and its perceived effectiveness in improving specific language skill.
For example, Barrios (2013) research at the university of Malaga, Spain on the perspectives of 75 teachers who enrolled in Tell Me More for a period of six months showed a degree of satisfaction with the program between moderate and low in terms of interest, usefulness and effectiveness to train in a spontaneous oral English and communicative use. The data indicated that respondents saw a moderate breakthrough in some communication and language skills such as oral and written comprehension, vocabulary, grammar or pronunciation. Also, some components and features of the program, for example, the technology of speech analysis that it incorporates, although they generated discontent and criticism among some users, accounted for other benefits. This circumstance showed that Tell Me More as a self-instruction tool was effective to some degree and that accounted for degree of satisfaction.
A study by Van Han and Van Rensburg (2014) on the effect of Computer Assisted Language Learning on the listening performance of students in the Test of English for International Communication (TOEIC) revealed a difference in scores between the treatment and the control group. The findings also revealed that students in the treatment group used effective listening strategies during the TOEIC test than the control group.
In another study by Yunus, Hasim, Embi and Lubis (2010), of 85 users who were University students and four lecturers in Malaysian University on their utilization of Tell Me More, the student participants found it useful for learning English. This was so because it helped them improve their proficiency in English. Participants in the study valued the adequacy of the program to improve communication, grammatical and lexical skills. They were pleased with TMM’S potential to facilitate learning and the originality of the materials and activities. The lecturers also indicated that the courseware was a useful supporting tool and it affirmed their positive perception of its suitability, ease of use and usefulness.
Li (2012) investigation into 160 students practices and attitudes during autonomous online learning revealed the students’ positive attitude towards Computer Assisted Language Learning. The participants revealed the effectiveness of CALL on their English language ability. They further reported that the continuous use of CALL programs would help them overcome problems such as ineffective learning strategies and limited oral and listening ability.
Nielson (2011) study on adult learners who used Rosetta Stone and Tell Me More to improve their proficiency in Spanish, Arabic and Chinese revealed that despite the ease of accessing the software, learners lacked compliance in using the resources due to compounding technological problems and insufficient support for their autonomous learning. This resulted in participants’ gradual loss of interest in the programs.
Another study by Perez (2014) on both paramedical and medical students in a Philippine university, students revealed no significant difference in students’ responses in relation to the effectiveness of Tell Me More in enhancing their communication skills. Users however disagreed that they encountered difficulties while using the language resource.
DelliCarpini (2012) emphasized the need to incorporate CALL programs into English language teaching and learning not only to ensure the development of online learning skills but also for learners to acquire the second language. Rodinadze & Zarbazoia (2012) also stressed that the improvement of receptive and productive skills, multimedia presentation, collaborative document editing and knowledge management are some of the benefits learners derive from educational technologies.
The above studies showed that research on TMM have focused primarily on users’ interaction, satisfaction, attitudes, usefulness and perceived effectiveness. Though useful to research, they do not give insight on the effect TMM has on the overall achievement of different proficiency levels. Additionally, what is lacking in research on TMM is the impact of recommended user guidelines such as time of use has on learners’ achievement. These reasons make it necessary to conduct a descriptive and statistical study on the effect TMM has on learners’ overall achievement test. To find this out, this study aimed at the questions below:
Tell Me More has been used as a supplementary courseware in Prince of Songkla University, a University in the south of Thailand, for some time (http://tmm.psu.ac.th/). The participants for this study were first year students who had enrolled and used TMM as part of a Fundamental English Reading and Writing course in the Academic Year 2015. The students studied different undergraduate degree programs. Before they used the program, they took a placement, progress and an achievement test incorporated in the TMM program. The placement test categorized the students into different levels of proficiency according to the number of items they answered correctly. The student could only use the program after taking the placement test. The students were expected to use the program for specific number of contact hours based on the level of proficiency they attain in the placement test (figure 3). The beginners were supposed to use the program for 50 hours, 40 hours for the intermediate level, 30 for the intermediate+ level and 20 for the advanced level. The students took the placement test at the beginning of the term to determine their level of proficiency. The progress test assessed the progress learners have made since they started using the program at the level assigned. The achievement test gave an overall evaluation of what the learners have learned at the end of the course.
They took the progress and an achievement test in the middle and at the end of the term to measure their progress and overall achievement respectively. The three tests were at different levels of difficulty. However, the achievement test was at a higher level of difficulty, which is comparable with standard tests such as Test of English for International Communication (TOEIC). This made the tests highly reliable.
Moreover, to make this online program more successful, the University required that the students to should assessed based on their use of TMM. The students were awarded 2% upon the fulfillment of the required hours of use. The administrator of the program in each faculty tracked the performance of students and reported them to the departments at the end of the 2015 academic year.
The data for the study was from first year undergraduate students who used the program in the 2015 Academic Year. The scores of 2,137 students who successfully completed the placement, progress and achievement tests in the TMM program were selected for analysis. The data were provided with consent from the administrators of the program at the Center for Learning Promotion and Development at the University.
The test instruments that were used to measure the improvement in learners’ proficiency were the placement, progress and achievement tests. These tests were incorporated into the full TMM learning package. The placement test was used to determine the level of proficiency of students; beginner, intermediate, intermediate+ and advanced. The progress test measured their progress over time and the achievement test aimed at measuring their accomplishment or knowledge after using the program for the required number of hours. The Tell Me More program provided students activities and games to improve the language ability. These activities were structured around dialogues, puzzles, picture associations, videos and questions, speech recognition, cultural texts etc. The placement and progress tests had 60 items each and were scored 10 points each. Both of the tests were at a similar level of difficulty. However, the achievement test was scored out of 800 points and was at a higher level of difficulty (figure 1). The TMM administrators tracked all activities of students on the program including time of usage.
The data was subjected to descriptive statistical analysis through which the following, frequencies, percentages, means, standard deviation and Z scores were derived. Since the tests were scored differently, a Z score analysis was done to compare and standardize them at the various proficiency levels. The Z scores difference between the placement and achievement test were computer to find out whether there was any improvement. However, the Z progress test scores were not used in analysis because it was at a similar level of difficulty with Z placement test and any difference between them may be due to chance since. This analysis was carried out because the tests were scored differently. A Z score ANOVA analysis and a pairwise comparison were conducted to find the difference between the proficiency levels.
The analysis of the placement test result indicated that the intermediate proficiency level had the highest number of students 846 (39%), followed by the beginners with 676 (32%) students. The intermediate+ level also had 450 (21%) students while the least was the advanced level with 165 (8%) students.
The comparison of the mean and Z score analysis for the Placement and Achievement tests scores in each proficiency level in table 1 were as follows. For the beginners, the mean and Z score in the placement test was (= 2.39, z= -0.96), progress test (= 3, z= -0.72) and achievement test (= 285.89, z= - 0.59). The Z achievement score reported for the beginners in all three tests showed an improvement in students’ achievement.
|Tests||Beginner (n=676)||Intermediate (n=846)||Intermediate+ (n=450)||Advanced (n=165)||Total (n=2137)|
|4.ZPlacement Test||- 0.96||0.23||- 0.20||0.29||1.01||0.44||2.23||0.25||0.00||1.00|
|5.ZProgress Test||- 0.72||0.49||- 0.29||0.58||0.90||0.72||1.97||0.41||0.00||1.00|
|6.ZAchievementtest||- 0.59||0.34||- 0.38||0.44||0.77||0.84||2.27||0.79||0.00||1.00|
|7.Zdiff (6-4)||0.37||0.40||- 0.18||0.45||- 0.24||0.68||0.04||0.70||0.00||0.58|
The mean and Z score for the intermediate level in all the three tests were as follows: placement test (= 3.86, z= -0.20), progress test (= 3.89, z= -0.29) and achievement test (= 306.66, z= -0.38). For the intermediate+ level: the mean and Z score for the placement test (= 6.22, z = 1.01), progress test (= 6.34, z= 0.90) and achievement test (= 419.38, z= 0.77). The Z achievement score reported for these levels indicated a drop in achievement.
The advanced proficiency level students had means and Z scores as follows: placement test (= 8.62, z= 2.23), progress test (= 8.53, z= 1.97) and achievement test (= 566.42, z= 0.77). The Z achievement score of the advanced group showed little improvement from the level they started.
A further analysis of the differences between the means of the Z scores of the placement and achievement test scores (Z diff 6-4) revealed a Z difference as follows beginner (z = 0.37), intermediate (z = -0.18), intermediate+ (-0.24) and advanced (0.04). This means that while the beginners got the highest improvement in the achievement test followed by the advanced groups, the intermediate and intermediate+ groups had a drop in their achievement.
From table 2, the analysis of the differences between the proficiency groups by comparing the Z difference using a one-way between groups ANOVA indicated a statistically significant difference between the four levels of proficiency at a significant level of p<.01 [F (3,2131) =1597.386, p=.000]. A pairwise comparison in table 3 further showed that the groups were statistically different from each other and were of different levels of ability.
|Sum of Squares||df||Mean Square||F||Sig.|
The mean score of hours of use of the program for the proficiency levels were analyzed in table 4. The result indicated that the beginners spent an average time of 74.46 hours on the program. The intermediate group recorded an average time of 70.16 hours on the program. The intermediate+ group spent an average of 49.07 hours on the program while the advanced group spent an average time of 22.39 hours on the program.
A comparison between the z placement and achievement test scores in table 1 indicated an improvement in the level of English for the beginner and advanced groups. This means that the TMM program improved the achievement of students at both the beginner and advanced levels. For the beginners, this confirms the findings in Lin (2014), that the use of technology that incorporates concepts and organizes information have a positive impact on students at low proficiency level. On the other hand, the intermediate and intermediate+ groups had little to no improvement in their achievement test. In general, the students at these levels had no progress in their achievement after using the program. The results might suggest that the TMM program is effective for students at the beginner level and to some extent effective for students at the advanced level. What could account for the no and limited achievement in the intermediate and intermediate+ groups respectively could be the mode of assessment. Assessment for the use of the program was based on time on task. The students may have focused on fulfilling the time requirement rather learning the content in the program. Melor (2007) also pointed out that computer access, time constraints, individual computer skills and hardware issues, learner socio-cultural backgrounds, previous knowledge and online learning experiences all have an effect on learning process which may affect achievement.
It could be concluded from the comparison of the Z difference using ANOVA p<.01 [F (3,2131) =1597.386, p=.000]. In table 2 that the learners significantly differed from each other. The significant differences between the various proficiency levels further confirmed why the TMM has a different impact on learners of different levels of proficiency. The students at the intermediate, intermediate+ and advanced levels may have found the program easy to do and not challenging enough to stretch them beyond their limits. The students at the beginner level may have seen the program as an opportunity to improve their level of English and may have exerted much effort. This signifies that the TMM program is suitable for students of lower to intermediate levels in English.
The analysis of the time on task on the TMM program was striking. The findings in table 4 indicated that despite the specific time requirement for each proficiency level, the beginners spent an average time of 74.46 hours in using the program. This was more than the required 50 contact hours. This may have accounted partly for the improvement in their achievement. For students at this level, the more time they spent on the program the better they became. The intermediate and intermediate+ groups on the other hand spent an average of 70.16 hours and 40.07 hours with the TMM program respectively. For students at the intermediate level, this was over the required 40 hours of use. Unlike the beginner group, they had no improvement their achievement. Interestingly, the more hours the students at this level spent using the program, the more they dropped. The average time of use recorded for students at the advanced level was 22.39 hours. The students at this level had a limited achievement even though they used the program a little over the 20 hours required.
These findings mean that time of use is beneficial to improving learners’ English ability especially for students of lower proficiency level. It confirmed the study that time of use is beneficial to learning achievement (McDaniel, 2011). Yet, it should not be the sole criteria for assessing learning progress and achievement, because students at a higher level who find the content of a language program not challenging enough made leave the program on to count the time to fulfill the course requirement. What may also hold is that students at the higher level may finish doing the activities in the program before time. Hence, for assessment purposes, the only option left is to leave the program on to count the time. This however raised concerns about the use of only hours of use as a measure of learning progress.
The study revealed the effect of Tell Me More on EFL students’ achievement. It further showed how assessment played a role in students’ achievement.
In this context, TMM had a positive impact on learners at the beginner in terms of their achievement. It also had a minimal impact on the achievement of users at the advanced level. However, TMM had no effect on the achievement of learners at the intermediate and intermediate+ levels. What could have caused the minimal to no improvement in the advanced, intermediate and intermediate+ levels is the manner of assessment (time of use). Time of use, which was used as the method of assessment, may have influenced students to use the program in unbeneficial ways. In other words, the students may have focused on meeting the time requirements for assessment purposes rather than learning the content of the program. The findings on students spending more hours on the program but achieving less attest to this (tables 1 & 4). Hence, TMM could be more effective in terms of enhancing learners’ achievement if follow context specific guidelines.
The study concludes that although contact hours with a computer assisted language learning program is beneficial especially for beginners, it should not be the sole indicator for measuring learning progress and achievement. Other innovative means of assessment such as giving specific and measurable learning goals to different proficiency levels based on their needs will be needed to complement the time requirement.
Despite the findings, this study had some limitations. The data was obtained from students’ scores in an academic year, therefore, factors such as perceptions, students’ ability, learning practices, previous experience of learning with technology, motivation, personality types and previous knowledge of English that may influence students in the learning process were not considered. Data on the above could be obtained from interviews and surveys. Furthermore, due to the descriptive and a little statistical nature of the study, the result may not be generalized to other users of the TMM program. Hence, a robust statistical analysis needs to be conducted to know the correlations between the factors such as time, proficiency level, perceptions and attitudes that may affect learners’ overall achievement and even on specific language skills.
Additionally, a follow up study is needed on what learners do (practices) or how learners interact when they log on to the TMM program. Such research will not only contribute to and expand the future knowledge base of computer assisted language learning but it will also guide and improve instructional and curriculum design for autonomous learning. Moreover, such study will provide additional insight to institutions of higher education who are considering implementing the TMM program or any computer-assisted language-learning program in their instructional design.
This work was supported by the Higher Education Research Promotion and Thailand’s Education Hub for Southern Region of ASEAN countries Project Office of the Higher Education Commission.
AbuSeileek, F.A. (2012). The effect of computer-assisted cooperative learning methods and group size on the EFL learners’ achievement in communication skills. Computers & Education, 58(1), 231–239. https://doi.org/10.1016/j.compedu.2011.07.011
Alsied, S. M., & Pathan, M. M. (2013). The use of computer technology in EFL classroom: Advantages and implications. International Journal of English Language & Translation Studies (IJ-ELTS), 1(1),61–71.
Amaral, L. A., & Meurers, D. (2011). On using intelligent computer-assisted language learning in real- life foreign language teaching and learning. ReCALL, 23(1), 4–24. https://doi.org/10.1017/S0958344010000261
Barrios Espinosa, E. (2013). Learning English using Tell Me More: Perspectives of university teaching staff as users of the online application. Revista de Educación, 362, 506–534. https://doi.org/10.4438/1988-592X-RE-2013-362-240
DelliCarpini, M. (2012). Building Computer Technology Skills in Tesol Teacher Education. Language Learning & Technology. 16(2) Retrieved from http://llt.msu.edu/issues/june2012/action.pdf
Elimat K. A. & AbuSeileek F. A. (2014). Automatic Speech Recognition Technology is an Effective Means for Teaching Pronunciation. The Jalt Call Journal, 10(1), 21–47. Retrieved from http://journal.jaltcall.org/articles/10_1_Elimat.pdf
Godwin-Jones, R. (2010). Emerging technologies tools and trends in self-paced language instruction. Language Learning & Technology, 11(2), 10–17. Retrieved from http://llt.msu.edu/vol11num2/emerging/
Lecercle, D. (Ed.). (2011). Giving knowledge for free: The emergence of open educational resources. Organisation for Economic Co-operation and Development. Retrieved from http://www.oecd.org/dataoecd/35/7/38654317.pdf
Li, R (2012). The Influence of Computer Applied Learning Environment on EFL or ESL Education. Theory and Practice in Language Studies, 2(1), 187–191. https://doi.org/10.4304/tpls.2.1.187-191
Melor, Y. M (2007). Malaysian ESL teachers’ use of ICT in their classrooms: expectations and realities. ReCALL 19(1), 79–95. https://doi.org/10.1017/S0958344007000614
Nielson, K. (2011). Self-Study with Language Learning Software in The Workplace: What Happens? Language Learning & Technology, 15(3), 110–129. Retrieved from http://llt.msu.edu/issues/october2011/nielson.pdf
Van Han N. & Van Rensburg, H. (2014). The Effect of Computer Assisted Language Learning (CALL) on Performance in the Test of English for International Communication (TOEIC) Listening Module. English Language Teaching, 7(2). http://dx.doi.org/10.5539/elt.v7n2p30
Yunus, M., Hasim, H., Embi, M. A. & Lubis, M. A. (2010). The Utilization of ICT in the Teaching and Learning of English: ‘Tell Me More’. Procedia. Social and Behavioural Sciences, 9, 685–691. https://doi.org/10.1016/j.sbspro.2010.12.218