Using Experimental Design in Assistive Learning Technology Trials : An Experience-Based Review

Assistive learning technologies are products that are aimed at assisting people with disabilities in improving their learning with minimum intervention of caregivers. Their efficacy has been assessed by means of experimental research trials. An experiment is characterized by the treatments and experimental units to be used, the way treatments are assigned to units, and the responses that are measured. The treatments and experimental units require representative participants or sample. However, because of the limits of numbers of participants or sample sizes, such kinds of studies have been delicate but challenging experiences. This paper attempts to review such valuable research experiences.


Introduction
In general assistive technologies (AT) are defined as items, pieces of equipments, or product systems, whether acquired commercially, modified, or customized, that is used to increase, maintain, or improve functional capabilities of individuals with disabilities [2].Assistive learning technologies include computer-based products that are aimed at assisting people with disabilities in improving their learning with minimum intervention of caregivers.Further study from the same resource found that about 80 percent of published studies on AT conducted to investigate the efficacy of the use of AT in impacting participants' skills and nearly half of them employed experimental design [1].
Using experimental design for the trials of AT efficacy is unique, delicate and challenging experiences [20] [21] [22].Participant of the trials are people or children with disabilities, such as with Autism Spectrum Disorder (ASD) or partially strokes.There have been several problems related with gaining access, parent or caregiver concern or permission, legal issues, and reporting the findings [19] [23].In the following parts of the paper, the reviews on experiences are attempted based on two selected trials, which are a mathematics tutor, called as MathTutor, and a serious game diagnosis, called as Vi-Per Games, for children with ASD.

Design and Setting
This research aimed mainly to examine the efficacy of a computer assisted instruction or tutor in helping students a Gede Pramudya Ananta with autism to learn elementary concepts of addition.For best result from the trial, pretest and post-test design of experiment was planned for the population of Malaysia resident children with ASD.The design can be depicted by the Table 1 as follows.
Access to the centers of caregivers or schools was sought from related authorization bodies.However, the access could not be gained for some months until the population changed to the children with those disabilities in Melaka.Besides, while at the process of selecting participants, preinterviews were conducted with the teachers and basic information about the students had been retrieved.Later, the participants were observed separately to verify if they did portray the required skills for the experiment.At the end of the observation period, 40 (forty) students with ASD who met these skills were selected as participants for the experiment, in which they were randomly divided into two groups; the intervention group that employed the MathTutor, and the control group that utilized the teacher-assisted learning approach.There were from several primary schools in Ayer Keroh, Melaka.
The participants were required to have certain prerequisite skills such as knowing how to use computers and mouse, being able to follow instructions orally or written, being able to focus on the activity for at least 10 minutes, and being capable to recognize numbers between 1 and 10.The research was held in an individualized education MATEC Web of Conferences 150, 05039 (2018) https://doi.org/10.1051/matecconf/201815005039MUCET 2017 classroom.During the experimental session, there was no other person in the classroom other than the researcher and the student.A laptop was used to present the intervention modules which were developed and presented in the tutor during the experimental session.Minimum prompt and assistance were given to the participants if they failed to follow the written or verbal instructions.

Results and Discussion
The following reports the performance of participants throughout the experiment.During the experiment, pre-test and post-tests were carried out to measure the effectiveness of the tutor as a learning tool.Before initiating the experiment, all the participants were required to sit for a pre-test in order to determine their initial skill level before treatment.Lastly, at the end of the last session, the participants were required to sit for post-test, where the questions were similar to those found in the pre-test.
As for the experiment, the control group consisted of randomly selected 20 students with ASD and they utilised the normal tutoring system throughout the experiment.The Wilcoxon Signed-Rank Test was used in order to determine whether there is a significant difference between the pre-test and post-test scores of the control group.As a result, the Wilcoxon Signed-Rank Test successfully determined the significant variance between the pre-test and post-test scores for before and after the experiment carried out within the control group (Z = -3.91,p < 0.05).This result indicated that a change did take place in the score for the control group.In precise, a difference was present between the pre-test and post-test scores in measuring the skill level of the control group.
On top of that, the use of box plots and scatter plots exemplify the distribution of pre-test and post-test scores of the participants from the control group.The plot indicates that 90% of the pre-test scores were distributed below 50% score.Hence, 90% of the participants scored less than or equivalent to 50% score.The right box, on the other hand, which shows the post-test scores, points out that about 35% of the participants attained above 60% score.This suggests that about 60% of the participants had high post-test scores (between 60% and 80%), thus displaying improvement in the control group for before and after the experiment.
In the experiment, the intervention group consisted of randomly selected 20 student diagnosed with autism, in which they experienced the tutor that ascertains the suitability of learning material based on the level of users' skill throughout the experiment.As such, the Wilcoxon Signed-Rank test was used in order to determine whether there is a significant difference between the pre-test and post-test scores of the experiment group.As a result, the test identified a significant difference between the pre-test and post-test scores derived from the experiment group (Z= -3.92, p<0.05).This result hints that a change did occur in the experiment group.In other words, a significant difference was detected between the pre-test and posttest scores in measuring the skill level for the intervention group.
The experimental case study was carried out to determine the significant difference between the intervention and control groups.As such, the post-test scores obtained from both groups were used to determine the results of the experiment.The Mann-Whitney U test was employed to compare the level of skill among participants in the experiment group, precisely looking into their basic addition concept in mathematics after the experiment, besides ensuring the presence of a significant variance between the post-test scores obtained from the intervention and control groups.Besides, the findings of the Mann-Whitney U test with the aim of determining the significant difference between post-test scores of the intervention and control groups.
It also portrays that the post-test rank average of the intervention group is 29.92, while that for the control group is 11.10.Hence, a vast difference of 18.82 points is recorded between the rank averages of the two groups.Based on the analysis performed, the value of (U = 11.50, p<0.05) obtained indicates a rather significant difference between the intervention and control groups.In short, the method exposed to the intervention group proved to be more effective, in comparison to the technique applied in the control group.
During the experiment, an informal observation was carried out on both control and experiment groups.The observation was performed when the participants attempted using the tutor.Besides, notes were recorded from the observations pertaining to their behaviour, progress, and if they had required guidance or assistance.
During the first session, all the participants exhibited some interest in using the computer as a learning tool to learn the addition skill although some seemed playful and needed help to stay focused at the initial stage of the session.Besides, some of the participants required guidance and assistance when accomplishing reinforcement exercises, especially those questions that involved numbers larger than five.As for the second session, some participants kept repeating the same mistakes as they did in the previous session.Moreover, they seemed distracted while using the tutor system, thus seeking guidance throughout the session.Meanwhile, in the third session, most of the participants required both guidance and assistance when attempting the sums.Furthermore, similar in the previous session, they were distracted by the surrounding while using the tutor system, hence they needed both guidance and assistance throughout the session.Next, in the fourth session, most of the participants stopped showing interest or were unmotivated during the lesson.Besides, the participants required heavy guidance and assistance at their sums.Additionally, a majority of the participants also seemed to rush in answering the reinforcement exercise.Later, during the fifth session, those distracted participants were unable to focus on the lesson, while more failed to show interest or motivation during the lesson.During this particular session, some participants required heavy guidance throughout the session.As for the sixth and the last sessions, almost all the participants were able to do the exercise with minimal guidance, especially during the reinforcement exercise.
On top of that, during the experiment, some participants did display improvement in behaviour, where they maintained good behaviour throughout the remaining experimental session, in comparison to the first three sessions.Furthermore, some participants showed that they did enjoy the experiment session.Nevertheless, the tutor system failed to attract the attention of these students to keep them engaged throughout the experiment.
In the experiment group session, during the first session, all the participants exhibited interest in using the computer as a learning tool to learn the addition skill, although they seemed playful and needed help to stay focused at the initial stage of the session.In fact, some participants showed that they knew how to calculate using their fingers even though the calculations made by some of them were not in line with verbal counting.However, some had required guidance and assistance during the reinforcement exercise, especially with questions involving numbers larger than five.During the second session, some participants portrayed that they did remember the lesson learned during the last session.Moreover, some counted using fingers, besides using the pictures provided with the questions.Nonetheless, some kept on repeating the same mistakes they did in the last session.In addition, a majority of the participants seemed disturbed by the sound of the answer button.Hence, in this session, a number of participants demanded guidance and assistance, especially when attempting reinforcement exercises.Next, in the third session, most of the participants proved to remember the lesson learned during the last session.Besides, most of the participants had managed to focus on both learning and assessment during the session.Nonetheless, some participants had needed guidance, while some others required both guidance and assistance when doing the calculations.In fact, most of the participants also seemed to enjoy the session and requested for the lesson to be repeated.
Meanwhile, in the fourth session, a majority of the participants displayed interest and motivation during the lesson, although some were distracted by the sound of the answer button.During this session, most of the participants only required guidance when attempting the calculations.In fact, they also seemed to enjoy the session and requested the lesson to be repeated.As for the fifth session, similar to the previous session, most participants exhibited interest and motivation during the lesson, although some were distracted by the sound of the answer button.Hence, those distracted had needed guidance and assistance while at reinforcement exercises.As for the sixth and the last sessions, all participant seemed eager to start the lesson.Furthermore, almost all participants had been able to accomplish the exercise efficiently, while only some required minimal guidance instructions during reinforcement exercise.The participants also seemed to enjoy the session and requested for a repeat.On top of that, during the experiment, almost all the participants displayed improvement in behaviour for they maintained good behaviour throughout the remaining experimental session, in comparison to the first two sessions.The participants also showed that they enjoyed the experiment session as they requested for the lesson to be repeated and regularly requested to use the computer.Thus, the method employed in the tutor, such as images, animations, and sounds, seemed to attract these students with autism, besides retaining their focus when learning took place.As a result, the tutor had been proven to enhance both engagement and interest among students with autism during the learning sessions.Moreover, all the students with autism actively participated during the lessons, where they counted together with the tutor.Hence, the participants were interested to pursue the learning sessions and asked to repeat the lesson that they had covered.Besides, the tutorial and the examples shown to them attracted their attention, thus aided them in memorizing the lesson learned.

Design and Setting
Early identification and diagnosis to children with ASD is required to introduce them with early intervention program in order to overcome, or to minimize at least, the problems related with ASD.The objective of this research is to develop serious games, called as Vi-Per Games, as a tool for caregivers or special education teachers to diagnose visual perception problems in students with ASD.The Chalfont diagnosis theory was employed for development of the tool [3].Then a trial was designed for the population of students or children with ASD in Melaka in order to achieve external validity of the design of experimental research.As above, gaining access was initiated from centers of rehabilitation or special classes of children or students with ASD.Even though, within the extra long time limit, the access was granted, but there are only a small number of students or children was able to participate in the study because small number of returned concern forms from the parents or caregivers.At last, only 10 (ten) students are willing to be the participants of this study.
This study utilized the mixed method, using both the quantitative and the qualitative methods.The usage of quantitative method is to validate the accuracy of the serious games for diagnosing and generating comprehensive and systematic diagnosis reports on visual perception problems in students with autism.On the other hand, the qualitative approach applied in this research is to validate the model and the perception of teachers towards the use of the serious games for diagnosing visual perception problems in students with ASD.
The aspects of research design such as the instruments used in this study are also addressed in this chapter.AD-DIE model is used as the research design to develop the visual perception diagnostic tool.This five-phase systematic model, namely Analysis, Design, Development, Implementation and Evaluation, is used to guide through the process of creating multimedia products for a variety of settings.Each phase represents a series of tasks that help to ensure development efforts stay on track, on target and on time.Completing each phase adequately increases the chance that the information presented ultimately remains relevant to the needs of the research.Selection of the participants can be considered as a purposive sampling.According to Chaudary [5], Bogdan and Biklen [6] since the participants were identified and selected in accordance with some purposive principles such as accessibility, hence the selection of the participants can be regarded as purposive sampling instead of random sampling.In addition, specific information obtained from the participants were relevant to the purpose of the study.
Data collection was conducted with the participation of special education school teachers and students with ASD of SMK Bandar Baru Uda and SMK Tun Syed Nasir Ismail, both are secondary schools in Johor Bahru.The details related to the collection methods used in this study is also reviewed and described.Data about each participant level of visual perception are ordinal type.They were coded or assigned an ordinal scale measurement.The Wilcoxon Signed-Rank test, a non-parametric test was employed on every sample in conforming no significant difference between participant conventional and Vi-Per Games test results.Meanwhile, data that describe the participant perception, statistic on using the prototype, comment, and general related responses are ordinal and/or nominal type.For the purpose of data analysis, descriptive statistics such as percentage, median, percentiles were employed.In general, this study involves the development of the Vi-Per Games system for diagnosing visual perception for autistic students and how these games are utilized by the teachers in the special education class in school.A diagnostic test was taken by the students in the initial stage.In addition, this method selected for this study and describes the research approach, research design, sampling strategy, data collection, as well as the data analysis of the study.

Research Approach
Dominant concepts of research domain demands details understanding towards research approach.According to Brewer et al. [15], quantitative research is referred to hypothesis testing research in which studies initiate with statements of theory from which the research hypotheses are derived.In addition, a qualitative research approach based on observations, document studies and interviews taken from real world is argued in order to meet the research objectives.However, a few alternative research approaches will also be discussed.From literature and discussions on various traditions and approaches to "good" research [7] [8] [9] [10] [11], four potential research strategies have been considered for the research project.There are logical theoretical research, quantitative experimental research, qualitative observational research and participatory action research.
Along with research strategies, two types of methods were applied in this study, which are quantitative method with experiment and qualitative method with use of questionnaires.The data was collected by employing the survey method and the opinions of the sample population were gathered using a questionnaire.Malhotra [12] and Sekaran [13] stated that face-to-face sessions can aid in collecting data for a survey.Besides that, data can be collected via telephone interviews, observations, e-mails, the Internet or personally administered questionnaires.Then, the Likert scale, a popular method of collecting data for surveys was used.This method is employed to measure attitudes.It requires respondents to provide answers to a number of statements.The answers range from strongly agree to strongly disagree as recommended by Saunders et al. [14].Therefore, personally administered techniques and the Internet were considered as the most applicable method to collect the data.
Quantitative and qualitative approaches were chosen to be applied in this study.The quantitative research applied in this research is to validate the accuracy of the serious games for diagnosing visual perception problems in students with autism and generate comprehensive and systematic diagnosis reports on visual perception problems.Meanwhile, the qualitative research applied in this research is to validate the model and the perception of teachers towards the use of the serious games for diagnosing visual perception problems in students with ASD.
In order to describe how the prototype was used, separatesample pre-test and post-test complemented by a case study design was employed.The participants were divided into size of 3, 3 and 4 separate and independent samples.One group pre-test and post-test pre-experimental design is described by Tuckman [16] and Kerlinger [17] as shown in Table 2.An observation or measurement is designated as O and a treatment or the presence of the treatment is designated as X.For ease of identification and referral each O carries an arbitrary subscript.In this study, to provide some information about the sample of participants, observation or measurement O 1 was carried out by applying pre-test or survey.Then, a treatment X was experimented on the subjects.Finally, to access the effect of the treatment X, another observation or measurement O 2 , was made on the members of the group.To overcome history bias which was the main inherent deficiency in this design, Tuckman [16] the one group pre-test and post-test was applied three times.The term history bias refers to the preconception triggered by events occurring in the environment concurrently when the experimental variable is being tested.The modified design adapted by Pramudya [18], which is called separate-sample pre-test and post-test design is illustrated in Table 3.
For this study, O 1 , O 3 , and O 5 consecutively represent the pre-test procedure applied to obtain data on participant levels of visual perception diagnosis by using the conventional method, which is Visual Perception Diagnostic Test Instrument, Ministry of Education Malaysia on three different separate samples.X stands for the instructional treatment, which is use for the prototype Vi-Per Games, and O 2 , O 4 and O 6 designate the post-test procedure.Questionnaires and data of autistic student statistics at the database were used to gather information about participant level of visual perception, teacher perception of the prototype, and other general related comments from the three samples after the treatment.

Results and Discussion
The results from the three separate samples indicate that there were no significant differences between the pre-test and the post-test scores of participants in this study.It shows that, in each of the three samples, the participant's pre and post-test scores were the same.As the collection of score marks was performed in three different independent samples on different timelines, the history bias in this study was avoided.In other words, it is unlikely that some other event would have occurred simultaneously to affect the use of the serious game environment in the three samples.This supports the conclusion that the test scores resulting from the use of the serious games are accurate.This suggests that the serious game approach by using Vi-Per Games as an alternative method may benefit teachers and autistic students in the diagnosis of the level of visual perception in autistic students.
Since Vi-Per Games might improve the diagnosis process, encourage students, and enhance their interest and motivation, these games can be used to assist autistic students to undergo diagnostic testing.Apparently, these findings are coherent with a previous study, which also indicate autistic student compatibility with games.In addition, the findings also highlight that games could help to analyze visual perception performance.

Conclusion
This paper is intended to review experiences on using experimental design in two different studies.The main purposes of the studies were to investigate the efficacy of two assistive learning technology named MathTutor and Vi-Per Games in helping students with ASD to learn elementary mathematics concepts of addition and assisting caregivers to diagnose children with ASD.Representative samples were designed in order to meet the external validity of the design or in order to generalize the findings.However, because of the limited concern from the parents or caregivers of the children as well as minimum access gained, only small samples or participants are employed.This forced the researchers to shift the paradigm, scope, and data analysis of the studies.

Table 1 .
Design of Experiment 1

Table 2 .
Design of Experiment 2