Models of e-testing

Transdisciplinary interest and preoccupations regarding computer-assisted instruction have stirred our urge to perform a thorough study of the field and trends as well as to uphold certain approaches that have emerged in specialized scientific research. We have decided to focus our analysis primarily on the assessment process as a part of computerassisted instruction; this in fact represents the major objective of our research, whereas the results of the experiments, performed in the context of the research projects undertaken, enabled us to set forth relevant conclusions about e-testing. Our computer-assisted assessment primarily includes tests with objective or semi-objective items, which inherently evince certain disadvantages specific to these items. Beginning with the year 2000 we succeeded to create such e-tests for different examinations for basic mathematics; these e-tests were used continuously in the last 17 years in different schools, especially for children with learning-problems. We also created e-tests for vocabulary-testing in language-instruction, which we applied at the University of Sibiu by student-evaluations but also in different activities with children of different ages. Some of these products were also published on different web-sites. We present these applications here, considering that they can be models for anyone, who wants to use the advantages of e-learning.


Introduction
In 1990 Prof.dr.h.c.mult.Helmar Gunter Frank from the University of Paderborn realized the first contract of collaboration between the university of Sibiu and the "Institut fur Kyberbnetik Paderborn" in order to promote research and instruction in the field of cybernetic-instruction (Bildungskybernetik), [2].During the following years several teachers and scientists from the university of Paderborn (Prof.on.Dr. Brigitte Meder, Prof. on.Dr. Gunter Lobin, Prof. on.Dr. Vera Barbadovska-Frank) but also from other universities, members of the "Akademio Internacia de la Sciencoj San Marino" (Prof.Dr. Fabricio Pennachietti from Torino, Prof. Dr. Carlo Minnaja from Padua, Prof. Dr. Hans-Dietrich Quednau from Munchen-Freising, Prof. on.Dr. Reinhard Fossmeier) visited the University of Sibiu in order to teach students or to present their research results to the stuff at different international conferences organized here.
These contacts permitted us to progress in the field of objective instruction realized on computers by using specific software.But only after ten years of theoretical research in the year 2000 we succeeded to finalize the first practical application.

Basic mathematical-knowledge e-tests
Our first applications of e-tests were destined to the pupils from the "Centrul Scolar pentru Educatie Incluziva nr. 2 din Sibiu" for pupils with hearing-problems.This school received in 1996 the version Authorware3 from their supporting association "Freunde des Taubstummenheims Hermannstadt" from Siegen/Germany, in order to create applications for children with hearing difficulties, [1].A collaboration-project of the "Centrul Scolar pentru Educatie Incluziva nr. 2 din Sibiu" with our research team from the University of Sibiu was started.So, we began to use the offered software in order to create useful applications for the children of the school.
The school-programs for the children with special educational needs (SEN) are centered on targets and competences specific for these institutions, with targets and references established for each branch of study.Important for every child in the primary school are the basic operations of addition and multiplication.Here the rapid oral exercises at the beginning of every instructional hour are important for fixing the knowledge in basic calculations.For pupils with hearing-difficulties such exercises can't be performed.So we produced for them suitable e-tests on the computer, in order to be used for rapid exercising of the basic mathematical operations: addition and multiplication.These exercises were also used in different user schools for children with SEN.We want to describe here one of our experiments.
The instruction and testing units, designed and used throughout the e-test-application for basic mathematics performed at the "Lucian Blaga" University of Sibiu (LBUS), have been developed in the authoring language Authorware, [3].
Programming of the e-test Addition till 10: The 45 items with exercises for adding the numbers from 1 to 9, without passing over the value 10, were distributed in 5 test-modules Ai (i = 1,5) with the same level of difficulty with 9 operations each.[4] When the children have finished the exercise all answers remain on the monitor, so that they can see the wrong answers and so the testing can help them to improve their knowledge and next time have a better result.In the figure above one can observe that the software-product fulfills the following operations: appreciates the results of the assessment delivering a mark for the number of correct answers and measures the pedagogical time (time for thinking), [5] necessary for finding the respond.We mention that the authoring language "Authorware", used for the realization of the didactical projects, has an option "time in section" (figure 2) which measures only the time sequence while the user is in the interrogation-cycle -the timesequence necessary for creating the response -but not the time necessary for going from one exercise to the following; so the system permits to pause between the exercises.

Fig. 2. Creation of the mark (nota) and the measuring of the pedagogical time
We want to mention here that, relating to the pedagogical time, the software permits a limitation of the time until one can respond, but we didn't use this option, considering that this would have negative psychological influence on the children.We wanted to be sure that first of all they know the correct answer and the rapidity of answering will be exercised only in the moment, when the knowledge-content is reached.For each exercise we indicated a pedagogical-time that should be reached (in this case for solving 9 additions we indicated 30 seconds) so that the learner has an indication how good his time was.We did not include the time-factor in the mark, because the rapidity of the answer might also depend on competences in typing on the computer, a factor we don't want to measure here.
This arithmetical e-test with five assessment-models Ai was used in the school-year 2009-2010, for the final evaluation in mathematics for four groups of subjects: The results obtained after the application of the e-test at 158 subjects (41 -G1, 66 -G2, 24 -G3 and 27 -G4) were registered in tables and the following graphics present a sample of the analyze performed in this experiment.
At the group G1, consisting of children with hearing problems, one can observe that the marks are not increasing from one exercise to the other -not even when the same exercise is repeated.But the pedagogical time necessary for solving the exercises was decreasing almost at every pupil.This decreasing of the pedagogical time made us conclude that, during performing the e-test, the pupils were adapted to the new difficulties caused by the situation of being tested by a computer-test, [6].
Although the "Proiectul aritmetic: Adunarea pana la 10" (Arithmetic-project: Addition until 10) composed from the modules Ai was based on simple items, the capacity of attention of the subjects decreased rapidly after solving only 9 of the 45 possible operations, so that we could conclude, that the intellectual effort made by these pupils with a learning-deficit, could not be more than to solve two modules in the same day -with a pause between these modules.This shows what efforts must be made by the didactical stuff in order to obtain at least some favorite results by teaching children with special educational needs, even if one takes only the basic mathematical elements.
Analogue we also created ten e-test-modules M1-M10 -with 10 items each -for testing the knowledge of the multiplication-table, to complete our Proiectul aritmetic (Arithmetical project).The application realized with these e-tests for testing the basic knowledge of the mathematical operations -addition or multiplication -, enabled us to get assessment results with a great level of objectivity, fidelity and validity.The testing of basic mathematical knowledge is important for pupils at the end of a school-year, when one has to decide if the assessed pupil can pass the finished form, has to repeat the form or has no chance to reach the purpose and must be declared to be a child with SEN, which must be instructed with specific methods and the instruction-plan for this pupil must be reduced.
The experiences gained with these applications encouraged us to create also larger tests.So we developed e-tests for measuring the competences in using the articles "der, die, das" in the German language.This experiment was described in several publications, [4,7].
The most complex e-test we created is made of 1008 items.We will describe it summery in the following:

Vocabulary-e-test
First we created an instruction-e-test by using a large Instruction Matrix consisting of 42 instruction-vectors.Each instruction module is laid on a separate line of the instruction matrix in order to obtain the vector structures: MI1 = [ a1 1; a1 2; a1 3; ...; a1 q ] Where aij represent instruction elements MI2 = [ a2 1; a2 2; a2 3; ...; a2 q ] (the index i indicates the instruction module, MI3 = [ a3 1; a3 2; a3 3; ...; a3 q ] and j shows the element ranking ... in the instruction module; MIm = [ am 1; am 2; am 3; ...; am q ] i,j,m,q are non-zero natural numbers; i ≤ m, j ≤ q) The Assessment Item Bank is made up of the totality of the elements in the instruction matrix A, while an assessment cycle includes at least one element from each MIi instruction module (i = 1,m ) randomly selected by the computer-assisted assessment program.Therefore the structure of an Assessment Module ME is: The elements ai k(i)MIi» i{1,2, ..., m} are randomly selected from the corresponding modules in keeping with This way of choosing the assessment-items represents a hindrance in running through all the elements of a comprehensive item bank during the final assessment of a candidate and requires that the assessment modules applied in any examination process should differ from the instruction modules.
The random selection of assessment elements from the multitude of the assessment item bank performed by a software product is meant to assure the objectivity of the assessment process.Furthermore, the assessment validity is substantiated by the fact that no instruction module is left untested.
The informational content selected for this project focused on the identification of Romanian equivalents to ILo (Internacia Lingvo de doktoro Esperanto) vocabulary elements, from a fundamental vocabulary of 1008 words (42 groups each with 24 words).The design of the assessment program prevented the random selection of elements from the entire item bank of 1008 elements, so that the assessment occurs cyclically, and a word is selected from each instruction module in order to accomplish an assessment module and thus assure a standardized assessment of all instruction modules.
The 42 instruction modules are placed sequentially in the program diagram performed in Auhorware (as shown in the two icons in the left-hand side of figure 4 given below), whereas the right-hand icon shows the representation of aij items in the loop attached to the decision icon referring to the MI1 instruction module: In the authoring language Authorware the assessment items are randomly selected from the content of the instruction module by means of the "decision icon" whose purpose -by means of the option "Randomly to Unused Path" -is to assure, for each assessment cycle the selection of a different item from the instruction module, thus avoiding item repetition in an assessment module since it might trigger redundant elements in the assessment program.The web-publishing of computer-assisted instruction and assessment programs, was made available on the website of the International Academy of Sciences San Marino at www.aissanmarino.org/kursoj; the image captured in figure 5 is available on this site.The instruction modules can be randomly accessed in order to select an assessment element provided that the instruction modules are not placed sequentially; instead they may be accessed by random selection for developing the ME assessment module.
The analysis of results obtained by these e-tests is meant to testify the efficiency of computer-assisted assessment programs, thus accomplishing computerized assessment models.The term e-learning, or electronic learning, is very general and defines the teaching and learning with or from a technical device, which presents the contents of the teaching (also called teaching programs).The authoring systems, such as Authorware and Toolbook, use multi-medial systems of finished programs, which can be used to present the most diverse subject contents in a multi-medial manner and to control the learning progress through interaction with the system.They are teaching systems, which take over the tasks of the teacher in the context of their technical possibilities.The learner interacts or communicates with the teaching system independently according to previously implemented rules and contents.Regardless of the course offer, these systems are also good for testing teaching content as described in this article.
We already mentioned that the software used in the presented application was created in the authoring language Authorware.At the University of Paderborn the authoring language used was Toolbook.We always hoped that there will appear a software which will be used for didactical purposes, that will be accepted generally in schools -such as Word is used for most people as a text editor.But this didn't happen.Most schools, which use e-learningmethods, use ready delivered instructional products, produced by specialized firms.The teachers from these schools do not produce their e-test themselves.So we considered that there is no need to instruct to future teachers a software-program they will never use and we produced our own application (a program created in Java) which we offered -free on charge -to our students.This software-product we called QUICK E-TEST.
We will not present here the information-content of the QUICK E-TEST created by a computer-engineer, because we are not interested in technical details of software-programs.We will analyze the way the common users can use the application QUICK E-TEST to create their Web-Based-Training-application.This application has four "bricks" [9], as it is shown by the symbols from the monitor-representation of the folder presented in the following figure 6: Fig. 6.The QUICK E-TEST files The content of the files "wellcome", "ASK" and "REPLY" can be modified by the student using a simple text-editor (for example Notepad).In the file "wellcome" one will publish the indication for the user, in "ASK" are introduced the sentences with a missing word and in "REPLY" the correct answer-possibilities.The software-program allows us to indicate more than one -so synonymous -correct answer; we underline here, that even if there are more than one possible correct answer, these answers must be precisely.In an e-test we can't use items with an uncertain answer.The "objectiveness" of our QUICK E-TEST consists in the establishment of the set of possible "correct" answers.We can't accept sentences with an infinite set of possible correct answers, like: "On the table there is a ..." in order to be programmed, because in the file "REPLY" the programmer has to indicate all possible correct answers.published in the web and a lot of people -including the whole group of producers -can use them often, then it can also financially worth to create and publish such applications.The scientific usefulness of such applications is always guaranteed.That's why we described these e-tests, considering that they can be models for those, who want to apply methods of e-testing.

Fig. 3 .
Fig.3.Marks obtained with the modules Ai at the group G1 (7 subjects registered as Si)

Fig. 4 .
Fig.4.Scheme of the program for the vocabulary-e-test in Authorware