A Comparative Analysis of Attainment of Program Outcomes for Courses with and without the Use of Modern Tools

An important consideration in higher education is that graduates meet or exceed the program outcomes (POs). While there exists anecdotal evidence that the use of modern tools i.e. computer modelling and simulation, improve attainment of these outcomes, there is little empirical research available. Where empirical evidence is available, the variables considered would almost certainly have a bearing on the outcomes. In this work, the attainment of the POs by undergraduate engineering students in courses with and without the use of modern tools, based on quantitative data, were compared. It was hypothesized that courses using modern tools would lead to better overall attainment of POs, compared to courses not using these tools. As a case study, the PO attainment of students in the Mechanical Engineering undergraduate program at Universiti Teknologi PETRONAS (UTP) was considered. Quantitative data obtained through UTP's outcomebased education (OBE) software was used to assess the overall attainment of the POs for all courses for a cohort of 126 Mechanical Engineering undergraduate students class of 2017. It was found that, for the case study considered, the usage of modern tools has led to slightly better attainment of some POs, with slightly poorer attainment in other POs. Specifically, attainment in POs where the cognitive or the knowledge domain is more dominant improved, as the usage of modern tools helped students to understand theoretical concepts better. Attainment in POs were the affective domain is more dominant recorded a slight decrease, and the incorporation of modern tools did not aid in the attainment of these POs. The study is at a preliminary stage and a more detailed study, involving more cohorts, is planned to establish a correlation (if any) between the use of modern tools in higher education and attainment of POs.


Introduction
In an outcome-based approach to education, each part of the educational system is based around outcomes i.e. a programme of study's goal.
It is expected that each student enrolled in the program of study will have achieved the goals of the program, the program outcomes. Program outcomes describes the skills and knowledge students should have attained upon completion of the program. These relate to the skills, knowledge, and behaviour that students acquire through the program. Courses are then developed to help students achieve the program outcomes, with individual courses having course outcomes that ultimately support the program outcomes. For Malaysian universities offering engineering programs, the Engineering Accreditation Council (EAC), a body delegated by the Board of Engineers Malaysia (BEM), stipulates the minimum standard for these programs, to ensure that the quality of the program is comparable to global practice. The council has specific 12 program outcomes which students of an engineering program are expected to attain. The following are the program outcomes for the Mechanical Engineering undergraduate program at Universiti Teknologi PETRONAS (UTP): 1. Apply knowledge of mathematics, science, engineering fundamentals and specialized engineering in solving complex engineering problems. 2. Identify research literature, investigate, analyse, evaluate and formulate solutions to complex Mechanical Engineering problems reaching substantiated conclusion using first principles of mathematics, natural sciences and engineering sciences. 3. Undertake holistic design and development of complex Mechanical Engineering problems and 4. Design systems, components or processes that meet specified needs with appropriate consideration for public health and safety, cultural, societal, and environmental considerations. 5. Perform investigation into complex Mechanical Engineering using research based knowledge including design of experiments, analysis and interpretation of data, and synthesis of information to provide valid conclusions. 6. Create, select and apply appropriate techniques, resources, and modern engineering and IT tools, including prediction and modelling, to complex engineering activities, with an understanding of the limitations. 7. Apply reasoning informed by contextual knowledge to assess societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to professional engineering practice. 8. Understand the impact of professional engineering solutions in societal and environmental contexts and demonstrate knowledge of and need for sustainable development. 9. Internalize and practice professional ethical principles with commitment to norms of engineering practices. 10. Communicate effectively on complex engineering activities with the engineering community and with society at large. 11. Function effectively in multi-disciplinary engineering as an individual and in a group with the capacity to be a leader. 12. Recognize the need to undertake life-long learning and possess the capacity to adapt to technological changes. 13. Demonstrate knowledge and understanding in project management, business acumen and entrepreneurship in multidisciplinary environments.
Of interest is the program outcome on modern tool usage, specifically on the creation, selection and application of appropriate techniques, resources, and modern engineering and IT tools, including prediction and modelling, to complex engineering activities, with an understanding of the limitations. Raju and Annigeri contends that computational work/simulation is the third vertex of a triangle in all branches of engineering, complementing observation and theory, and that computing skill is an integral part of engineering education, not an add on [1]. The use of modern tools in most engineering programs in Malaysia is typically embedded in courses. However, not all courses adopt modern tool usage. It is hypothesized that courses employing the use of modern tools would lead to better overall attainment of POs, primarily because it complements attainment of the other POs. For example, the usage of modern tools should complement attainment in program outcomes where the cognitive domain is more dominant. In this work, the attainment of POs for courses using and not using modern tools will be compared. As a case study, the attainment of the POs for a cohort of 126 Mechanical Engineering undergraduate students class of 2017 at Universiti Teknologi PETRONAS for courses with and without the use of modern tools are compared. Quantitative data, obtained through UTP's OBE software (see Figure 1), will be used to assess the student's attainment of the POs, an indirect measurement based on the mapping of course outcomes (COs) to program outcomes (see Figure 2), where the attainment of POs is based on the attainment of COs.

Literature Review
Roy et al contends that for engineering courses where understanding of technical concepts with practical understanding is highly relevant, the usage of modern tool is crucial [2]. They presented an approach to evaluate course outcomes (COs) and program outcomes (POs). The evaluation is done by taking components from continuous internal assessment, modern tool usage, assignments and even attendance. The outcomes of different courses are then mapped to program outcomes. Kanmani and Babu reviewed existing approaches to teaching modern tools to engineering students and suggests a complete rethinking on how engineering education approaches computation skills to arrive to a holistic and integrated approach [3]. They concluded that the use of technology in the context of outcome-based education improves learning outcomes. Prasad and Reddy examined the effectiveness of modern tools on the attainment of COs and POs in outcome-based education (OBE) and found that computer-based teaching led to better attainment of POs [4]. It should be noted that these studies looked at the use of technology in general, rather than computer modelling and simulation tools specifically. Sharma and Phadke discusses the effect on performance of students after improvisations in content delivery and assessment tools [5]. In their study on improving course outcomes attainment, they found that incorporating the use of modern tools such as MATLAB helped students to understand theoretical concepts better, leading to an overall development in the competency of a graduate.

Methodology
Baharom et al proposed a methodology, adopted in this work, for the indirect measurement of program outcomes (POs) of individual students based on their coursework marks, which are mapped to the course outcomes (COs) and POs [8]. Assessment tools (ATs) e.g. homework, test, final exams, etc., are used to evaluate CO and PO attainment. The relationships between the ATs, COs and POs are defined in the form of matrices. The CO to PO mapping, which links the course assessments to the PO, is first established. ATs that measure attainment of COs are designed, and assessment questions prepared based on the design of the matrix. All the matrices are treated as sets of linearly dependent systems. The mapping of the AT to the PO is then derived by multiplying the AT-CO matrix and the CO-PO matrices. The PO measurements can then be obtained by using students marks in their courses (see Figure 3).

Discussion
Although there were slight improvements in attainment of most program outcomes for courses incorporating the use of modern tools, compared to courses without the use these tools, the difference was not as significant as expected. Furthermore, attainment of PO7, PO10 and PO11 recorded a slight decrease. Specifically, the attainment of the following program outcomes for courses incorporating the use of modern tools witness a slight reduction: • Understand the impact of professional engineering solutions in societal and environmental contexts and demonstrate knowledge of and need for sustainable development. • Function effectively in multi-disciplinary engineering as an individual and in a group with the capacity to be a leader. • Recognize the need to undertake life-long learning and possess the capacity to adapt to technological changes. This could in part be because for the POs listed above, the affective domain is more dominant, while the use of modern tools is more beneficial in attainment of POs where the cognitive or the knowledge domain is more dominant. As such, the incorporation of modern tools did not aid in the attainment of these 3 POs. For the class of 2017, Mechanical Engineering undergraduate students at Universiti Teknologi PETRONAS (UTP), based on quantitative data obtained through UTP's outcomebased education (OBE) software, slightly better attainment of POs where the cognitive or the knowledge domain is more dominant was achieved for courses incorporating the use of modern tools, compared to courses without the use these tools. Nevertheless, attainment of PO7, PO10 and PO11 recorded a slight decrease, and the incorporation of modern tools did not aid in the attainment of these POs.