Development of an Automated Instrument for Measuring the Usefulness of Mathematics Tutorial Courseware

Selecting mathematics courseware (PKM) by teachers is essential in order to get the most useful courseware for students. Empirical research is hindered by the lack of standard instruments with valid and reliable factor, criteria and metrics. Therefore the primary purpose of this study was to develop an instrument for evaluating the usefulness of mathematics courseware called IPKpkm and to provide psychometric evidence of validity and reliability. Through out four phases of this study, 696 participants were involved and six types of PKM were used. Each participant was required to evaluate the courseware heuristically and complete the IPKpkm form. First, based on the theoretical perspective, a hypothesize model with three factors which were usability, functionality and efficiency, five sub-factors and seven criteria were proposed. Preliminary study produced initial instrument (IPKpkm_1) consisting of 85 checklist metrics. Second, each metric was reviewed for clarity and appropriateness of metrics content by an expert panel consisting of eight lecturers from four local universities and two Assistant Directors from the Curriculum Development Center. Based on the experts’ judgments, 67 metrics were selected to form the first version of the questionnaire with a five-point-Likert-type scale (IPKpkm_2). IPKpkm_2 was administered to 35 secondary mathematics teachers Metrics analysis was conducted and three metrics were eliminated. Third, IPKpkm_3 with 64 metrics were administered to 626 mathematics teachers in five zones through out Malaysia. Analysis using SPSS 12 was conducted to find the variable that showed significant contribution to the PKM usefulness score. Findings suggested that three variables, which were teacher’s academic background, computer usage experience and evaluation error, had significant impact to PKM usefulness score. From these findings PKM usefulness mean score formula were constructed. Exploratory and confirmatory analysis revealed a three factors solution similar to the hypothesized model. Exploratory factor analysis suggested eight criteria and reduced the metrics to 57 (IPKpkm_4). Results from the initial, first, second and third confirmatory factor analysis using AMOS 5, produced 34 metrics instrument (IPKpkm_5) that provided good fit models. Goodness-of-fit analysis had supported the hypothesized models and construct validity of IPKpkm_5. Convergent and divergent validity were supported hence added strong evidence of construct validity. Fourth, IPKpkm_5 were compared to similar instrument called Instrument for Evaluating Chinese Educational Software (IECES). Both instruments were administered to 26 Third Year Mathematics Education, UKM students. Positive correlation of both scores supported the criterion validity of IPKpkm_5 and thus completed validity tests. Results from construct reliability showed there was good to very good reliability for the three factors and eight criteria. Finally, a web-based evaluation system to measure the usefulness of PKM (SPKpkm) was developed and tested. The system makes the evaluation process more efficient and can generate the mean score automatically.

Preview PDF FSKTM_2007_15.pdf Download (550kB)

Actions (login required)

View Item