UPM Institutional Repository

A novel performance metric for building an optimized classifier


Hossin, Mohammad and Sulaiman, Md. Nasir and Mustapha, Aida and Mustapha, Norwati (2011) A novel performance metric for building an optimized classifier. Journal of Computer Science, 7 (4). pp. 582-590. ISSN 1549-3636; ESSN: 1552-6607


Problem statement: Typically, the accuracy metric is often applied for optimizing the heuristic or stochastic classification models. However, the use of accuracy metric might lead the searching process to the sub-optimal solutions due to its less discriminating values and it is also not robust to the changes of class distribution. Approach: To solve these detrimental effects, we propose a novel performance metric which combines the beneficial properties of accuracy metric with the extended recall and precision metrics. We call this new performance metric as Optimized Accuracy with Recall-Precision (OARP). Results: In this study, we demonstrate that the OARP metric is theoretically better than the accuracy metric using four generated examples. We also demonstrate empirically that a naïve stochastic classification algorithm, which is Monte Carlo Sampling (MCS) algorithm trained with the OARP metric, is able to obtain better predictive results than the one trained with the conventional accuracy metric. Additionally, the t-test analysis also shows a clear advantage of the MCS model trained with the OARP metric over the accuracy metric alone for all binary data sets. Conclusion: The experiments have proved that the OARP metric leads stochastic classifiers such as the MCS towards a better training model, which in turn will improve the predictive results of any heuristic or stochastic classification models.

Download File

[img] PDF
Restricted to Repository staff only

Download (154kB)

Additional Metadata

Item Type: Article
Divisions: Faculty of Computer Science and Information Technology
DOI Number: https://doi.org/10.3844/jcssp.2011.582.590
Publisher: Science Publications
Keywords: Accuracy metric; Extended precision; Extended recall; Monte carlo sampling (MCS); Optimized accuracy with recall-precision (OARP); Optimized classifier; Performance metric
Depositing User: Nabilah Mustapa
Date Deposited: 08 Jun 2016 08:58
Last Modified: 08 Jun 2016 08:58
Altmetrics: http://www.altmetric.com/details.php?domain=psasir.upm.edu.my&doi=10.3844/jcssp.2011.582.590
URI: http://psasir.upm.edu.my/id/eprint/22462
Statistic Details: View Download Statistic

Actions (login required)

View Item View Item