BrownBag, 4th of Feb. 2009: Is your classifier statistically significantly better than the others

Dragi Kocev

In machine learning, typically, when proposing new algorithm/classifier or some modification of an existing algorithm, we are making a hypothesis that it improves the performance over the existing algorithms. However, it is very important to check whether the improvement is statistically significant or not.
We will address the issue of statistical comparison of the performance of classifiers. Through examples, we will present some statistical procedures for comparison of the performance of two classifiers over multiple datasets and the performance of multiple classifiers over multiple datasets.