4. Performance Measures

Here we explore various functions that evaluate the performance of a classifier. We start by defining some notations:

• \(C\) is the confusion matrix, where entry \(C_{ij}\) is the number of objects in class \(i\) but have been as class \(j\),
• \(m_i\) is the number of objects in class \(i\): \(m_i = \sum_j C_{ij}\),
• \(m\) is the total sample size: \(m = \sum_i m_i\),
• \(k\) is the number of classes.

4.1. Naive Accuracy

One simple way to evaluate the overall performance of a classifier is to compute the naive accuracy rate, which is simply the total fraction of objects that have been correctly classified:

\[\text{Naive Accuracy Rate} = \dfrac{\sum_i C_{ii}}{m}\]

This is implemented in the function naive_accuracy().

4.2. Balanced Accuracy

However, if the dataset is imbalanced, this measure would not work well. A better approach is to use the posterior balanced accuracy. Let \(A_i\) be the accuracy rate of class \(i\):

\[A_i = \dfrac{C_{ii}}{m_i}\]

Before running a classifier, we know nothing of its performance, so we can assume the accuracy rate follows a flat prior distribution. In particular, the Beta distribution with parameters \(\alpha = \beta = 1\) (i.e. a uniform distriubtion) seems appropriate here:

\[A_i \sim Beta(1, 1) \qquad \forall i\]

Given an accuracy rate \(A_i\) for each class \(i\), the number of correct predictions in class \(i\) will follow a Binomial distribution with \(A_i\) as the probability of success:

\[\big( C_{ii} \mid A_i \big) \sim Bin\big(m_i, A_i\big) \qquad \forall i\]

In Bayesian terms, this is our likelihood. Now we know that with respect to a Binomial likelihood, the Beta distribution is conjugate to itself. Thus the posterior distribution of \(A_i\) will also be Beta with parameters:

\[\big( A_i \mid C_{ii} \big) \sim Beta \big( 1 + C_{ii}, 1 + m_i - C_{ii} \big) \qquad \forall i\]

get_beta_parameters() is a helper function that extracts the beta paremeters form a confusion matrix.

4.2.1. Convultion

One way to define the balanced accuracy \(A\) is to take the average of the individual accuracy rates \(A_i\):

\[A = \dfrac{1}{k} \sum_i A_i\]

We call \(\big( A \mid C \big)\) the posterior balanced accuracy. One nice feature of this measure is that it’s a probability distribution (instead of a simple point estimate). This allows us to construct confidence intervals, etc. And even though there is no closed form solution for the density function of \(\big( A \mid C \big)\), we can still compute it by performing the convolution \(k\) times. This is implemented in convolve_betas().

4.2.2. Expected Balanced Accuracy

We have just approximated the density of the sum of \(k\) Beta distributions. The next step is to present our results. One measure we can report is the expected value of the posterior balanced accuracy. This is implemented in balanced_accuracy_expected().

4.2.3. Distribution of Balanced Accuracy

We can also construct an empirical distribution for the posterior expected accuracy. First we need to compute the pdf of the sum of beta distributions \(\sum_i A_i\), given a subset \(x\) of the domain. See beta_sum_pdf().

However we’re interested in the average of the accuracy rates, \(A = \dfrac{1}{k} \sum_i A_i = \dfrac{1}{k} A_T\). We can rewrite the pdf of \(A\) as:

\[\begin{split}F_A (a) &= \mathbb{P} (A \leq a) \\ &= \mathbb{P}\bigg( \dfrac{1}{k} \sum_i A_i \leq a \bigg) \\ &= \mathbb{P}\bigg( \sum_i A_i \leq ka \bigg) \\ &= F_{A_T}(ka)\end{split}\]

Differnetiating with respect to \(a\), we’d get:

\[\begin{split}f_A(a) &= \dfrac{\partial}{\partial a} F_A(a) \\ &= \dfrac{\partial}{\partial a} F_{A_T}(ka) \\ &= k \cdot f_{A_T} (ka)\end{split}\]

See beta_avg_pdf() for the implementation.

To make a violin plot of the posterior balanced accuracy, we need to run a Monte Carlo simulation, which requires us to have the inverse cdf of \(A\). beta_sum_cdf(), beta_avg_cdf(), and beta_avg_inv_cdf() are used to approximate the integral of the pdf using the trapezium rule.

4.3. Recall

We can also compute the recall of each class. The recall of class \(i\) is defined as:

\[\text{Recall}_i = \dfrac{C_{ii}}{\sum_j C_{ij}}\]

Intuitively, the recall measures a classifier’s ability to find all the positive samples (and hence minimising the number of false negatives). See recall().

4.4. Precision

Another useful measure is the precision. The precision of class \(i\) is defined as:

\[\text{Precision}_i = \dfrac{C_{ii}}{\sum_j C_{ji}}\]

Intuitively, the precision measures a classifier’s ability to minimise the number of false positives. See precision().