### Search

Enter search terms or a module, class or function name.

# 4. Performance Measures¶

Here we explore various functions that evaluate the performance of a classifier. We start by defining some notations:

• $$C$$ is the confusion matrix, where entry $$C_{ij}$$ is the number of objects in class $$i$$ but have been as class $$j$$,
• $$m_i$$ is the number of objects in class $$i$$: $$m_i = \sum_j C_{ij}$$,
• $$m$$ is the total sample size: $$m = \sum_i m_i$$,
• $$k$$ is the number of classes.

## 4.1. Naive Accuracy¶

One simple way to evaluate the overall performance of a classifier is to compute the naive accuracy rate, which is simply the total fraction of objects that have been correctly classified:

$\text{Naive Accuracy Rate} = \dfrac{\sum_i C_{ii}}{m}$

This is implemented in the function naive_accuracy().

## 4.2. Balanced Accuracy¶

However, if the dataset is imbalanced, this measure would not work well. A better approach is to use the posterior balanced accuracy. Let $$A_i$$ be the accuracy rate of class $$i$$:

$A_i = \dfrac{C_{ii}}{m_i}$

Before running a classifier, we know nothing of its performance, so we can assume the accuracy rate follows a flat prior distribution. In particular, the Beta distribution with parameters $$\alpha = \beta = 1$$ (i.e. a uniform distriubtion) seems appropriate here:

$A_i \sim Beta(1, 1) \qquad \forall i$

Given an accuracy rate $$A_i$$ for each class $$i$$, the number of correct predictions in class $$i$$ will follow a Binomial distribution with $$A_i$$ as the probability of success:

$\big( C_{ii} \mid A_i \big) \sim Bin\big(m_i, A_i\big) \qquad \forall i$

In Bayesian terms, this is our likelihood. Now we know that with respect to a Binomial likelihood, the Beta distribution is conjugate to itself. Thus the posterior distribution of $$A_i$$ will also be Beta with parameters:

$\big( A_i \mid C_{ii} \big) \sim Beta \big( 1 + C_{ii}, 1 + m_i - C_{ii} \big) \qquad \forall i$

get_beta_parameters() is a helper function that extracts the beta paremeters form a confusion matrix.

### 4.2.1. Convultion¶

One way to define the balanced accuracy $$A$$ is to take the average of the individual accuracy rates $$A_i$$:

$A = \dfrac{1}{k} \sum_i A_i$

We call $$\big( A \mid C \big)$$ the posterior balanced accuracy. One nice feature of this measure is that it’s a probability distribution (instead of a simple point estimate). This allows us to construct confidence intervals, etc. And even though there is no closed form solution for the density function of $$\big( A \mid C \big)$$, we can still compute it by performing the convolution $$k$$ times. This is implemented in convolve_betas().

### 4.2.2. Expected Balanced Accuracy¶

We have just approximated the density of the sum of $$k$$ Beta distributions. The next step is to present our results. One measure we can report is the expected value of the posterior balanced accuracy. This is implemented in balanced_accuracy_expected().

### 4.2.3. Distribution of Balanced Accuracy¶

We can also construct an empirical distribution for the posterior expected accuracy. First we need to compute the pdf of the sum of beta distributions $$\sum_i A_i$$, given a subset $$x$$ of the domain. See beta_sum_pdf().

However we’re interested in the average of the accuracy rates, $$A = \dfrac{1}{k} \sum_i A_i = \dfrac{1}{k} A_T$$. We can rewrite the pdf of $$A$$ as:

$\begin{split}F_A (a) &= \mathbb{P} (A \leq a) \\ &= \mathbb{P}\bigg( \dfrac{1}{k} \sum_i A_i \leq a \bigg) \\ &= \mathbb{P}\bigg( \sum_i A_i \leq ka \bigg) \\ &= F_{A_T}(ka)\end{split}$

Differnetiating with respect to $$a$$, we’d get:

$\begin{split}f_A(a) &= \dfrac{\partial}{\partial a} F_A(a) \\ &= \dfrac{\partial}{\partial a} F_{A_T}(ka) \\ &= k \cdot f_{A_T} (ka)\end{split}$

See beta_avg_pdf() for the implementation.

To make a violin plot of the posterior balanced accuracy, we need to run a Monte Carlo simulation, which requires us to have the inverse cdf of $$A$$. beta_sum_cdf(), beta_avg_cdf(), and beta_avg_inv_cdf() are used to approximate the integral of the pdf using the trapezium rule.

## 4.3. Recall¶

We can also compute the recall of each class. The recall of class $$i$$ is defined as:

$\text{Recall}_i = \dfrac{C_{ii}}{\sum_j C_{ij}}$

Intuitively, the recall measures a classifier’s ability to find all the positive samples (and hence minimising the number of false negatives). See recall().

## 4.4. Precision¶

Another useful measure is the precision. The precision of class $$i$$ is defined as:

$\text{Precision}_i = \dfrac{C_{ii}}{\sum_j C_{ji}}$

Intuitively, the precision measures a classifier’s ability to minimise the number of false positives. See precision().