Confidence Intervals

When classifying images using test data, true classifier performance (P_c) is estimated as

$$\widehat{P_c}=\frac{\textrm{Number of correct classifications}} {\textrm{Number of classified samples}}$$ (1.4)

Each classified sample is therefore a Bernoulli trial and the estimate of classifier performance ( $\widehat{P_c}$) is distributed as a binomial random variable with a mean of P_c and a variance of $\frac{P_c(1-P_c)}{N}$. As the number of samples is increased, the binomial distribution can be approximated by a Gaussian distribution with the same mean and covariance. It is then possible to assign a confidence interval to the performance estimate, $\widehat{P_c}$[18, p. 250]

$$P_c = \widehat{P_c}\pm z_u\sqrt{\frac{\widehat{P_c}(1-\widehat{P_c})}{N}}$$ (1.5)

where z_u satisfies

$$u=\frac{1}{\sqrt{2\pi}}\int_{-z_u}^{z_u}e^{-\frac{z^2}{2}}dz$$ (1.6)

for a particular value of u (u=0.95 and z_u=1.96 below, unless otherwise stated).