To determine whether the new ad hoc features could substitute for the Zernike and Haralick features, the 10 class HeLa data set was classified using only the 22 ad hoc features. The performance of the 10 BPNN trials using the non-thresholded classification rule is summarized in Table 3.7. The overall performance of the ad hoc features is not as good as that obtained when using all of the features (76% vs. 81%). Furthermore, the ad hoc features result in a greater degree of confusion between the giantin and GPP130 classes, and show a large drop in the rate of correct classification for actin, which is now more likely to be confused with the mitochondrial and tubulin patterns.
| True | Output of the Classifier | |||||||||
| Classification | DNA | ER | Giant. | GPP | LAMP | Mito. | Nucle. | Actin | TfR | Tubul. |
| DNA | 99% | 0% | 0% | 0% | 0% | 0% | 1% | 0% | 0% | 0% |
|---|---|---|---|---|---|---|---|---|---|---|
| ER | 0% | 85% | 0% | 0% | 3% | 5% | 0% | 1% | 2% | 5% |
| Giantin | 0% | 0% | 67% | 27% | 3% | 0% | 1% | 1% | 1% | 0% |
| GPP130 | 0% | 0% | 24% | 70% | 2% | 0% | 2% | 0% | 1% | 0% |
| LAMP2 | 0% | 1% | 5% | 1% | 74% | 4% | 2% | 2% | 10% | 0% |
| Mito. | 0% | 5% | 1% | 1% | 3% | 71% | 0% | 8% | 3% | 8% |
| Nucleolin | 1% | 0% | 2% | 1% | 1% | 0% | 94% | 0% | 0% | 0% |
| Actin | 0% | 1% | 0% | 0% | 1% | 14% | 0% | 68% | 3% | 13% |
| TfR | 1% | 8% | 2% | 0% | 21% | 0% | 0% | 2% | 63% | 3% |
| Tubulin | 0% | 5% | 1% | 0% | 1% | 7% | 0% | 8% | 7% | 71% |
The ad hoc features were also classified using a BPNN with thresholded outputs. These results are summarized in Table 3.8. As before, the classification rate for all samples is worse (significantly) than that obtained using all features. Although the performance of the BPNN on non-unknown samples is similar to the all features case (82% vs 86%), it is obtained only at the expense of placing many more samples into the unknown category. The worst classes in this regard are actin which went from 9% unknown with all features to 41% with just the ad hoc features, and the mitochondria pattern which went from 15% to 42% unknown. A glaring problem with the thresholded classifier is the presence of very large variances between performance on the 10 networks, 300 on the test data. Looking at the individual BPNN classifiers one at a time, it can be determined that this variance is due to differences in threshold selection. Since each of the 10 trial BPNNs is thresholded differently and since the criterion for thresholding is not maximization of the classification rate per se, the result is large differences in classification rates for the 10 networks. This problem is further exacerbated by classifiers like the ones summarized in Table 3.8 for which the probability of making an error in classification for a particular class is close to the probability of making a correct classification, as with actin and the mitochondrial pattern. In both of these classes the probability of placing a sample in the unknown category is very near the probability of recognizing it correctly. Because the fraction of samples categorized as unknown was not as high when using all of the features (see Table 3.5), there was also not as much variability between the 10 classifiers trained with that features set.
| True | Output of the Classifier | ||||||||||
| Classification | DNA | ER | Giant. | GPP | LAMP | Mito. | Nucle. | Actin | TfR | Tubul. | Unk. |
| DNA | 98% | 0% | 0% | 0% | 0% | 0% | 1% | 0% | 0% | 0% | 4% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| (99%) | |||||||||||
| ER | 0% | 68% | 0% | 0% | 0% | 2% | 0% | 0% | 1% | 3% | 25% |
| (91%) | |||||||||||
| Giantin | 0% | 0% | 50% | 17% | 1% | 0% | 0% | 0% | 1% | 0% | 30% |
| (72%) | |||||||||||
| GPP130 | 0% | 0% | 16% | 54% | 0% | 0% | 1% | 0% | 1% | 0% | 27% |
| (74%) | |||||||||||
| LAMP2 | 0% | 0% | 4% | 0% | 50% | 1% | 1% | 0% | 3% | 0% | 41% |
| (85%) | |||||||||||
| Mito. | 0% | 2% | 1% | 0% | 2% | 46% | 0% | 3% | 2% | 2% | 42% |
| (80%) | |||||||||||
| Nucleolin | 0% | 0% | 0% | 0% | 0% | 0% | 84% | 0% | 0% | 0% | 14% |
| (99%) | |||||||||||
| Actin | 0% | 0% | 0% | 0% | 0% | 5% | 0% | 45% | 2% | 8% | 41% |
| (75%) | |||||||||||
| TfR | 1% | 4% | 1% | 0% | 15% | 0% | 0% | 0% | 43% | 2% | 35% |
| (66%) | |||||||||||
| Tubulin | 0% | 4% | 0% | 0% | 0% | 2% | 0% | 3% | 2% | 53% | 36% |
| (83%) | |||||||||||
Application of the ad hoc features to a kNN classifier produced the results in Table 3.9. Again, the confusion between giantin and GPP130 is worse, and actin is more confused with both the mitochondrial pattern and tubulin. Surprisingly, however, the kNN classifier performance on the transferrin receptor is much better (difference of 20%) with the ad hoc features than with all features. This performance gain is overwhelmed, however, by the significant decreases in the classes mentioned above, and by more moderate losses in other classes.
While the ad hoc features contain enough information to recognize some patterns reasonably well (DNA, ER, nucleolin), there is still clearly some value in retaining the Zernike and Haralick features.
| True | Output of the Classifier | ||||||||||
| Classification | DNA | ER | Giant. | GPP | LAMP | Mito. | Nucle. | Actin | TfR | Tubul. | Unk. |
| DNA | 96% | 1% | 0% | 0% | 0% | 0% | 2% | 0% | 0% | 0% | 1% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| (97%) | |||||||||||
| ER | 0% | 81% | 0% | 0% | 4% | 1% | 0% | 0% | 1% | 2% | 11% |
| (91%) | |||||||||||
| Giantin | 0% | 0% | 49% | 21% | 4% | 0% | 3% | 0% | 1% | 0% | 21% |
| (62%) | |||||||||||
| GPP130 | 0% | 0% | 25% | 47% | 4% | 0% | 1% | 0% | 1% | 0% | 22% |
| (60%) | |||||||||||
| LAMP2 | 0% | 6% | 6% | 0% | 60% | 3% | 1% | 0% | 8% | 0% | 15% |
| (70%) | |||||||||||
| Mito. | 0% | 9% | 0% | 1% | 8% | 36% | 0% | 7% | 3% | 15% | 21% |
| (46%) | |||||||||||
| Nucleolin | 0% | 2% | 0% | 0% | 1% | 1% | 88% | 0% | 0% | 0% | 6% |
| (94%) | |||||||||||
| Actin | 0% | 3% | 0% | 0% | 4% | 21% | 0% | 24% | 4% | 23% | 21% |
| (30%) | |||||||||||
| TfR | 1% | 6% | 1% | 0% | 21% | 1% | 0% | 1% | 49% | 6% | 14% |
| (57%) | |||||||||||
| Tubulin | 0% | 6% | 0% | 1% | 1% | 7% | 0% | 12% | 3% | 52% | 18% |
| (64%) | |||||||||||
(mean 
95% CI) with a variance of 11 across all 10 networks. The
average performance on the training data is
with a
variance of 3.9. ( 19990527)
(mean 
(variance of 18.6) for samples that are not
classified as unknown (average of values in parentheses, below).
The average performance on the corresponding training data is
for all samples with a variance of 440 and 
(variance of 6) for those not placed in the unknown category. The
percentages of non-unknown samples that were classified correctly
are included in parentheses. ( 19990527)
(mean 
(variance
of 3.1) for those samples not classified as unknown. The
percentage of non-unknown samples that were classified correctly are
included in parentheses. ( 19990607)