Suboptimal Comparison of Partitions

Jonathon J. O’Brien (), Michael T. Lawson, Devin K. Schweppe and Bahjat F. Qaqish
Additional contact information
Jonathon J. O’Brien: Harvard Medical School
Michael T. Lawson: University of North Carolina at Chapel Hill
Devin K. Schweppe: Harvard Medical School
Bahjat F. Qaqish: University of North Carolina at Chapel Hill

Journal of Classification, 2020, vol. 37, issue 2, No 11, 435-461

Abstract: Abstract The distinction between classification and clustering is often based on a priori knowledge of classification labels. However, in the purely theoretical situation where a data-generating model is known, the optimal solutions for clustering do not necessarily correspond to optimal solutions for classification. Exploring this divergence leads us to conclude that no standard measures of either internal or external validation can guarantee a correspondence with optimal clustering performance. We provide recommendations for the suboptimal evaluation of clustering performance. Such suboptimal approaches can provide valuable insight to researchers hoping to add a post hoc interpretation to their clusters. Indices based on pairwise linkage provide the clearest probabilistic interpretation, while a triplet-based index yields information on higher level structures in the data. Finally, a graphical examination of receiver operating characteristics generated from hierarchical clustering dendrograms can convey information that would be lost in any one number summary.

Keywords: Classification; Clustering; Sensitivity; Specificity; Triplet index; Hierarchical receiver operating characteristic (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s00357-019-09329-1 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jclass:v:37:y:2020:i:2:d:10.1007_s00357-019-09329-1

Ordering information: This journal article can be ordered from
http://www.springer. ... hods/journal/357/PS2

DOI: 10.1007/s00357-019-09329-1

Access Statistics for this article

Journal of Classification is currently edited by Douglas Steinley

More articles in Journal of Classification from Springer, The Classification Society
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().