 FST and the triangle inequality for biallelic markersIlana M. Arbisser and Noah A. Rosenberg Theoretical Population Biology, 2020, vol. 133, issue C, 117-129 Abstract: The population differentiation statistic FST, introduced by Sewall Wright, is often treated as a pairwise distance measure between populations. As was known to Wright, however, FST is not a true metric because allele frequencies exist for which it does not satisfy the triangle inequality. We prove that a stronger result holds: for biallelic markers whose allele frequencies differ across three populations, FSTnever satisfies the triangle inequality. We study the deviation from the triangle inequality as a function of the allele frequencies of three populations, identifying the frequency vector at which the deviation is maximal. We also examine the implications of the failure of the triangle inequality for four-point conditions for placement of groups of four populations on evolutionary trees. Next, we study the extent to which FST fails to satisfy the triangle inequality in human genomic data, finding that some loci produce deviations near the maximum. We provide results describing the consequences of the theory for various types of data analysis, including multidimensional scaling and inference of neighbor-joining trees from pairwise FST matrices. Keywords: FST; Genetic distance; Inequalities; Population structure (search for similar items in EconPapers) Downloads: (external link) Related works: Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text Persistent link: https://EconPapers.repec.org/RePEc:eee:thpobi:v:133:y:2020:i:c:p:117-129 DOI: 10.1016/j.tpb.2019.05.003 Access Statistics for this article Theoretical Population Biology is currently edited by Jeremy Van Cleve More articles in Theoretical Population Biology from Elsevier |
Is your work missing from RePEc? Here is how to contribute.
Questions or problems? Check the EconPapers FAQ or send mail to .
EconPapers is hosted by the
School of Business at Örebro University.