 The textcat Package for n-Gram Based Text Categorization in RKurt Hornik, Patrick Mair, Johannes Rauch, Wilhelm Geiger, Christian Buchta and Ingo Feinerer Journal of Statistical Software, 2013, vol. 052, issue i06 Abstract: Identifying the language used will typically be the first step in most natural language processing tasks. Among the wide variety of language identification methods discussed in the literature, the ones employing the Cavnar and Trenkle (1994) approach to text categorization based on character n-gram frequencies have been particularly successful. This paper presents the R extension package textcat for n-gram based text categorization which implements both the Cavnar and Trenkle approach as well as a reduced n-gram approach designed to remove redundancies of the original approach. A multi-lingual corpus obtained from the Wikipedia pages available on a selection of topics is used to illustrate the functionality of the package and the performance of the provided language identification methods. Date: 2013-02-07 Downloads: (external link) Related works: Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text Persistent link: https://EconPapers.repec.org/RePEc:jss:jstsof:v:052:i06 Access Statistics for this article Journal of Statistical Software is currently edited by Bettina Grün, Edzer Pebesma and Achim Zeileis More articles in Journal of Statistical Software from Foundation for Open Access Statistics |
Is your work missing from RePEc? Here is how to contribute.
Questions or problems? Check the EconPapers FAQ or send mail to .
EconPapers is hosted by the
School of Business at Örebro University.