Corpus‐based statistical screening for content‐bearing terms

Won Kim and W. John Wilbur

Journal of the American Society for Information Science and Technology, 2001, vol. 52, issue 3, 247-259

Abstract: An important problem in the indexing of natural language text is how to identify those words and phrases that reflect the content of the text. In general, automatic indexing has dealt with this problem by removing instances of a few hundred common words known as stop words, and treating the remaining words as though they were content bearing. This approach is acceptable for some applications such as statistical estimates of the similarity of queries and documents for the purpose of document retrieval. However, when the indexing terms are to be examined by a human as a means of accessing the literature, it greatly improves efficiency if most of the noncontent‐bearing words and phrases can be eliminated from the indexing. Here we present three statistical techniques for identifying content‐bearing phrases within a natural language database. We demonstrate the effectiveness of the methods on test data, and show how all three methods can be combined to produce a single improved method.

Date: 2001
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/1097-4571(2000)9999:99993.0.CO;2-7

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamist:v:52:y:2001:i:3:p:247-259

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1532-2890

Access Statistics for this article

More articles in Journal of the American Society for Information Science and Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().