cat.lump.aq.textextraction.wikipedia.prepro

Class TermExtractor

java.lang.Object

cat.lump.aq.textextraction.wikipedia.prepro.TermExtractor

public class TermExtractor
extends java.lang.Object

A class that extracts terms according to different definitions. Only one definition is given now. The distinction among languages should be done here.

Since:

May 28, 2015

Author:

cristina

Constructor Summary

Constructors

Constructor and Description
TermExtractor(java.util.Locale language) Creates a new TermExtractor

Method Summary

Methods

Modifier and Type	Method and Description
java.util.List<java.lang.String>	getTerms(java.lang.String text, int minimumSize) Transforms the content of a String into a set of terms.
static void	main(java.lang.String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TermExtractor

public TermExtractor(java.util.Locale language)

Creates a new TermExtractor

Parameters:

language -

Method Detail

getTerms

public java.util.List<java.lang.String> getTerms(java.lang.String text,
                                        int minimumSize)

Transforms the content of a String into a set of terms. A term is obtained as follows:

1. The content is converted to lower case
2. The tokens which are composed only by punctuation and symbols are removed.
3. The tokens which appears in the stopwords list are removed
4. The remaining tokens are stemmed with a Snowball/Lucene Stemmer
5. Diacritics are removed (Now it is needed for French and Romanian only \\TODO)
6. It selects only the tokens composed by letters and with a minimum size (usually 4 characters, but always 3 for Arabic).

Parameters:

text - The String from which terms are extracted.

minimumSize - integer with the minimum size to keep a token as a term

Returns:

The list of tokens resulting of the treatment explained above.

main

public static void main(java.lang.String[] args)

Parameters:

args -