cat.lump.ie.textprocessing

Class TextPreprocessor

java.lang.Object

cat.lump.ie.textprocessing.TextPreprocessor

public class TextPreprocessor
extends java.lang.Object

This class represents an "interface" to the different text processing tools available in this package.

The currently available operations are:

• toLowerCase()
• removePunctuation()
• removeDiacritics()
• removeStopwords()
• stem()
• onlyAlphabetic(4)

  Note that a copy of the originally set string is stored and the changes are carried out on a copy

Author:

albarron

Constructor Summary

Constructors

Constructor and Description
TextPreprocessor(java.util.Locale lan)

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getOriginalString()
java.lang.String	getString()
java.util.List<java.lang.String>	getTokens()
java.lang.String	normalizeAndDeAposText(java.lang.String text) Normalize the text by shrinking white spaces in one as well as substituting quotations, dashes and dots.
static java.lang.String	normalizeText(java.lang.String text) Normalize the text by shrinking white spaces in one as well as substituting quotations, dashes and dots.
void	removeDiacritics() Removes the diacritics from the string
void	removeEngStopwords() Eliminates all the English stopwords from the string
void	removeNonAlphabetic(int minimumSize) Remove any token which is not in [:alpha:] character class.
void	removeNonAlphaNumeric(int minimumSize) Remove any token which is not the in [:alnum:] character class.
void	removePunctuation() Removes the punctuation marks from the string
void	removeStopwords() Eliminates all the stopwords from the string
void	setString(java.lang.String str) Stores a copy of the original string and generates a tokenized copy.
void	setStringTokens(java.lang.String str) Stores a copy of the original string and generates a tokenized copy.
void	stem()
void	stemLucene()
void	toLowerCase() Converts the string to lowercase

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextPreprocessor

public TextPreprocessor(java.util.Locale lan)

Method Detail

toLowerCase

public void toLowerCase()

Converts the string to lowercase

removeDiacritics

public void removeDiacritics()

Removes the diacritics from the string

removePunctuation

public void removePunctuation()

Removes the punctuation marks from the string

removeStopwords

public void removeStopwords()

Eliminates all the stopwords from the string

removeEngStopwords

public void removeEngStopwords()

Eliminates all the English stopwords from the string

removeNonAlphaNumeric

public void removeNonAlphaNumeric(int minimumSize)

Remove any token which is not the in [:alnum:] character class. It also removes the tokens with a length lower than a minimum size.

Parameters:

minimumSize - Minimum size of accepted tokens. If equals to 0, all the tokens will be accepted

removeNonAlphabetic

public void removeNonAlphabetic(int minimumSize)

Remove any token which is not in [:alpha:] character class. It also removes the tokens with a length less than minimum size. However, a minimum size of 0 implies any length restriction.

Parameters:

minimumSize - Minimum size of accepted tokens. If it equals 0, all the tokens will be accepted

normalizeText

public static java.lang.String normalizeText(java.lang.String text)

Normalize the text by shrinking white spaces in one as well as substituting quotations, dashes and dots. TODO include an example

Parameters:

text -

Returns:

normalizeAndDeAposText

public java.lang.String normalizeAndDeAposText(java.lang.String text)

Normalize the text by shrinking white spaces in one as well as substituting quotations, dashes and dots. It substitutes apostrophes by a space as a previous step for tokenisation with only BreakIterators

Parameters:

text -

Returns:

stem

public void stem()

stemLucene

public void stemLucene()

setString

public void setString(java.lang.String str)

Stores a copy of the original string and generates a tokenized copy.

Parameters:

str -

setStringTokens

public void setStringTokens(java.lang.String str)

Stores a copy of the original string and generates a tokenized copy. Apostrophes are substituted by spaces as a previous step for tokenisation with only BreakIterators.

Parameters:

str -

getString

public java.lang.String getString()

getTokens

public java.util.List<java.lang.String> getTokens()

getOriginalString

public java.lang.String getOriginalString()