cat.lump.aq.textextraction.wikipedia.categories

Class DomainKeywords

java.lang.Object

cat.lump.aq.textextraction.wikipedia.categories.DomainKeywords

public class DomainKeywords
extends java.lang.Object

This class gets the most common terms in the articles belonging to, at least, one category of a given domain. A domain is defined by one category (root) and all of its subcategories.

Terms are stemmed and stopwords are not included.

TODO UNITTEST!!!

Author:

jboldoba

Constructor Summary

Constructors

Constructor and Description
DomainKeywords(java.util.Locale locale, int year)

Method Summary

Methods

Modifier and Type	Method and Description
void	computeTF() Gets the term frequency tuples resulting the treatment of a set of TODO this should be private!!
void	computeTF(java.lang.String categoryName) As computeTF() but including the title of the root category
java.lang.String	getLang()
java.util.List<TermFrequencyTuple>	getTermTuples()
java.util.List<TermFrequencyTuple>	getTopTuples()
java.util.List<TermFrequencyTuple>	getTopTuplesPlus(java.lang.String category)
int	getYear()
void	loadArticles(int categoryID) Loads the articles of the given category ID.
void	loadArticles(java.lang.String categoryName) Loads the articles of the given category.
static void	main(java.lang.String[] args) Main method.
void	setLang(java.util.Locale lang) Sets the language.
void	setMaxSize(int m) Defines the maximum number of terms that should be considered as domain terms
void	setMinNumArticles(int min) Defines the minimum number of articles required to build the vocabulary
void	setPercentage(int t) Defines the percentage of terms that should be considered as domain terms
void	setYear(int year)
void	toFile(java.io.File file) Saves the top list of a text file.
void	toFile(java.io.File file, java.util.List<TermFrequencyTuple> tfs) Saves the given list of TermFrequencyTuples into a file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DomainKeywords

public DomainKeywords(java.util.Locale locale,
              int year)

Method Detail

computeTF

public void computeTF()

Gets the term frequency tuples resulting the treatment of a set of TODO this should be private!! temporarily moved to fix issues with the main Wikipedia pages. ATTENTION: the contents, originally implementing all the tf-related computation, are now in cat.lump.ir.weighting.TermFrequency

Parameters:

articles - The Wikipedia pages to process

computeTF

public void computeTF(java.lang.String categoryName)

As computeTF() but including the title of the root category

Parameters:

articles - The Wikipedia pages to process

categoryName - The title of the root category

loadArticles

public void loadArticles(int categoryID)
                  throws WikiApiException

Loads the articles of the given category ID.

Parameters:

categoryID - Identifier of the category. The category must exist in the Wikipedia database

Throws:

WikiApiException - Raised if creating a Wikipedia connector is not possible for the given language and year.

loadArticles

public void loadArticles(java.lang.String categoryName)
                  throws WikiApiException

Loads the articles of the given category. The category is defined by its name.

Parameters:

categoryName - Name of the category

Throws:

WikiApiException - Raised if creating a Wikipedia connector is not possible for the given language and year or if the category name is not valid in that Wikipedia connector

toFile

public void toFile(java.io.File file)

Saves the top list of a text file. The resulting file contains one tab-separated term per line with frequency and term.

Parameters:

file - The file to save the list into.

toFile

public void toFile(java.io.File file,
          java.util.List<TermFrequencyTuple> tfs)

Saves the given list of TermFrequencyTuples into a file

Parameters:

file -

tfs -

getYear

public int getYear()

getLang

public java.lang.String getLang()

getTermTuples

public java.util.List<TermFrequencyTuple> getTermTuples()

getTopTuples

public java.util.List<TermFrequencyTuple> getTopTuples()

getTopTuplesPlus

public java.util.List<TermFrequencyTuple> getTopTuplesPlus(java.lang.String category)

setPercentage

public void setPercentage(int t)

Defines the percentage of terms that should be considered as domain terms

Parameters:

t - an integer in the range (0,100]

setMaxSize

public void setMaxSize(int m)

Defines the maximum number of terms that should be considered as domain terms

Parameters:

max - an integer

setMinNumArticles

public void setMinNumArticles(int min)

Defines the minimum number of articles required to build the vocabulary

Parameters:

min - is an integer

setLang

public void setLang(java.util.Locale lang)

Sets the language.

Parameters:

lang - The locale for the new language

setYear

public void setYear(int year)

main

public static void main(java.lang.String[] args)

Main method. It can be used as tool to obtain a domain list of terms. This program writes a file in the user working directory with the terms sorted by their frequency in descending order. The file is called "categoryID_language.dict"

Parameters:

args - List of parameters:

• Language: Language of the Wikipedia
• Year: Year of the Wikipedia
• Category: ID of the category which defines the domain
• Top: Desired percentage of the dictionary, e.g. top=10 implies write the 10% of the dictionary with the highest frequencies.