cat.lump.aq.textextraction.wikipedia.categories

Class ArticleTextExtractor

java.lang.Object

cat.lump.aq.textextraction.wikipedia.categories.ArticleTextExtractor

public class ArticleTextExtractor
extends java.lang.Object

This class provides methods to load a list of Wikipedia articles IDs and preprocess them.

Author:

jboldoba TODO check whether lists are already removed TODO (potentially) apply the language identifier in order to discard noisy sentences

Constructor Summary

Constructors

Constructor and Description
ArticleTextExtractor(java.util.Locale language, int year) Creates a preprocessor without any page to preprocess.
ArticleTextExtractor(java.util.Locale language, int year, java.io.File listOfPages) Creates a preprocessor with the pages listed in listOfPages.

Method Summary

Methods

Modifier and Type	Method and Description
void	addPage(int id) Adds a page ID to the set of page IDs
void	addPages(java.util.Collection<java.lang.Integer> ids) Adds a collection of pages to the set of page IDs
void	addPreprocess(TypePreprocess preprocess) Adds a new preprocess to the available ones.
static void	extractEntireWikipedia(java.util.Locale locale, int year, java.io.File directory) The entire Wiki for the set language and year
static void	extractSpecificArticles(java.util.Locale locale, int year, java.io.File pagesFile, java.io.File directory) Extract only the articles specified in the pagesFile
static void	extractSpecificArticles(java.util.Locale locale, int year, java.lang.Integer[] articleIDs, java.io.File directory)
java.util.Set<TypePreprocess>	getAvailablePreprocesses()
java.io.File	getRootDirectory()
boolean	isAvailablePreprocess(TypePreprocess preprocess) Query if exists the preprocess within the available preprocesses.
void	loadPages(java.io.File list) Loads all the page IDs of the file list.
void	loadPages(java.lang.Integer[] list)
static void	main(java.lang.String[] args) Extract texts from Wikipedia articles and save them into text files, after some given preprocessing.
void	preprocess(TypePreprocess preprocess) Preprocess all the pages with the preprocess method indentified by preprocess
void	preprocessAll() Applies all the available preprocessing steps to the pages.
void	removePage(int id)
boolean	removePreprocess(TypePreprocess preprocess) Removes the preprocess if it belongs to the set of available preprocesses.
void	setRootDirectory(java.io.File directory) Changes the root directory of the preprocessor.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArticleTextExtractor

public ArticleTextExtractor(java.util.Locale language,
                    int year)

Creates a preprocessor without any page to preprocess. The root directory is set to the user's home.

Parameters:

language - Language of the Wikipedia

year - Year of the dump of Wikipedia

ArticleTextExtractor

public ArticleTextExtractor(java.util.Locale language,
                    int year,
                    java.io.File listOfPages)
                     throws java.io.IOException

Creates a preprocessor with the pages listed in listOfPages. This file must have one ID by line. The root directory is set to the user's home.

Parameters:

language - Language of the Wikipedia

year - Year of the dump of Wikipedia

listOfPages - File with one page ID by line.

Throws:

java.io.IOException

Method Detail

preprocessAll

public void preprocessAll()
                   throws java.lang.InterruptedException,
                          WikiApiException

Applies all the available preprocessing steps to the pages.

Throws:

java.lang.InterruptedException

WikiApiException

preprocess

public void preprocess(TypePreprocess preprocess)
                throws java.lang.InterruptedException,
                       WikiApiException

Preprocess all the pages with the preprocess method indentified by preprocess

Parameters:

preprocess - Identifier of the preprocess

Throws:

java.lang.InterruptedException

WikiApiException

loadPages

public void loadPages(java.io.File list)
               throws java.io.IOException

Loads all the page IDs of the file list. The older loaded pages are kept in the preprocessor.

Parameters:

list - File with one page ID by line.

Throws:

java.io.IOException

loadPages

public void loadPages(java.lang.Integer[] list)

addPage

public void addPage(int id)

Adds a page ID to the set of page IDs

Parameters:

id - ID of the page

removePage

public void removePage(int id)

addPages

public void addPages(java.util.Collection<java.lang.Integer> ids)

Adds a collection of pages to the set of page IDs

Parameters:

ids - Collection of page IDs

addPreprocess

public void addPreprocess(TypePreprocess preprocess)

Adds a new preprocess to the available ones. If preprocess is already added, nothing is done

Parameters:

preprocess - The new preprocess. It can't be a null object.

removePreprocess

public boolean removePreprocess(TypePreprocess preprocess)

Removes the preprocess if it belongs to the set of available preprocesses.

Parameters:

preprocess - Preprocess to remove.

Returns:

.

setRootDirectory

public void setRootDirectory(java.io.File directory)

Changes the root directory of the preprocessor. If directory isn't a valid directory (with writable permissions), any change is done.

Parameters:

directory - A valid directory.

isAvailablePreprocess

public boolean isAvailablePreprocess(TypePreprocess preprocess)

Query if exists the preprocess within the available preprocesses.

Parameters:

preprocess - Desired preprocess.

Returns:

if the desired preprocess is available.

extractSpecificArticles

public static void extractSpecificArticles(java.util.Locale locale,
                           int year,
                           java.lang.Integer[] articleIDs,
                           java.io.File directory)

extractSpecificArticles

public static void extractSpecificArticles(java.util.Locale locale,
                           int year,
                           java.io.File pagesFile,
                           java.io.File directory)

Extract only the articles specified in the pagesFile

Parameters:

locale -

year -

pagesFile -

directory -

getAvailablePreprocesses

public java.util.Set<TypePreprocess> getAvailablePreprocesses()

Returns:

The set of available preprocesses.

getRootDirectory

public java.io.File getRootDirectory()

Returns:

The current root directory.

extractEntireWikipedia

public static void extractEntireWikipedia(java.util.Locale locale,
                          int year,
                          java.io.File directory)

The entire Wiki for the set language and year

main

public static void main(java.lang.String[] args)

Extract texts from Wikipedia articles and save them into text files, after some given preprocessing. The Wikipedia edition is defined with the language and year provided. Moreover, an output root directory is given to write the files.

Parameters:

args - language, year [file with ids]