cat.lump.aq.textextraction.wikipedia.utilities

Class CommonArticlesFinder

java.lang.Object

cat.lump.aq.textextraction.wikipedia.utilities.CommonArticlesFinder

public class CommonArticlesFinder
extends java.lang.Object

A class to identify the common articles across n languages in Wikipedia from files with the list of IDs for every language. Redirects are not resolved as it is necessary for dumps before 2015. Use CommonNamespaceFinder instead for the following dumps. The process queries directly to the SQL database.

Since:

Feb 12, 2015 TODO: Convert into a cli? TODO: Generalise for every article (no only those in a list) Si? (either here or generalise CommonCategoriesExtractor to deal with articles)

Author:

cristina

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String	pairs_db DB with langlinks & articles pairs

Constructor Summary

Constructors

Constructor and Description
CommonArticlesFinder(java.lang.String[] langs, int year, java.lang.String[] filesID, java.io.File folder) Instantiates the object with the provided languages.

Method Summary

Methods

Modifier and Type	Method and Description
void	checkAllTablesAvailable() Checks if all the tables needed are in the database.
void	closeConnection()
void	findIntersection(java.lang.String smallestLang) Looks for the articles that appear in all the selected languages langs simultaneously.
void	findUnion(java.lang.String largestLang) Looks for the articles that appear in any of the selected languages langs and builds a set with its union.
java.lang.String	getPrefixOutputFile() getters
java.lang.String	lookForMaximumNumber() Given the list of articles for every language String[] filesID the language from String[] langs with more articles is returned.
java.lang.String	lookForMinimumNumber() Given the list of articles for every language String[] filesID the language from String[] langs with less articles is returned.
static void	main(java.lang.String[] args) Example for using the class

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pairs_db

protected static final java.lang.String pairs_db

DB with langlinks & articles pairs

Constructor Detail

CommonArticlesFinder

public CommonArticlesFinder(java.lang.String[] langs,
                    int year,
                    java.lang.String[] filesID,
                    java.io.File folder)
                     throws java.lang.Throwable

Instantiates the object with the provided languages. The language with less articles is the base to find pairs/tuples in the intersection. The year must be one of the available dumps. Tables "langlinks" and "page" are necessary Languages are required in the Wikipedia code format (e.g. en for English and es for Spanish). The DB connections are invoked according to these parameters.

Parameters:

langs[] -

year -

folder -

Throws:

java.lang.Throwable

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Throwable

Example for using the class

Parameters:

args -

Throws:

java.lang.Throwable

findUnion

public void findUnion(java.lang.String largestLang)

Looks for the articles that appear in any of the selected languages langs and builds a set with its union. So, for other languages than the original, if the articles exist in the DB but do not appear in the ID files, they are added. Generates the list listUnion from which a file is printed by getAndPrintAllInfoUnionIDs(List<Integer> listUnion, String largestLang)

Parameters:

largestLang -

findIntersection

public void findIntersection(java.lang.String smallestLang)

Looks for the articles that appear in all the selected languages langs simultaneously. If they exist in the DB but do not appear in the files with the IDs, the articles are discarded. "ID \t title \t" of the articles is printed in a file with the information for all the languages concatenated in a row.

Parameters:

smallestLang -

lookForMinimumNumber

public java.lang.String lookForMinimumNumber()

Given the list of articles for every language String[] filesID the language from String[] langs with less articles is returned. The position of the language and the file must correspond in the arrays. TODO is this better than extracting the language from the name of the file?

Returns:

String minLang

lookForMaximumNumber

public java.lang.String lookForMaximumNumber()

Given the list of articles for every language String[] filesID the language from String[] langs with more articles is returned. The position of the language and the file must correspond in the arrays.

Returns:

String maxLang

closeConnection

public void closeConnection()

checkAllTablesAvailable

public void checkAllTablesAvailable()

Checks if all the tables needed are in the database. Exits otherwise.

getPrefixOutputFile

public java.lang.String getPrefixOutputFile()

getters