cat.lump.aq.textextraction.wikipedia.utilities

Class ArticlesTranslator

java.lang.Object

cat.lump.aq.textextraction.wikipedia.utilities.ArticlesTranslator

public class ArticlesTranslator
extends java.lang.Object

A class to translate all the articles from L1 into L2 in a folder with the structure of Wikicardi: path/plain/L1/index/id.L1.txt The index files with the position of the articles and its length is required.

Since:

Feb 18, 2015 TODO: Convert into a cli? TODO: think of moses as a server and/or apache's tika library

Author:

cristina

Constructor Summary

Constructors

Constructor and Description
ArticlesTranslator(java.lang.String[] langs, java.lang.String pathArticles)

Method Summary

Methods

Modifier and Type	Method and Description
void	generateSetsCommon(java.lang.String commonArticles, boolean onlyMissing) Looks for the index files in the given folder and generates temporal files (tokenised) to be translated with only the articles that also appear in file String commonArticles.
void	generateSetsFullFolder(java.util.HashSet<java.lang.Integer> commonIDs, boolean onlyMissing) Looks for the index files in the given folder and generates temporal files (tokenised) to be translated.
static void	main(java.lang.String[] args)
java.lang.String	mirrorCommonArticles2Langs(java.lang.String commonArticles) Method for converting the file with the articles in common L1.icat1.L2.icat2.method into the mirror file with L2.icat2.L1.icat1.method.
void	reconstructTradArticles(java.util.HashSet<java.lang.Integer> commonIDs, boolean onlyMissing) Given the original index files and the temporal files already translated, the method reconstructs the translation of every individual article and saves them in path/plain/L1/index/id.trad.L2.txt
void	reconstructTradArticlesCommon(java.lang.String commonArticles, boolean onlyMissing) Given the original index files and the temporal files already translated, the method reconstructs the translation of every individual article that was originally in the file of common articles and saves them in path/plain/L1/index/id.trad.L2.txt
void	translateSets(java.lang.String translator, int sgeThreads) Main method to call the decoder String translator for all the files generated by generateSetsFullFolder().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArticlesTranslator

public ArticlesTranslator(java.lang.String[] langs,
                  java.lang.String pathArticles)

Method Detail

main

public static void main(java.lang.String[] args)

Parameters:

args -

translateSets

public void translateSets(java.lang.String translator,
                 int sgeThreads)
                   throws java.lang.InterruptedException,
                          java.util.concurrent.ExecutionException

Main method to call the decoder String translator for all the files generated by generateSetsFullFolder(). The processes are distributed among the available processors.

Parameters:

translator -

Throws:

java.lang.InterruptedException

java.util.concurrent.ExecutionException

reconstructTradArticlesCommon

public void reconstructTradArticlesCommon(java.lang.String commonArticles,
                                 boolean onlyMissing)

Given the original index files and the temporal files already translated, the method reconstructs the translation of every individual article that was originally in the file of common articles and saves them in path/plain/L1/index/id.trad.L2.txt

Parameters:

onlyMissing -

reconstructTradArticles

public void reconstructTradArticles(java.util.HashSet<java.lang.Integer> commonIDs,
                           boolean onlyMissing)

Given the original index files and the temporal files already translated, the method reconstructs the translation of every individual article and saves them in path/plain/L1/index/id.trad.L2.txt

Parameters:

commonIDs -

onlyMissing -

generateSetsCommon

public void generateSetsCommon(java.lang.String commonArticles,
                      boolean onlyMissing)

Looks for the index files in the given folder and generates temporal files (tokenised) to be translated with only the articles that also appear in file String commonArticles. TODO also lowercase?? These files are the concatenation of the articles in the corresponding index folder with length

Parameters:

onlyMissing -

generateSetsFullFolder

public void generateSetsFullFolder(java.util.HashSet<java.lang.Integer> commonIDs,
                          boolean onlyMissing)

Looks for the index files in the given folder and generates temporal files (tokenised) to be translated. (TODO also lowercase??) These files are the concatenation of the articles in the corresponding index folder with length

Parameters:

commonIDs -

onlyMissing -

mirrorCommonArticles2Langs

public java.lang.String mirrorCommonArticles2Langs(java.lang.String commonArticles)

Method for converting the file with the articles in common L1.icat1.L2.icat2.method into the mirror file with L2.icat2.L1.icat1.method. It can be used to translate articles in both directions.

Parameters:

commonArticles -

Returns:

String with the name of the new file