cat.lump.ir.retrievalmodels.document

Class PseudoCognates

java.lang.Object

cat.lump.ir.retrievalmodels.document.PseudoCognates

All Implemented Interfaces:

java.io.Serializable

public class PseudoCognates
extends java.lang.Object

A document representation based in Simard cognateness model.
A token t is candidate of be a cognate of another if:

1. t is entirely composed of letters and digits, but contains at least one digit
2. t is exclusively composed of letters, and is at least four letters long
3. t is a single punctuation character

Reference: Michel Simard, George F. Foster and Piere Isabelle. Using Cognates to Align Sentences in Bilingual Corpora

The cognates are preprocessed as follows:

• casefolding
• if it's a cognate exclusively composed of letters, then it's truncated to 4 letters

WARN: Code copied from cat.talp.lump.aq.text.SimardCognateness (project lump)

Author:

jboldoba

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Dictionary	dictionary
protected int	GLOBAL_COUNTER Keeps track of the amount of considered tokens
protected java.util.Map<java.lang.Integer,java.lang.Double>	TERMS The terms in the document together with their weights

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	PseudoCognates(Dictionary dictionary, java.util.Locale language)

Method Summary

Methods

Modifier and Type	Method and Description
protected java.lang.String	getDummyWord()
java.util.Map<java.lang.String,java.lang.Double>	getNormalizedRepresentation()
java.util.List<java.lang.String>	getRepresentation()
protected java.util.List<java.lang.String>	getTokens(java.lang.String text)
java.util.Map<java.lang.String,java.lang.Double>	getWeightedRepresentation()
int	length()
protected java.lang.String	preprocess(java.lang.String text)
void	setText(java.lang.String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TERMS

protected java.util.Map<java.lang.Integer,java.lang.Double> TERMS

The terms in the document together with their weights

GLOBAL_COUNTER

protected int GLOBAL_COUNTER

Keeps track of the amount of considered tokens

dictionary

protected Dictionary dictionary

Constructor Detail

PseudoCognates

protected PseudoCognates(Dictionary dictionary,
              java.util.Locale language)

Method Detail

setText

public void setText(java.lang.String text)

preprocess

protected java.lang.String preprocess(java.lang.String text)

getDummyWord

protected java.lang.String getDummyWord()

Returns:

A random token

length

public int length()

Returns:

Number of terms in the representation

getTokens

protected java.util.List<java.lang.String> getTokens(java.lang.String text)

Parameters:

text -

Returns:

tokens in the text

getRepresentation

public java.util.List<java.lang.String> getRepresentation()

getWeightedRepresentation

public java.util.Map<java.lang.String,java.lang.Double> getWeightedRepresentation()

Returns:

map of terms and their weight

getNormalizedRepresentation

public java.util.Map<java.lang.String,java.lang.Double> getNormalizedRepresentation()

Returns:

map of terms and their normalized weight