cat.lump.ir.sim.ml.esa

Class SimilarityESA

java.lang.Object

cat.lump.ir.sim.ml.esa.Esa

cat.lump.ir.sim.ml.esa.SimilarityESA

All Implemented Interfaces:

Similarity

Direct Known Subclasses:

SimilarityCLESA, SimilarityESAdocs, SimilarityESAlines

public abstract class SimilarityESA
extends Esa

Implementation of Explicit Semantic Analysis as described in:
Gabrilovich, Evgeniy, and Shaul Markovitch. "Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis". 1606–1611. Hyderabad, India, 2007.

For this version the inverted index is based on Lucene.
In brief, the process is as follows. Given two documents d₁ and d₂:

• query d₁ (d₂) to the previously generated index (with LuceneIndexer)
• retrieve and normalize the resulting relevance vectors for d₁ (d₂)
• compute the cosine similarity between the vector representations d₁ and d₂

Since:

April 18 2012

Version:

0.2.1

Author:

albarron

See Also:

cat.lump.ir.sim.EsaGenerator#

Field Summary

Fields

Modifier and Type	Field and Description
protected EsaGenerator	esaGen A generator of ESA vectorial representations
protected java.lang.String	objectA Identifier for object A
protected java.lang.String	objectB Identifier for object B
protected boolean	overrideObjects Whether previously computed semantic representations should be discarded (if they exist)
protected java.io.File	textsA Path to documents A
protected java.io.File	textsB Path to documents B

Fields inherited from class cat.lump.ir.sim.ml.esa.Esa

esaVectorsA, esaVectorsB, log

Constructor Summary

Constructors

Constructor and Description
SimilarityESA(java.lang.String indexPath, java.lang.String lan, java.lang.Boolean overrideObjects) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract EsaVectors	computeVectors(java.io.File documentsPath, java.lang.String object, java.lang.String set) Computes the vectors for the texts in the given set.
void	computeVectorsA() Compute the characteristic vectors for dataset A
void	computeVectorsB() Compute the characteristic vectors for dataset B
protected boolean	objectExists(java.io.File object, java.lang.String id) Checks whether the vector-representation object exists.
protected void	saveObject(java.io.File objFile, EsaVectors esa) Saves a textual representation into an object file
protected abstract void	setDocumentsPath(java.io.File documentsApath, java.io.File documentsBpath) A method that loads the texts in collections A and B.
protected void	setObjects() Set the name of the resulting vector objects

Methods inherited from class cat.lump.ir.sim.ml.esa.Esa

computePairwiseSimilarities, computeSimilarities, computeSimilarity, displaySimilarities, documentsExist, exitError, getPairwiseSimilarities, getSimilarities, getSimilaritiesMatrix, getSimilarity, getSimilarity

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

textsA

protected java.io.File textsA

Path to documents A

textsB

protected java.io.File textsB

Path to documents B

overrideObjects

protected boolean overrideObjects

Whether previously computed semantic representations should be discarded (if they exist)

esaGen

protected EsaGenerator esaGen

A generator of ESA vectorial representations

objectA

protected java.lang.String objectA

Identifier for object A

objectB

protected java.lang.String objectB

Identifier for object B

Constructor Detail

SimilarityESA

public SimilarityESA(java.lang.String indexPath,
             java.lang.String lan,
             java.lang.Boolean overrideObjects)

Constructor. It sets the path to the Lucene index, the language of the texts we want to compare and if previously computed ESA vectors should be discarded. If overrideObjects==false, previously-computed vector representations of the texts are looked for in the shape of a Java object file. If it doesn't exists, it will be created and saved for future use. TODO move the parameterization to a properties file

Parameters:

indexPath - path to Lucene's index

lan - language to work with

overrideObjects - if previously computed vectors will be discarded

Method Detail

setDocumentsPath

protected abstract void setDocumentsPath(java.io.File documentsApath,
                    java.io.File documentsBpath)

A method that loads the texts in collections A and B.

Parameters:

documentsApath -

documentsBpath -

setObjects

protected void setObjects()

Set the name of the resulting vector objects

computeVectorsA

public void computeVectorsA()
                     throws java.lang.ClassNotFoundException,
                            java.io.IOException

Compute the characteristic vectors for dataset A

Throws:

java.io.IOException

java.lang.ClassNotFoundException

computeVectorsB

public void computeVectorsB()
                     throws java.lang.ClassNotFoundException,
                            java.io.IOException

Compute the characteristic vectors for dataset B

Throws:

java.io.IOException

java.lang.ClassNotFoundException

computeVectors

protected abstract EsaVectors computeVectors(java.io.File documentsPath,
                        java.lang.String object,
                        java.lang.String set)
                                      throws java.lang.ClassNotFoundException,
                                             java.io.IOException

Computes the vectors for the texts in the given set. If a previously computed object exists (and overrideObjects==False), it just loads the previously computed vectors.

Parameters:

documentsPath - path to the documents

object - name of the (previously generated object)

set - whether we are processing A or B

Returns:

Throws:

java.io.IOException

java.lang.ClassNotFoundException

saveObject

protected void saveObject(java.io.File objFile,
              EsaVectors esa)
                   throws java.io.FileNotFoundException,
                          java.io.IOException

Saves a textual representation into an object file

Parameters:

objFile -

esa -

Throws:

java.io.FileNotFoundException

java.io.IOException

objectExists

protected boolean objectExists(java.io.File object,
                   java.lang.String id)

Checks whether the vector-representation object exists. Note that if override_object==True, it will claim no objects exists in order to compute it again.

Parameters:

object - object to check

id - flag to report

Returns:

true if exists and override_object==false