DomainVocabulary (WikiTailor)

java.lang.Object
- cat.lump.aq.textextraction.wikipedia.categories.DomainVocabulary

All Implemented Interfaces:

java.io.Serializable
```
public class DomainVocabulary
extends java.lang.Object
implements java.io.Serializable
```
A DomainVocabulary instance is used to store a set of terms with its frequency. It means, each entry contains the number of times that has been added to the dictionary (its frequency).

It is able to insert terms from a text as well as a collection of strings. Before adding new terms they are preprocessed as follows:
1. Case folding the input.
2. Remove stop words.
3. Remove diacritics.
4. Apply a stemming process.
5. Filter only alphabetic tokens with a size greater than or equal to a given value (By default, characters).
After this preprocess, the unknown terms are added to the dictionary with a frequency of 1, but the previously contained are modified by incrementing its frequency by 1. The preprocess behavior can be easily changed extending this class and redefining the preprocess(String) function.

It also provides functions to get its content in other ways. On the one hand, it is able to save them in a file (binary or textual) with the serialize(File) and toFile(File) functions respectively. On the other hand, it can be transformed into a list of TermFrequencyTuple using the toList() function.
Author:

cmops

See Also:
TermFrequencyTuple, Serialized Form

Constructor Summary

Constructors
Constructor and Description
`DomainVocabulary(DomainVocabulary vocabulary)` Creates a new vocabulary identical to the given one.
`DomainVocabulary(java.util.Locale language)` Creates an empty vocabulary.
`DomainVocabulary(java.util.Locale language, java.util.Collection<TermFrequencyTuple> terms)` Creates a vocabulary which include some initial terms with an initial frequency.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`addTerms(java.util.Collection<java.lang.String> terms)` Add new terms from a collection of words.
`void`	`addTerms(java.lang.String text)` Add new terms or modify the already added with the tokens obtained by the preprocessing of the given text.
`boolean`	`contains(java.lang.String term)` Checks if a term is contained in the vocabulary.
`int`	`getFrequency(java.lang.String term)` Gives the frequency of the given term.
`java.util.Locale`	`getLanguage()`
`int`	`getMinimumSize()`
`java.util.Map<java.lang.String,java.lang.Integer>`	`getTerms()`
`DomainVocabulary`	`getTop(float percentage)` Extracts the most frequent terms of the vocabulary.
`DomainVocabulary`	`getTop(int qtt)` Extract the `qtt` most frequent terms of the vocabulary.
`void`	`insertFromFile(java.io.File input)` Inserts the terms stored in the given file into the domain vocabulary.
`static DomainVocabulary`	`loadfromFile(java.io.File input)` Creates a `DomainVocabulary` instance by reading a binary file which contains a domain vocabulary.
`protected java.util.Collection<java.lang.String>`	`preprocess(java.lang.String text)` Preprocess a text to extract its terms as defined in TermExtractor.getTerms();
`void`	`serialize(java.io.File output)` Saves the vocabulary in a binary file.
`void`	`setMinimumSize(int size)` Changes the minimum size of the tokens to be accepted as terms of the vocabulary.
`void`	`setTerms(java.util.Collection<TermFrequencyTuple> terms)` Sets a collection of tuples formed by a term and its frequency as vocabulary.
`void`	`toFile(java.io.File output)` Exports the vocabulary to a textual file.
`java.util.List<TermFrequencyTuple>`	`toList()` Exports the vocabulary as a list of tuples which contains the term and its frequency.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - DomainVocabulary
```
public DomainVocabulary(java.util.Locale language)
```
    Creates an empty vocabulary.
    
    Parameters:
    language - Language of the vocabulry.
  - DomainVocabulary
```
public DomainVocabulary(java.util.Locale language,
                java.util.Collection<TermFrequencyTuple> terms)
```
    Creates a vocabulary which include some initial terms with an initial frequency.
    
    Parameters:
    language - Language of the vocabulary.
    terms - Initial terms to include in the vocabulary.
  - DomainVocabulary
```
public DomainVocabulary(DomainVocabulary vocabulary)
```
    Creates a new vocabulary identical to the given one. It means, it copies the given vocabulary.
    
    Parameters:
    vocabulary - The reference vocabulary
- Method Detail
  - addTerms
```
public void addTerms(java.util.Collection<java.lang.String> terms)
```
    Add new terms from a collection of words. Only will be accepted the not empty tokens after preprocess them.
    
    Parameters:
    terms - Collection of terms to add.
  - addTerms
```
public void addTerms(java.lang.String text)
```
    Add new terms or modify the already added with the tokens obtained by the preprocessing of the given text.
    
    Parameters:
    text - The text with the new terms.
  - contains
```
public boolean contains(java.lang.String term)
```
    Checks if a term is contained in the vocabulary.
    
    Parameters:
    term - The term to check.
    
    Returns:
    true if the term is contained in the vocabulary. false otherwise.
  - getFrequency
```
public int getFrequency(java.lang.String term)
```
    Gives the frequency of the given term.
    
    Parameters:
    term - The term to check its frequency.
    
    Returns:
    The absolute frequency of the term. If the term is not contained in the vocabulary, it returns 0.
  - getTop
```
public DomainVocabulary getTop(float percentage)
```
    Extracts the most frequent terms of the vocabulary.
    
    Parameters:
    percentage - The percentage which defines the top to extract. The percentage is given by a value between 0 and 1.
    
    Returns:
    A new DomainVocabulary instance which contains the extracted terms.
  - getTop
```
public DomainVocabulary getTop(int qtt)
```
    Extract the qtt most frequent terms of the vocabulary.
    
    Parameters:
    qtt - Number of terms to extract
    
    Returns:
    A new DomainVocabulary instance which contains the extracted terms.
  - toList
```
public java.util.List<TermFrequencyTuple> toList()
```
    Exports the vocabulary as a list of tuples which contains the term and its frequency. This list has not any order.
    
    Returns:
    A list of tuples with the contents of the vocabulary.
    See Also:
    TermFrequencyTuple
  - toFile
```
public void toFile(java.io.File output)
            throws java.io.IOException
```
    Exports the vocabulary to a textual file. This file is composed by N lines where each one contains one vocabulary entry. The DomainVocabulary instances saves in this way can be loaded using the insertFromFile(File) function. Each line has the following format: "term\tfrequency".
    
    Parameters:
    output - Path to the output file. If it doesn't exists, it will be created. If it already exists, it will be overwritten.
    
    Throws:
    
    java.io.IOException
  - serialize
```
public void serialize(java.io.File output)
```
    Saves the vocabulary in a binary file. The DomainVocabulary instances saves in this way can be loaded using the loadfromFile(File) function.
    
    Parameters:
    output - Path to the output file. If it doesn't exists, it will be created. If it already exists, it will be overwritten.
  - loadfromFile
```
public static DomainVocabulary loadfromFile(java.io.File input)
```
    Creates a DomainVocabulary instance by reading a binary file which contains a domain vocabulary.
    
    Parameters:
    input - Path to the binary file which contains a domain vocabulary.
    
    Returns:
    A DomainVocabulary instance which is equal to the one stored in the input file.
  - insertFromFile
```
public void insertFromFile(java.io.File input)
```
    Inserts the terms stored in the given file into the domain vocabulary. The unkown terms of the file are added to the current vocabulary and the previously added terms are modified incrementing its frequency by the frequency stored in the file.
    
    Parameters:
    input - Path to the readable file which contains a set of terms stored with the format used by the toFile(File) function.
  - preprocess
```
protected java.util.Collection<java.lang.String> preprocess(java.lang.String text)
```
    Preprocess a text to extract its terms as defined in TermExtractor.getTerms();
    
    Parameters:
    text - The text to preprocess.
    
    Returns:
    The collection of tokens accepted from the given text.
  - setMinimumSize
```
public void setMinimumSize(int size)
```
    Changes the minimum size of the tokens to be accepted as terms of the vocabulary.
    
    Parameters:
    size - The new size.
  - setTerms
```
public void setTerms(java.util.Collection<TermFrequencyTuple> terms)
```
    Sets a collection of tuples formed by a term and its frequency as vocabulary. This call implies the purging of the current vocabulry.
    
    Parameters:
    terms - The new terms of the vocabulary.
  - getLanguage
```
public java.util.Locale getLanguage()
```
    Returns:
    The locale that defines the language used by this vocabulary.
  - getMinimumSize
```
public int getMinimumSize()
```
    Returns:
    The minimum size of the terms after preprocess them to be accepted as new term.
  - getTerms
```
public java.util.Map<java.lang.String,java.lang.Integer> getTerms()
```
    Returns:
    The terms and its frequency. Each key of the map is a term of the vocabulary and the value related to the key is the frequency of that term.

Class DomainVocabulary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

DomainVocabulary

DomainVocabulary

DomainVocabulary

Method Detail

addTerms

addTerms

contains

getFrequency

getTop

getTop

toList

toFile

serialize

loadfromFile

insertFromFile

preprocess

setMinimumSize

setTerms

getLanguage

getMinimumSize

getTerms