org.ow2.weblab.service.language

Class LanguageExtraction

java.lang.Object

org.ow2.weblab.service.language.LanguageExtraction

All Implemented Interfaces:

org.ow2.weblab.core.services.Analyser

public class LanguageExtraction
extends java.lang.Object
implements org.ow2.weblab.core.services.Analyser

This class is a WebLab Web service for identifying the language of a Text.
It's a wrapper of the NGramJ project: "http://ngramj.sourceforge.net/". It uses the CNGram system that can computes character string instead of raw text files.
This algorithm return for each input text a score associated to every language profile previously learned (.ngp files). The score is a double between '0' and '1'. '1' meaning that this text is written in this language for sure. '0' on the opposite means that this text is not written in this language. The sum of score equals '1'.
Our wrapper annotate every Text section of a Document in input (or the Text if the input is a Text). It fails if the input is something else. On each Text it uses CNGram to determine which language profile are the best candidate to be annotated (using DC:language property). It can be configured using one of its custom constructor that enable the specification of those 8 parameters:

• maxNbValues: It's a positive integer value. The list of annotated language on a given Text could not be greater that this value.
• addTopLevelAnnot: It's a boolean value. It defines whether or not to annotate the whole document with the language extracted from the concatenation of every Text content.
• addMediaUnitLevelAnnot: It's a boolean value. It defines whether or not to annotate the each Text section with the language guessed.
• profilesFolderPath: It's a String that represents a folder path; This folder contains .ngp files that will be loaded instead of default CNGram 28 languages.
• isProducedByObject: It's a String value that should be a valid URI. It defines the URI to be used as object of every isProducedBy statements on annotations created by the service.
• unknownLanguageCode: It's the String value that will be annotated when no language can be clearly identified. When null, nothing is annotated.
• minSingleValue: It's a double value between 0 and 1. If the best language score is greater than this value, it will be the only one annotated on a given Text.
• minMultipleValue: It's a double value between 0 and 1. Every language score that are greater than this value, will be annotated on a given Text.

Those 8 properties are optional. Default values are:

• maxNbValues: '1'
• addTopLevelAnnot: false
• addMediaUnitLevelAnnot: true
• profilesFolderPath: in this case, we use the default constructor for CNGram profile that will use default profile given in their jar file. These 28 profiles are named using ISO 639-1 two letters language code; it means that the DC:language annotation resulting will be in this format. If you want to use another format, you have use a custom profiles folder (containing .ngp files).
• isProducedByObject: null in this case, no isProducedBy annotation will be created.
• unknownLanguageCode: 'und' is used, since it is the code for undetermined in ISO-639-x.
• minSingleValue: '0.75'
• minMultipleValue: '0.15'

Author:

EADS IPCC Team

Date:

2009-11-05

Constructor Summary

Constructors

Constructor and Description
LanguageExtraction() The default constructor that initialises each parameter as state in the class header.
LanguageExtraction(int maxNbValues, boolean addTopLevelAnnot, boolean addMediaUnitLevelAnnot, java.lang.String profilesFolderPath)
LanguageExtraction(int maxNbValues, boolean addTopLevelAnnot, boolean addMediaUnitLevelAnnot, java.lang.String profilesFolderPath, java.lang.String isProducedByObject)
LanguageExtraction(int maxNbValues, boolean addTopLevelAnnot, boolean addMediaUnitLevelAnnot, java.lang.String profilesFolderPath, java.lang.String isProducedByObject, java.lang.String unknownLanguageCode)
LanguageExtraction(int maxNbValues, boolean addTopLevelAnnot, boolean addMediaUnitLevelAnnot, java.lang.String profilesFolderPath, java.lang.String isProducedByObject, java.lang.String unknownLanguageCode, double minSingleValue, double minMultipleValue)

Method Summary

Methods

Modifier and Type	Method and Description
NGramProfilesPatched	loadProfiles(java.lang.String profilesFolderPath) Loads a NGramProfilesPatched from the parameter or uses default one (if null or empty).
org.ow2.weblab.core.services.analyser.ProcessReturn	process(org.ow2.weblab.core.services.analyser.ProcessArgs processArgs)
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

LanguageExtraction

public LanguageExtraction()

The default constructor that initialises each parameter as state in the class header.

LanguageExtraction

public LanguageExtraction(int maxNbValues,
                  boolean addTopLevelAnnot,
                  boolean addMediaUnitLevelAnnot,
                  java.lang.String profilesFolderPath)

Parameters:

maxNbValues - It's a positive integer value. The list of annotated language on a given Text could not be greater that this value.

addTopLevelAnnot - It's a boolean value. It defines whether or not to annotate the whole document with the language extracted from the concatenation of every Text content.

addMediaUnitLevelAnnot - It's a boolean value. It defines whether or not to annotate the each Text section with the language guessed.

profilesFolderPath - It's a String that represents a folder path; This folder contains .ngp files that will be loaded instead of default CNGram 28 languages.

LanguageExtraction

public LanguageExtraction(int maxNbValues,
                  boolean addTopLevelAnnot,
                  boolean addMediaUnitLevelAnnot,
                  java.lang.String profilesFolderPath,
                  java.lang.String isProducedByObject)

Parameters:

maxNbValues - It's a positive integer value. The list of annotated language on a given Text could not be greater that this value.

addTopLevelAnnot - It's a boolean value. It defines whether or not to annotate the whole document with the language extracted from the concatenation of every Text content.

addMediaUnitLevelAnnot - It's a boolean value. It defines whether or not to annotate the each Text section with the language guessed.

profilesFolderPath - It's a String that represents a folder path; This folder contains .ngp files that will be loaded instead of default CNGram 28 languages.

isProducedByObject - It's a String value that should be a valid URI. It defines the URI to be used as object of every isProducedBy statements on annotations created by the service.

LanguageExtraction

public LanguageExtraction(int maxNbValues,
                  boolean addTopLevelAnnot,
                  boolean addMediaUnitLevelAnnot,
                  java.lang.String profilesFolderPath,
                  java.lang.String isProducedByObject,
                  java.lang.String unknownLanguageCode)

Parameters:

maxNbValues - It's a positive integer value. The list of annotated language on a given Text could not be greater that this value.

addTopLevelAnnot - It's a boolean value. It defines whether or not to annotate the whole document with the language extracted from the concatenation of every Text content.

addMediaUnitLevelAnnot - It's a boolean value. It defines whether or not to annotate the each Text section with the language guessed.

profilesFolderPath - It's a String that represents a folder path; This folder contains .ngp files that will be loaded instead of default CNGram 28 languages.

isProducedByObject - It's a String value that should be a valid URI. It defines the URI to be used as object of every isProducedBy statements on annotations created by the service.

unknownLanguageCode - It's the String value that will be annotated when no language can be clearly identified. When null, nothing is annotated.

LanguageExtraction

public LanguageExtraction(int maxNbValues,
                  boolean addTopLevelAnnot,
                  boolean addMediaUnitLevelAnnot,
                  java.lang.String profilesFolderPath,
                  java.lang.String isProducedByObject,
                  java.lang.String unknownLanguageCode,
                  double minSingleValue,
                  double minMultipleValue)

Parameters:

maxNbValues - It's a positive integer value. The list of annotated language on a given Text could not be greater that this value.

addTopLevelAnnot - It's a boolean value. It defines whether or not to annotate the whole document with the language extracted from the concatenation of every Text content.

addMediaUnitLevelAnnot - It's a boolean value. It defines whether or not to annotate the each Text section with the language guessed.

profilesFolderPath - It's a String that represents a folder path; This folder contains .ngp files that will be loaded instead of default CNGram 28 languages.

isProducedByObject - It's a String value that should be a valid URI. It defines the URI to be used as object of every isProducedBy statements on annotations created by the service.

unknownLanguageCode - It's the String value that will be annotated when no language can be clearly identified. When null, nothing is annotated.

minSingleValue - It's a double value between 0 and 1. If the best language score is greater than this value, it will be the only one annotated on a given Text.

minMultipleValue - It's a double value between 0 and 1. Every language score that are greater than this value, will be annotated on a given Text.

Method Detail

loadProfiles

public NGramProfilesPatched loadProfiles(java.lang.String profilesFolderPath)

Loads a NGramProfilesPatched from the parameter or uses default one (if null or empty).

Parameters:

profilesFolderPath - The path to a folder to be loaded. If null or empty, uses default profiles shipped with NGramJ

Returns:

An instance of NGramProfilesPatched

Throws:

InitialisationException - If the initialisation fails

process

public org.ow2.weblab.core.services.analyser.ProcessReturn process(org.ow2.weblab.core.services.analyser.ProcessArgs processArgs)
                                                            throws org.ow2.weblab.core.services.InvalidParameterException,
                                                                   org.ow2.weblab.core.services.UnexpectedException

Specified by:

process in interface org.ow2.weblab.core.services.Analyser

Throws:

org.ow2.weblab.core.services.InvalidParameterException

org.ow2.weblab.core.services.UnexpectedException

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object