eus.ixa.ixa.pipe.pos

Class StringUtils

java.lang.Object

eus.ixa.ixa.pipe.pos.StringUtils

public final class StringUtils
extends Object

Pattern matching and other utility string functions.

Version:

2013-03-19

Author:

ragerri

Method Summary

Modifier and Type	Method and Description
static void	computeShortestEditScript(String wordForm, String lemma, int[][] distance, StringBuffer permutations) Computes the Shortest Edit Script (SES) to convert a word into its lemma.
static String	decodeShortestEditScript(String wordForm, String permutations) Read predicted SES by the lemmatizer model and apply the permutations to obtain the lemma from the wordForm.
static List<Integer>	exactStringFinder(String pattern, String sentence) Finds a pattern (typically a named entity string) in a sentence string.
static List<Integer>	exactTokenFinder(String pattern, String[] tokens) Finds a pattern (typically a named entity string) in a tokenized sentence.
static List<Integer>	exactTokenFinderIgnoreCase(String pattern, String[] tokens) Finds a pattern (typically a named entity string) in a tokenized sentence.
static List<File>	getFilesInDir(File inputPath) Recursively get every file in a directory and add them to a list.
static String	getShortestEditScript(String wordForm, String lemma) Get the SES required to go from a word to a lemma.
static String	getStringFromSpan(opennlp.tools.util.Span reducedSpan, String[] tokens) It takes a NE span indexes and the tokens in a sentence and produces the string to which the NE span corresponds to.
static String	getStringFromTokens(String[] tokens) Gets the String joined by a space of an array of tokens.
static int[][]	levenshteinDistance(String wordForm, String lemma) Computes the Levenshtein distance of two strings in a matrix.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

exactTokenFinderIgnoreCase

public static List<Integer> exactTokenFinderIgnoreCase(String pattern,
                                                       String[] tokens)

Finds a pattern (typically a named entity string) in a tokenized sentence. It outputs the Span indexes of the named entity found, if any.

Parameters:

pattern - a string to find

tokens - an array of tokens

Returns:

token spans of the pattern (e.g. a named entity)

exactTokenFinder

public static List<Integer> exactTokenFinder(String pattern,
                                             String[] tokens)

Finds a pattern (typically a named entity string) in a tokenized sentence. It outputs the Span indexes of the named entity found, if any

Parameters:

pattern - a string to find

tokens - an array of tokens

Returns:

token spans of the pattern (e.g. a named entity)

exactStringFinder

public static List<Integer> exactStringFinder(String pattern,
                                              String sentence)

Finds a pattern (typically a named entity string) in a sentence string. It outputs the offsets for the start and end characters named entity found, if any.

Parameters:

pattern - the pattern to be searched

sentence - the sentence

Returns:

a list of integers corresponding to the characters of the string found

getStringFromSpan

public static String getStringFromSpan(opennlp.tools.util.Span reducedSpan,
                                       String[] tokens)

It takes a NE span indexes and the tokens in a sentence and produces the string to which the NE span corresponds to. This function is used to get the Named Entity or Name textual representation from a Span

Parameters:

reducedSpan - a Span

tokens - an array of tokens

Returns:

named entity string

getStringFromTokens

public static String getStringFromTokens(String[] tokens)

Gets the String joined by a space of an array of tokens.

Parameters:

tokens - an array of tokens representing a tokenized sentence

Returns:

sentence the sentence corresponding to the tokens

getFilesInDir

public static List<File> getFilesInDir(File inputPath)

Recursively get every file in a directory and add them to a list.

Parameters:

inputPath - the input directory

Returns:

the list containing all the files

levenshteinDistance

public static int[][] levenshteinDistance(String wordForm,
                                          String lemma)

Computes the Levenshtein distance of two strings in a matrix. Based on pseudo-code provided here: https://en.wikipedia.org/wiki/Levenshtein_distance#Computing_Levenshtein_distance which in turn is based on the paper Wagner, Robert A.; Fischer, Michael J. (1974), "The String-to-String Correction Problem", Journal of the ACM 21 (1): 168-173

Parameters:

wordForm - the form

lemma - the lemma

Returns:

the distance

computeShortestEditScript

public static void computeShortestEditScript(String wordForm,
                                             String lemma,
                                             int[][] distance,
                                             StringBuffer permutations)

Computes the Shortest Edit Script (SES) to convert a word into its lemma. This is based on Chrupala's PhD thesis (2007).

Parameters:

wordForm - the token

lemma - the target lemma

distance - the levenshtein distance

permutations - the number of permutations

decodeShortestEditScript

public static String decodeShortestEditScript(String wordForm,
                                              String permutations)

Read predicted SES by the lemmatizer model and apply the permutations to obtain the lemma from the wordForm.

Parameters:

wordForm - the wordForm

permutations - the permutations predicted by the lemmatizer model

Returns:

the lemma

getShortestEditScript

public static String getShortestEditScript(String wordForm,
                                           String lemma)

Get the SES required to go from a word to a lemma.

Parameters:

wordForm - the word

lemma - the lemma

Returns:

the shortest edit script