tokenize

Type Members

1. class JavaWordTokenizer extends Tokenizer

   A Word Segmenter backed by Java's BreakIterator.

2. case class RegexSearchTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   Finds all occurrences of the given pattern in the document.

3. case class RegexSplitTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   Splits the input document according to the given pattern.

4. trait SimpleEnglishTokenizer extends Tokenizer

   Simple English document tokenizer that splits up words on whitespace or punctuation, but keeps word-internal punctuation within the word.

5. trait Tokenizer extends (String) ⇒ Iterable[String] with Serializable

   Abstract trait for tokenizers, which act as functions from a String to an Iterable[String].

6. class WhitespaceTokenizer extends RegexSplitTokenizer

   Tokenizes by splitting on the regular expression \s+.