preprocess

Type Members

1. class JavaSentenceSegmenter extends SentenceSegmenter

   

   A Sentence Segmenter backed by Java's BreakIterator.

   A Sentence Segmenter backed by Java's BreakIterator. Given an input string, it will return an iterator over sentences

2. class JavaWordTokenizer extends Tokenizer

   

   A Word Segmenter backed by Java's BreakIterator.

   A Word Segmenter backed by Java's BreakIterator. Given an input string, it will return an iterator over sentences Doesn't return spaces, does return punctuation.

3. class MLSentenceSegmenter extends SentenceSegmenter with Serializable

   

   Annotations

   @SerialVersionUID()

4. class NewLineSentenceSegmenter extends SentenceSegmenter

   

   TODO move to chalk

5. case class RegexSearchTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   

   Finds all occurrences of the given pattern in the document.

6. case class RegexSplitTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   

   Splits the input document according to the given pattern.

   Splits the input document according to the given pattern. Does not return the splits.

7. class SegmentingIterator extends Iterator[Span]

   

8. trait SentenceSegmenter extends StringAnalysisFunction[Any, Sentence] with (String) ⇒ Iterable[String] with Serializable

   

9. class StreamSentenceSegmenter extends AnyRef

   

   TODO

10. trait Tokenizer extends StringAnalysisFunction[Sentence, Token] with Serializable with (String) ⇒ IndexedSeq[String]

    

    Abstract trait for tokenizers, which annotate sentence-segmented text with tokens.

    Abstract trait for tokenizers, which annotate sentence-segmented text with tokens. Tokenizers work with both raw strings and epic.slab.StringSlabs.

    Annotations

    @SerialVersionUID()

11. class TreebankTokenizer extends Tokenizer with Serializable

    

    Annotations

    @SerialVersionUID()

12. class WhitespaceTokenizer extends RegexSplitTokenizer

    

    Tokenizes by splitting on the regular expression \s+.