Package org.languagetool.dev.index

Class AnyCharTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.languagetool.dev.index.AnyCharTokenizer
All Implemented Interfaces:
Closeable, AutoCloseable
public final class AnyCharTokenizer extends org.apache.lucene.analysis.Tokenizer
A tokenizer that renders the whole input as one token.
Author:
Tao Lin

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

    org.apache.lucene.util.AttributeSource.State

  • Field Summary

    Fields inherited from class org.apache.lucene.analysis.Tokenizer

    input

    Fields inherited from class org.apache.lucene.analysis.TokenStream

    DEFAULT_TOKEN_ATTRIBUTE_FACTORY

  • Constructor Summary

    Constructors
    Constructor
    Description
    AnyCharTokenizer()
    Construct a new AnyCharTokenizer.
    AnyCharTokenizer(org.apache.lucene.util.AttributeFactory factory)
    Construct a new AnyCharTokenizer using a given AttributeFactory.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    end()
     
    boolean
    incrementToken()
     
    protected boolean
    isTokenChar(int c)
    Collects any characters.
    protected int
    normalize(int c)
     
    void
    reset()
     

    Methods inherited from class org.apache.lucene.analysis.Tokenizer

    close, correctOffset, setReader

    Methods inherited from class org.apache.lucene.util.AttributeSource

    addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • AnyCharTokenizer

      public AnyCharTokenizer()
      Construct a new AnyCharTokenizer.

    • AnyCharTokenizer

      public AnyCharTokenizer(org.apache.lucene.util.AttributeFactory factory)
      Construct a new AnyCharTokenizer using a given AttributeFactory.
      Parameters:
      factory - the attribute factory to use for this Tokenizer

  • Method Details

    • isTokenChar

      protected boolean isTokenChar(int c)
      Collects any characters.

    • normalize

      protected int normalize(int c)

    • incrementToken

      public boolean incrementToken() throws IOException
      Specified by:
      incrementToken in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • end

      public void end() throws IOException
      Overrides:
      end in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • reset

      public void reset() throws IOException
      Overrides:
      reset in class org.apache.lucene.analysis.Tokenizer
      Throws:
      IOException