lux.index.analysis

Class XmlTextTokenStream

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

lux.index.analysis.XmlTokenStreamBase

lux.index.analysis.TextOffsetTokenStream

lux.index.analysis.XmlTextTokenStream

All Implemented Interfaces:

Closeable, AutoCloseable

public final class XmlTextTokenStream
extends TextOffsetTokenStream

Extracts tokens from an s9api XML document tree (XdmNode) in order to make them available to Lucene classes that accept TokenStreams, like the indexer and highlighter.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.lucene.analysis.TokenStream	origWrapped

Fields inherited from class lux.index.analysis.XmlTokenStreamBase

charStream, contentIter, curNode, defVis, eltVis, empty, EMPTY, termAtt

Constructor Summary

Constructors

Constructor and Description
XmlTextTokenStream(String fieldName, org.apache.lucene.analysis.Analyzer analyzer, org.apache.lucene.analysis.TokenStream wrapped, net.sf.saxon.s9api.XdmNode doc, Offsets offsets) Creates a TokenStream returning tokens drawn from the text content of the document.

Method Summary

Methods inherited from class lux.index.analysis.TextOffsetTokenStream

resetTokenizer

Methods inherited from class lux.index.analysis.XmlTokenStreamBase

getDefaultVisibility, getElementVisibility, getWrappedTokenStream, incrementToken, incrementWrappedTokenStream, reset, reset, setDefaultVisibility, setElementVisibility, setWrappedTokenStream

Methods inherited from class org.apache.lucene.analysis.TokenStream

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

origWrapped

protected final org.apache.lucene.analysis.TokenStream origWrapped

Constructor Detail

XmlTextTokenStream

public XmlTextTokenStream(String fieldName,
                  org.apache.lucene.analysis.Analyzer analyzer,
                  org.apache.lucene.analysis.TokenStream wrapped,
                  net.sf.saxon.s9api.XdmNode doc,
                  Offsets offsets)

Creates a TokenStream returning tokens drawn from the text content of the document.

Parameters:

fieldName - nominally: the field to be analyzed; the analyzer receives this when the token stream is reset at node boundaries

analyzer - specifies what text processing to apply to node text

wrapped - a TokenStream generated by the analyzer

doc - tokens will be drawn from all of the text in this document

offsets - if provided, character offsets are captured in this object In theory this can be used for faster highlighting, but until that is proven, this should always be null.