lux.index.analysis

Class ElementTokenStream

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

lux.index.analysis.XmlTokenStreamBase

lux.index.analysis.TextOffsetTokenStream

lux.index.analysis.ElementTokenStream

All Implemented Interfaces:

Closeable, AutoCloseable

public final class ElementTokenStream
extends TextOffsetTokenStream

A TokenStream that extracts text from a Saxon Document model (XdmNode) and generates a token for every "word" for every element that contains it. TODO: control over element transparency

Each element name may be one of: transparent, opaque, hidden, or container. The default may be set to either opaque or transparent. Unless hidden, text is tagged with its parent element. If its parent is transparent, it is also tagged with ancestor elements, stopping at the first opaque or container element. In addition, visible (non-hidden) text is tagged with all ancestor container elements.

ElementVisibility.OPAQUE

text content is indexed only with the parent element tag. Opaque elements' start and end tags act as phrase boundaries.

ElementVisibility.TRANSPARENT

if the parent element is transparent, the text is also indexed as if it were a child of the grandparent element; and the same rule applies recursively.

ElementVisibility.HIDDEN

descendant content of hidden elements is not indexed.

ElementVisibility.CONTAINER

a container element tags all of its visible descendants, regardless of opacity.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class lux.index.analysis.XmlTokenStreamBase

charStream, contentIter, curNode, defVis, eltVis, empty, EMPTY, termAtt

Constructor Summary

Constructors

Constructor and Description
ElementTokenStream(String fieldName, org.apache.lucene.analysis.Analyzer analyzer, org.apache.lucene.analysis.TokenStream wrapped, net.sf.saxon.s9api.XdmNode doc, Offsets offsets)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	updateNodeAtts()

Methods inherited from class lux.index.analysis.TextOffsetTokenStream

resetTokenizer

Methods inherited from class lux.index.analysis.XmlTokenStreamBase

getDefaultVisibility, getElementVisibility, getWrappedTokenStream, incrementToken, incrementWrappedTokenStream, reset, reset, setDefaultVisibility, setElementVisibility, setWrappedTokenStream

Methods inherited from class org.apache.lucene.analysis.TokenStream

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ElementTokenStream

public ElementTokenStream(String fieldName,
                  org.apache.lucene.analysis.Analyzer analyzer,
                  org.apache.lucene.analysis.TokenStream wrapped,
                  net.sf.saxon.s9api.XdmNode doc,
                  Offsets offsets)

Method Detail

updateNodeAtts

protected void updateNodeAtts()