lux.index.analysis

Class XmlTokenStreamBase

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

lux.index.analysis.XmlTokenStreamBase

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

TextOffsetTokenStream

public abstract class XmlTokenStreamBase
extends org.apache.lucene.analysis.TokenStream

This is the root of a set of xml-aware TokenStream classes that work by selecting text a node at a time from an XML document, and then passing that text to the wrapped TokenStream. The wrapped TokenStream is re-used for each text node. The outermost link in the chain will be a TokenFilter that applies a sequence of structure-related Attributes to each text token (ie a list of QNames, but can be any kind of structural attribute that should be composed with each text token).

The token stream topology is: this( this.wrapped (this.tokenizer )) For example, for the element-text field we have ElementTokenStream (a subclass of this class):

ElementTokenStream (QNameTokenFilter (LowerCaseFilter (StandardTokenizer)))

We can't follow the standard Lucene pattern of Analyzer as a factory for a TokenStream since we want to be able to extend any arbitrary textual Analyzer, but the constraints of the Analyzer class design prevent it from being extended in a straightforward manner. Thus we have essentially an outer (XML) stream wrapping an inner (Text) stream.

FIXME: make the constructor protected; allow construction only through static builders defined on each derived class. This will enable us to hide the complexity of wrapping the token stream, which is the same pattern for each of these; only the classes vary. But we can't do the work in the constructor due to Java structural issues.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
protected Reader	charStream
protected Iterator<net.sf.saxon.s9api.XdmNode>	contentIter
protected net.sf.saxon.s9api.XdmNode	curNode
protected ElementVisibility	defVis
protected HashMap<net.sf.saxon.s9api.QName,ElementVisibility>	eltVis
protected static net.sf.saxon.s9api.XdmSequenceIterator	EMPTY
protected org.apache.lucene.analysis.tokenattributes.CharTermAttribute	termAtt

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
ElementVisibility	getDefaultVisibility()
ElementVisibility	getElementVisibility(net.sf.saxon.s9api.QName qname)
org.apache.lucene.analysis.TokenStream	getWrappedTokenStream()
boolean	incrementToken()
protected boolean	incrementWrappedTokenStream()
void	reset()
void	reset(Reader reader)
void	setDefaultVisibility(ElementVisibility vis)
void	setElementVisibility(net.sf.saxon.s9api.QName qname, ElementVisibility vis) sets the visibility of elements with the given name
protected void	setWrappedTokenStream(org.apache.lucene.analysis.TokenStream wrapped)

Methods inherited from class org.apache.lucene.analysis.TokenStream

end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

curNode

protected net.sf.saxon.s9api.XdmNode curNode

contentIter

protected Iterator<net.sf.saxon.s9api.XdmNode> contentIter

termAtt

protected org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAtt

charStream

protected Reader charStream

defVis

protected ElementVisibility defVis

eltVis

protected HashMap<net.sf.saxon.s9api.QName,ElementVisibility> eltVis

EMPTY

protected static final net.sf.saxon.s9api.XdmSequenceIterator EMPTY

Method Detail

reset

public void reset()
           throws IOException

Overrides:

reset in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

reset

public void reset(Reader reader)
           throws IOException

Throws:

IOException

incrementToken

public boolean incrementToken()
                       throws IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

getWrappedTokenStream

public org.apache.lucene.analysis.TokenStream getWrappedTokenStream()

Returns:

the underlying stream of text tokens to which additional xml-related attributes are added by this.

setWrappedTokenStream

protected void setWrappedTokenStream(org.apache.lucene.analysis.TokenStream wrapped)

incrementWrappedTokenStream

protected boolean incrementWrappedTokenStream()
                                       throws IOException

Throws:

IOException

getElementVisibility

public ElementVisibility getElementVisibility(net.sf.saxon.s9api.QName qname)

Parameters:

qname - the name of an element as an s9api QName

Returns:

the explicitly-specified visibility of the element name, or null if the element has the default visibility.

setElementVisibility

public void setElementVisibility(net.sf.saxon.s9api.QName qname,
                        ElementVisibility vis)

sets the visibility of elements with the given name

Parameters:

qname - the name of an element as an s9api QName

vis - the visibility of the element's content from the perspective of containing elements. visibility.

getDefaultVisibility

public ElementVisibility getDefaultVisibility()

Returns:

the visibility of elements not explicitly specified using setElementVisibility. Always ElementVisibility.OPAQUE.

setDefaultVisibility

public void setDefaultVisibility(ElementVisibility vis)