org.apache.mahout.utils.nlp.collocations.llr

Class BloomTokenFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilter

All Implemented Interfaces:

Closeable, AutoCloseable

public final class BloomTokenFilter
extends org.apache.lucene.analysis.TokenFilter

Emits tokens based on bloom filter membership.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Constructor Summary

Constructors

Constructor and Description
BloomTokenFilter(org.apache.hadoop.util.bloom.Filter filter, boolean keepMembers, org.apache.lucene.analysis.TokenStream in)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	incrementToken()

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end, reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BloomTokenFilter

public BloomTokenFilter(org.apache.hadoop.util.bloom.Filter filter,
                boolean keepMembers,
                org.apache.lucene.analysis.TokenStream in)

Parameters:

filter - tokens will be checked for membership in this bloom filter

in - the tokenstream to read.

keepMembers - keep memoers of the bloom filter? If true works like a whitelist and members found in the list are kept and all others are dropped. If false works like a stoplist and members found in the filter are dropped all others are kept.

Method Detail

incrementToken

public boolean incrementToken()
                       throws IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

IOException