org.calrissian.accumulorecipes.featurestore.hadoop

Class FeaturesInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.accumulo.core.client.mapreduce.AbstractInputFormat<K,V>

org.apache.accumulo.core.client.mapreduce.InputFormatBase<org.apache.accumulo.core.data.Key,Feature>

org.calrissian.accumulorecipes.featurestore.hadoop.FeaturesInputFormat

public class FeaturesInputFormat
extends org.apache.accumulo.core.client.mapreduce.InputFormatBase<org.apache.accumulo.core.data.Key,Feature>

A Hadoop InputFormat that allows any Feature to be streamed into a map/reduce job based on a given query.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.accumulo.core.client.mapreduce.InputFormatBase

org.apache.accumulo.core.client.mapreduce.InputFormatBase.RangeInputSplit, org.apache.accumulo.core.client.mapreduce.InputFormatBase.RecordReaderBase<K,V>

Nested classes/interfaces inherited from class org.apache.accumulo.core.client.mapreduce.AbstractInputFormat

org.apache.accumulo.core.client.mapreduce.AbstractInputFormat.AbstractRecordReader<K,V>

Field Summary

Fields inherited from class org.apache.accumulo.core.client.mapreduce.AbstractInputFormat

CLASS, log

Constructor Summary

Constructors

Constructor and Description
FeaturesInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.accumulo.core.data.Key,Feature>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
static void	setInputInfo(org.apache.hadoop.mapreduce.Job job, String username, byte[] password, org.apache.accumulo.core.security.Authorizations auths)
static void	setQueryInfo(org.apache.hadoop.mapreduce.Job job, Date start, Date end, TimeUnit timeUnit, String group, String type, String name, Class<? extends Feature> featureType)
static void	setQueryInfo(org.apache.hadoop.mapreduce.Job job, Date start, Date end, TimeUnit timeUnit, String group, String type, String name, Class<? extends Feature> featureType, FeatureRegistry registry) Query for a specific set of feature rollups for a specific time range and unit of time.

Methods inherited from class org.apache.accumulo.core.client.mapreduce.InputFormatBase

addIterator, fetchColumns, getAutoAdjustRanges, getFetchedColumns, getInputTableName, getIterators, getRanges, getTabletLocator, isIsolated, isOfflineScan, setAutoAdjustRanges, setInputTableName, setLocalIterators, setOfflineTableScan, setRanges, setScanIsolation, usesLocalIterators

Methods inherited from class org.apache.accumulo.core.client.mapreduce.AbstractInputFormat

getAuthenticationToken, getInputTableConfig, getInputTableConfigs, getInstance, getLogLevel, getPrincipal, getScanAuthorizations, getSplits, getTabletLocator, getToken, getTokenClass, isConnectorInfoSet, setConnectorInfo, setConnectorInfo, setLogLevel, setMockInstance, setScanAuthorizations, setZooKeeperInstance, setZooKeeperInstance, validateOptions

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FeaturesInputFormat

public FeaturesInputFormat()

Method Detail

setInputInfo

public static void setInputInfo(org.apache.hadoop.mapreduce.Job job,
                String username,
                byte[] password,
                org.apache.accumulo.core.security.Authorizations auths)
                         throws org.apache.accumulo.core.client.AccumuloSecurityException

Throws:

org.apache.accumulo.core.client.AccumuloSecurityException

setQueryInfo

public static void setQueryInfo(org.apache.hadoop.mapreduce.Job job,
                Date start,
                Date end,
                TimeUnit timeUnit,
                String group,
                String type,
                String name,
                Class<? extends Feature> featureType)
                         throws IOException

Throws:

IOException

setQueryInfo

public static void setQueryInfo(org.apache.hadoop.mapreduce.Job job,
                Date start,
                Date end,
                TimeUnit timeUnit,
                String group,
                String type,
                String name,
                Class<? extends Feature> featureType,
                FeatureRegistry registry)
                         throws IOException

Query for a specific set of feature rollups for a specific time range and unit of time. At a minimum, a group needs to be specified. Type and name are optional. The requested feature type to query determines the vector that will be streamed into the map/reduce job. A registry allows pluggable feature types to be propagated down to a store where features have been ingested with matching feature configs. NOTE: It can be dangerous to apply a registry to a feature

Throws:

IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.accumulo.core.data.Key,Feature> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                     org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                       throws IOException,
                                                                                                              InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.accumulo.core.data.Key,Feature>

Throws:

IOException

InterruptedException