org.apache.hadoop.hbase.mapreduce

Class HFileOutputFormat2

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2

Direct Known Subclasses:

MultiTableHFileOutputFormat

@InterfaceAudience.Public
public class HFileOutputFormat2
extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

Writes HFiles. Passed Cells must arrive in order. Writes current time as the sequence id for the file. Sets the major compacted attribute on created @{link HFiles. Calling write(null,null) will forcibly roll all HFiles being written.

Using this class as part of a MapReduce job is best done using configureIncrementalLoad(Job, TableDescriptor, RegionLocator).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATABLOCK_ENCODING_OVERRIDE_CONF_KEY
static String	LOCALITY_SENSITIVE_CONF_KEY Keep locality while generating HFiles for bulkload.
static String	STORAGE_POLICY_PROPERTY
static String	STORAGE_POLICY_PROPERTY_CF_PREFIX
protected static byte[]	tableSeparator

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

BASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART

Constructor Summary

Constructors

Constructor and Description
HFileOutputFormat2()

Method Summary

Modifier and Type	Method and Description
protected static byte[]	combineTableNameSuffix(byte[] tableName, byte[] suffix)
static void	configureIncrementalLoad(org.apache.hadoop.mapreduce.Job job, org.apache.hadoop.hbase.client.TableDescriptor tableDescriptor, org.apache.hadoop.hbase.client.RegionLocator regionLocator) Configure a MapReduce Job to perform an incremental load into the given table.
static void	configureIncrementalLoad(org.apache.hadoop.mapreduce.Job job, org.apache.hadoop.hbase.client.Table table, org.apache.hadoop.hbase.client.RegionLocator regionLocator) Configure a MapReduce Job to perform an incremental load into the given table.
static void	configureIncrementalLoadMap(org.apache.hadoop.mapreduce.Job job, org.apache.hadoop.hbase.client.TableDescriptor tableDescriptor)
org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected static byte[]	getTableNameSuffixedWithFamily(byte[] tableName, byte[] family)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tableSeparator

protected static final byte[] tableSeparator

DATABLOCK_ENCODING_OVERRIDE_CONF_KEY

public static final String DATABLOCK_ENCODING_OVERRIDE_CONF_KEY

See Also:

Constant Field Values

LOCALITY_SENSITIVE_CONF_KEY

public static final String LOCALITY_SENSITIVE_CONF_KEY

Keep locality while generating HFiles for bulkload. See HBASE-12596

See Also:

Constant Field Values

STORAGE_POLICY_PROPERTY

public static final String STORAGE_POLICY_PROPERTY

See Also:

Constant Field Values

STORAGE_POLICY_PROPERTY_CF_PREFIX

public static final String STORAGE_POLICY_PROPERTY_CF_PREFIX

See Also:

Constant Field Values

Constructor Detail

HFileOutputFormat2

public HFileOutputFormat2()

Method Detail

combineTableNameSuffix

protected static byte[] combineTableNameSuffix(byte[] tableName,
                                               byte[] suffix)

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                                                         throws IOException,
                                                                                                                                                InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

Throws:

IOException

InterruptedException

getTableNameSuffixedWithFamily

protected static byte[] getTableNameSuffixedWithFamily(byte[] tableName,
                                                       byte[] family)

configureIncrementalLoad

public static void configureIncrementalLoad(org.apache.hadoop.mapreduce.Job job,
                                            org.apache.hadoop.hbase.client.Table table,
                                            org.apache.hadoop.hbase.client.RegionLocator regionLocator)
                                     throws IOException

Configure a MapReduce Job to perform an incremental load into the given table. This

• Inspects the table to configure a total order partitioner
• Uploads the partitions file to the cluster and adds it to the DistributedCache
• Sets the number of reduce tasks to match the current number of regions
• Sets the output key/value class to match HFileOutputFormat2's requirements
• Sets the reducer up to perform the appropriate sorting (either KeyValueSortReducer or PutSortReducer)

The user should be sure to set the map output value class to either KeyValue or Put before running this function.

Throws:

IOException

configureIncrementalLoad

public static void configureIncrementalLoad(org.apache.hadoop.mapreduce.Job job,
                                            org.apache.hadoop.hbase.client.TableDescriptor tableDescriptor,
                                            org.apache.hadoop.hbase.client.RegionLocator regionLocator)
                                     throws IOException

Configure a MapReduce Job to perform an incremental load into the given table. This

• Inspects the table to configure a total order partitioner
• Uploads the partitions file to the cluster and adds it to the DistributedCache
• Sets the number of reduce tasks to match the current number of regions
• Sets the output key/value class to match HFileOutputFormat2's requirements
• Sets the reducer up to perform the appropriate sorting (either KeyValueSortReducer or PutSortReducer)

The user should be sure to set the map output value class to either KeyValue or Put before running this function.

Throws:

IOException

configureIncrementalLoadMap

public static void configureIncrementalLoadMap(org.apache.hadoop.mapreduce.Job job,
                                               org.apache.hadoop.hbase.client.TableDescriptor tableDescriptor)
                                        throws IOException

Throws:

IOException