A B C D E F G H I M N O P R S T U V W X

A

abortJob(JobContext, JobStatus.State) - Method in class org.apache.hadoop.tools.mapred.CopyCommitter

ACLS_NOT_SUPPORTED - Static variable in class org.apache.hadoop.tools.DistCpConstants

AclsNotSupportedException(String) - Constructor for exception org.apache.hadoop.tools.CopyListing.AclsNotSupportedException

addToConf(Configuration, DistCpOptionSwitch, String) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Helper function to add an option to hadoop configuration object

addToConf(Configuration, DistCpOptionSwitch) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Helper function to set an option to hadoop configuration object

addWorker(WorkRequestProcessor<T, R>) - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Add another worker that will consume WorkRequest<T> items from input queue, process each item using supplied processor, and for every processed item output WorkReport<R> to output queue.

appendToConf(Configuration) - Method in class org.apache.hadoop.tools.DistCpContext

appendToConf(Configuration) - Method in class org.apache.hadoop.tools.DistCpOptions

Add options to configuration.

B

blockingTake() - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Blocking take from ProducerConsumer output queue (catches exceptions and retries forever).

build() - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

This is the single entry point for constructing DistCpOptions objects.

Builder(List<Path>, Path) - Constructor for class org.apache.hadoop.tools.DistCpOptions.Builder

Builder(Path, Path) - Constructor for class org.apache.hadoop.tools.DistCpOptions.Builder

buildListing(Path, DistCpContext) - Method in class org.apache.hadoop.tools.CopyListing

Build listing function creates the input listing that distcp uses to perform the copy.

C

checkFileSystemAclSupport(FileSystem) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Determines if a file system supports ACLs by running a canary getAclStatus request on the file system root.

checkFileSystemXAttrSupport(FileSystem) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Determines if a file system supports XAttrs by running a getXAttrs request on the file system root.

checkOutputSpecs(JobContext) - Method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

checksumsAreEqual(FileSystem, Path, FileChecksum, FileSystem, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Utility to compare checksums for the paths specified.

cleanup(Mapper<Text, CopyListingFileStatus, Text, Text>.Context) - Method in class org.apache.hadoop.tools.mapred.CopyMapper

close() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation of RecordReader::close().

close() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

commitJob(JobContext) - Method in class org.apache.hadoop.tools.mapred.CopyCommitter

compareFileLengthsAndChecksums(FileSystem, Path, FileChecksum, FileSystem, Path, boolean) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Utility to compare file lengths and checksums for source and target.

CONF_LABEL_APPEND - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_ATOMIC_COPY - Static variable in class org.apache.hadoop.tools.DistCpConstants

Constants mapping to command line switches/input options

CONF_LABEL_BANDWIDTH_MB - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_BLOCKS_PER_CHUNK - Static variable in class org.apache.hadoop.tools.DistCpConstants

DistCp Blocks Per Chunk: "distcp.blocks.per.chunk".

CONF_LABEL_COPY_BUFFER_SIZE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_COPY_LISTING_CLASS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_COPY_STRATEGY - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_DELETE_MISSING - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_DIFF - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_DIRECT_WRITE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_DISTCP_JOB_ID - Static variable in class org.apache.hadoop.tools.DistCpConstants

DistCp job id for consumers of the Disctp

CONF_LABEL_FILTERS_FILE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_IGNORE_FAILURES - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_LISTING_FILE_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_LISTSTATUS_THREADS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_LOG_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_MAX_CHUNKS_IDEAL - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_MAX_CHUNKS_TOLERABLE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_MAX_MAPS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_META_FOLDER - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_MIN_RECORDS_PER_CHUNK - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_OVERWRITE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_PRESERVE_RAWXATTRS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_PRESERVE_STATUS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_RDIFF - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SIMPLE_LISTING_FILESTATUS_SIZE - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SIMPLE_LISTING_RANDOMIZE_FILES - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SKIP_CRC - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SOURCE_LISTING - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SPLIT_RATIO - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_SYNC_FOLDERS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TARGET_FINAL_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TARGET_PATH_EXISTS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TARGET_WORK_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TOTAL_BYTES_TO_BE_COPIED - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TOTAL_NUMBER_OF_RECORDS - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_TRACK_MISSING - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_VERBOSE_LOG - Static variable in class org.apache.hadoop.tools.DistCpConstants

CONF_LABEL_WORK_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

COPY_BUFFER_SIZE_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants

CopyCommitter - Class in org.apache.hadoop.tools.mapred

The CopyCommitter class is DistCp's OutputCommitter implementation.

CopyCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.tools.mapred.CopyCommitter

Create a output committer

CopyFilter - Class in org.apache.hadoop.tools

Interface for excluding files from DistCp.

CopyFilter() - Constructor for class org.apache.hadoop.tools.CopyFilter

CopyListing - Class in org.apache.hadoop.tools

The CopyListing abstraction is responsible for how the list of sources and targets is constructed, for DistCp's copy function.

CopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.CopyListing

Protected constructor, to initialize configuration.

CopyListing.AclsNotSupportedException - Exception in org.apache.hadoop.tools

CopyListing.XAttrsNotSupportedException - Exception in org.apache.hadoop.tools

CopyListingFileStatus - Class in org.apache.hadoop.tools

CopyListingFileStatus is a view of FileStatus, recording additional data members useful to distcp.

CopyListingFileStatus() - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

Default constructor.

CopyListingFileStatus(FileStatus) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

Creates a new CopyListingFileStatus by copying the members of the given FileStatus.

CopyListingFileStatus(FileStatus, long, long) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

CopyListingFileStatus(long, boolean, int, long, long, long, FsPermission, String, String, Path) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

CopyListingFileStatus(long, boolean, int, long, long, long, FsPermission, String, String, Path, long, long) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

CopyListingFileStatus(CopyListingFileStatus) - Constructor for class org.apache.hadoop.tools.CopyListingFileStatus

CopyMapper - Class in org.apache.hadoop.tools.mapred

Mapper class that executes the DistCp copy operation.

CopyMapper() - Constructor for class org.apache.hadoop.tools.mapred.CopyMapper

CopyMapper.Counter - Enum in org.apache.hadoop.tools.mapred

Hadoop counters for the DistCp CopyMapper.

CopyOutputFormat<K,V> - Class in org.apache.hadoop.tools.mapred

The CopyOutputFormat is the Hadoop OutputFormat used in DistCp.

CopyOutputFormat() - Constructor for class org.apache.hadoop.tools.mapred.CopyOutputFormat

CopyReadException(Throwable) - Constructor for exception org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.CopyReadException

createAndSubmitJob() - Method in class org.apache.hadoop.tools.DistCp

Create and submit the mapreduce job.

createInputFileListing(Job) - Method in class org.apache.hadoop.tools.DistCp

Create input listing by invoking an appropriate copy listing implementation.

createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat

Implementation of Inputformat::createRecordReader().

createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.UniformSizeInputFormat

Implementation of InputFormat::createRecordReader().

D

DEFAULT_BANDWIDTH_MB - Static variable in class org.apache.hadoop.tools.DistCpConstants

DEFAULT_FILE_STATUS_SIZE - Static variable in class org.apache.hadoop.tools.SimpleCopyListing

DEFAULT_LISTSTATUS_THREADS - Static variable in class org.apache.hadoop.tools.DistCpConstants

DEFAULT_MAPS - Static variable in class org.apache.hadoop.tools.DistCpConstants

DEFAULT_RANDOMIZE_FILE_LISTING - Static variable in class org.apache.hadoop.tools.SimpleCopyListing

description - Variable in class org.apache.hadoop.tools.util.RetriableCommand

DistCp - Class in org.apache.hadoop.tools

DistCp is the main driver-class for DistCpV2.

DistCp(Configuration, DistCpOptions) - Constructor for class org.apache.hadoop.tools.DistCp

Public Constructor.

DistCpConstants - Class in org.apache.hadoop.tools

Utility class to hold commonly used constants.

DistCpContext - Class in org.apache.hadoop.tools

This is the context of the distcp at runtime.

DistCpContext(DistCpOptions) - Constructor for class org.apache.hadoop.tools.DistCpContext

DistCpOptions - Class in org.apache.hadoop.tools

The Options class encapsulates all DistCp options.

DistCpOptions.Builder - Class in org.apache.hadoop.tools

The builder of the DistCpOptions.

DistCpOptions.FileAttribute - Enum in org.apache.hadoop.tools

File attributes for preserve.

DistCpOptionSwitch - Enum in org.apache.hadoop.tools

Enumeration mapping configuration keys to distcp command line options.

DistCpUtils - Class in org.apache.hadoop.tools.util

Utility functions used in DistCp.

DistCpUtils() - Constructor for class org.apache.hadoop.tools.util.DistCpUtils

doBuildListing(Path, DistCpContext) - Method in class org.apache.hadoop.tools.CopyListing

The interface to be implemented by sub-classes, to create the source/target file listing.

doBuildListing(Path, DistCpContext) - Method in class org.apache.hadoop.tools.FileBasedCopyListing

Implementation of CopyListing::buildListing().

doBuildListing(Path, DistCpContext) - Method in class org.apache.hadoop.tools.GlobbedCopyListing

Implementation of CopyListing::buildListing().

doBuildListing(Path, DistCpContext) - Method in class org.apache.hadoop.tools.SimpleCopyListing

doBuildListing(SequenceFile.Writer, DistCpContext) - Method in class org.apache.hadoop.tools.SimpleCopyListing

Collect the list of <sourceRelativePath, sourceFileStatus> to be copied and write to the sequence file.

doBuildListingWithSnapshotDiff(SequenceFile.Writer, DistCpContext) - Method in class org.apache.hadoop.tools.SimpleCopyListing

Build a copy list based on the snapshot diff report.

doExecute(Object...) - Method in class org.apache.hadoop.tools.mapred.RetriableDirectoryCreateCommand

Implementation of RetriableCommand::doExecute().

doExecute(Object...) - Method in class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand

Implementation of RetriableCommand::doExecute().

doExecute(Object...) - Method in class org.apache.hadoop.tools.util.RetriableCommand

Implement this interface-method define the command-logic that will be retried on failure (i.e.

DUPLICATE_INPUT - Static variable in class org.apache.hadoop.tools.DistCpConstants

DynamicInputFormat<K,V> - Class in org.apache.hadoop.tools.mapred.lib

DynamicInputFormat implements the "Worker pattern" for DistCp.

DynamicInputFormat() - Constructor for class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat

DynamicRecordReader<K,V> - Class in org.apache.hadoop.tools.mapred.lib

The DynamicRecordReader is used in conjunction with the DynamicInputFormat to implement the "Worker pattern" for DistCp.

E

equals(Object) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

execute() - Method in class org.apache.hadoop.tools.DistCp

Implements the core-execution.

execute(Object...) - Method in class org.apache.hadoop.tools.util.RetriableCommand

The execute() method invokes doExecute() until either: 1.

F

FileBasedCopyListing - Class in org.apache.hadoop.tools

FileBasedCopyListing implements the CopyListing interface, to create the copy-listing for DistCp, by iterating over all source paths mentioned in a specified input-file.

FileBasedCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.FileBasedCopyListing

Constructor, to initialize base-class.

G

getAccessTime() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getAcl(FileSystem, FileStatus) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Returns a file's full logical ACL.

getAclEntries() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

Returns the full logical ACL.

getAtomicWorkPath() - Method in class org.apache.hadoop.tools.DistCpContext

getAtomicWorkPath() - Method in class org.apache.hadoop.tools.DistCpOptions

getAttribute(char) - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute

getBlockSize() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getBlocksPerChunk() - Method in class org.apache.hadoop.tools.DistCpContext

getBlocksPerChunk() - Method in class org.apache.hadoop.tools.DistCpOptions

getBytesPerSec() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

Getter for the read-rate from this stream, since creation.

getBytesToCopy() - Method in class org.apache.hadoop.tools.CopyListing

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

getBytesToCopy() - Method in class org.apache.hadoop.tools.FileBasedCopyListing

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

getBytesToCopy() - Method in class org.apache.hadoop.tools.GlobbedCopyListing

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

getBytesToCopy() - Method in class org.apache.hadoop.tools.SimpleCopyListing

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

getChunkContext(Configuration) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat

getChunkLength() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getChunkOffset() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getCommitDirectory(Job) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

Getter for the final commit-directory.

getConfigLabel() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Get Configuration label for the option

getContext() - Method in class org.apache.hadoop.tools.DistCp

Returns the context.

getCopyBufferSize() - Method in class org.apache.hadoop.tools.DistCpContext

getCopyBufferSize() - Method in class org.apache.hadoop.tools.DistCpOptions

getCopyFilter(Configuration) - Static method in class org.apache.hadoop.tools.CopyFilter

Public factory method which returns the appropriate implementation of CopyFilter.

getCopyListing(Configuration, Credentials, DistCpContext) - Static method in class org.apache.hadoop.tools.CopyListing

Public Factory method with which the appropriate CopyListing implementation may be retrieved.

getCopyStrategy() - Method in class org.apache.hadoop.tools.DistCpContext

getCopyStrategy() - Method in class org.apache.hadoop.tools.DistCpOptions

getCredentials() - Method in class org.apache.hadoop.tools.CopyListing

get credentials to update the delegation tokens for accessed FS objects

getCurrentKey() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation of RecordReader::getCurrentKey().

getCurrentValue() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation of RecordReader::getCurrentValue().

getException() - Method in class org.apache.hadoop.tools.util.WorkReport

getFileListingKey(Path, CopyListingFileStatus) - Method in class org.apache.hadoop.tools.CopyListing

Returns the key for an entry in the copy listing sequence file.

getFileListingPath() - Method in class org.apache.hadoop.tools.DistCp

Get default name of the copy listing file.

getFileListingValue(CopyListingFileStatus) - Method in class org.apache.hadoop.tools.CopyListing

Returns the value for an entry in the copy listing sequence file.

getFileSize(Path, Configuration) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Retrieves size of the file at the specified path.

getFiltersFile() - Method in class org.apache.hadoop.tools.DistCpContext

getFiltersFile() - Method in class org.apache.hadoop.tools.DistCpOptions

getFormatter() - Static method in class org.apache.hadoop.tools.util.DistCpUtils

getFromSnapshot() - Method in class org.apache.hadoop.tools.DistCpContext

getFromSnapshot() - Method in class org.apache.hadoop.tools.DistCpOptions

getGroup() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getInt(Configuration, String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Utility to retrieve a specified key from a Configuration.

getItem() - Method in class org.apache.hadoop.tools.util.WorkReport

getItem() - Method in class org.apache.hadoop.tools.util.WorkRequest

getLen() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getLogPath() - Method in class org.apache.hadoop.tools.DistCpContext

getLogPath() - Method in class org.apache.hadoop.tools.DistCpOptions

getLong(Configuration, String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Utility to retrieve a specified key from a Configuration.

getMapBandwidth() - Method in class org.apache.hadoop.tools.DistCpContext

getMapBandwidth() - Method in class org.apache.hadoop.tools.DistCpOptions

getMaxMaps() - Method in class org.apache.hadoop.tools.DistCpContext

getMaxMaps() - Method in class org.apache.hadoop.tools.DistCpOptions

getModificationTime() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getNumberOfPaths() - Method in class org.apache.hadoop.tools.CopyListing

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

getNumberOfPaths() - Method in class org.apache.hadoop.tools.FileBasedCopyListing

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

getNumberOfPaths() - Method in class org.apache.hadoop.tools.GlobbedCopyListing

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

getNumberOfPaths() - Method in class org.apache.hadoop.tools.SimpleCopyListing

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

getNumListstatusThreads() - Method in class org.apache.hadoop.tools.DistCpContext

getNumListstatusThreads() - Method in class org.apache.hadoop.tools.DistCpOptions

getOption() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Get CLI Option corresponding to the distcp option

getOutputCommitter(TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

getOwner() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getPath() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getPermission() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getPos() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

getPreserveAttributes() - Method in class org.apache.hadoop.tools.DistCpContext

getPreserveAttributes() - Method in class org.apache.hadoop.tools.DistCpOptions

getProgress() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation of RecordReader::getProgress().

getRelativePath(Path, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Gets relative path of child path with respect to a root path For ex.

getReplication() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getRetry() - Method in class org.apache.hadoop.tools.util.WorkReport

getRetry() - Method in class org.apache.hadoop.tools.util.WorkRequest

getSizeToCopy() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

getSourceFileListing() - Method in class org.apache.hadoop.tools.DistCpContext

getSourceFileListing() - Method in class org.apache.hadoop.tools.DistCpOptions

getSourcePaths() - Method in class org.apache.hadoop.tools.DistCpContext

getSourcePaths() - Method in class org.apache.hadoop.tools.DistCpOptions

getSplitChunkPath(Path, CopyListingFileStatus) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

getSplits(JobContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicInputFormat

Implementation of InputFormat::getSplits().

getSplits(JobContext) - Method in class org.apache.hadoop.tools.mapred.UniformSizeInputFormat

Implementation of InputFormat::getSplits().

getStrategy(Configuration, DistCpContext) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Returns the class that implements a copy strategy.

getStringDescriptionFor(long) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

getSuccess() - Method in class org.apache.hadoop.tools.util.WorkReport

getSwitch() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Get Switch symbol

getTargetPath() - Method in class org.apache.hadoop.tools.DistCpContext

getTargetPath() - Method in class org.apache.hadoop.tools.DistCpOptions

getToSnapshot() - Method in class org.apache.hadoop.tools.DistCpContext

getToSnapshot() - Method in class org.apache.hadoop.tools.DistCpOptions

getTotalBytesRead() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

Getter for the number of bytes read from this stream, since creation.

getTotalSleepTime() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

Getter the total time spent in sleep.

getTrackPath() - Method in class org.apache.hadoop.tools.DistCpOptions

getWorkCnt() - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Returns number of pending ProducerConsumer items (submitted to input queue for processing via put() method but not yet consumed by take() or blockingTake().

getWorkingDirectory(Job) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

Getter for the working directory.

getXAttrs() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

Returns all xAttrs.

getXAttrs(FileSystem, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Returns a file's all xAttrs.

GlobbedCopyListing - Class in org.apache.hadoop.tools

GlobbedCopyListing implements the CopyListing interface, to create the copy listing-file by "globbing" all specified source paths (wild-cards and all.)

GlobbedCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.GlobbedCopyListing

Constructor, to initialize the configuration.

H

hashCode() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

hasWork() - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Returns true if there are items in ProducerConsumer that are either pending for processing or waiting to be consumed.

HDFS_RESERVED_RAW_DIRECTORY_NAME - Static variable in class org.apache.hadoop.tools.DistCpConstants

Value of reserved raw HDFS directory when copying raw.* xattrs.

I

initialize() - Method in class org.apache.hadoop.tools.CopyFilter

Default initialize method does nothing.

initialize(InputSplit, TaskAttemptContext) - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation for RecordReader::initialize().

initialize() - Method in class org.apache.hadoop.tools.RegexCopyFilter

Loads a list of filter patterns for use in shouldCopy.

INVALID_ARGUMENT - Static variable in class org.apache.hadoop.tools.DistCpConstants

isDirectory() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

isErasureCoded() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

isSplit() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

isTargetPathExists() - Method in class org.apache.hadoop.tools.DistCpContext

M

main(String[]) - Static method in class org.apache.hadoop.tools.DistCp

Main function of the DistCp program.

map(Text, CopyListingFileStatus, Mapper<Text, CopyListingFileStatus, Text, Text>.Context) - Method in class org.apache.hadoop.tools.mapred.CopyMapper

Implementation of the Mapper::map().

MAX_CHUNKS_IDEAL_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants

MAX_CHUNKS_TOLERABLE_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants

Constants for DistCp default values of configurable values

MAX_NUM_LISTSTATUS_THREADS - Static variable in class org.apache.hadoop.tools.DistCpOptions

maxMaps(int) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

MIN_RECORDS_PER_CHUNK_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants

N

nextKeyValue() - Method in class org.apache.hadoop.tools.mapred.lib.DynamicRecordReader

Implementation of RecordReader::nextValue().

NONE_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

NONE_PATH_NAME - Static variable in class org.apache.hadoop.tools.DistCpConstants

Constants for NONE file deletion

O

OptionsParser - Class in org.apache.hadoop.tools

The OptionsParser parses out the command-line options passed to DistCp, and interprets those specific to DistCp, to create an Options object.

OptionsParser() - Constructor for class org.apache.hadoop.tools.OptionsParser

org.apache.hadoop.tools - package org.apache.hadoop.tools

DistCp is a tool for replicating data using MapReduce jobs for concurrent copy operations.

org.apache.hadoop.tools.mapred - package org.apache.hadoop.tools.mapred

org.apache.hadoop.tools.mapred.lib - package org.apache.hadoop.tools.mapred.lib

org.apache.hadoop.tools.util - package org.apache.hadoop.tools.util

P

packAttributes(EnumSet<DistCpOptions.FileAttribute>) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Pack file preservation attributes into a string, containing just the first character of each preservation attribute

parse(String[]) - Static method in class org.apache.hadoop.tools.OptionsParser

The parse method parses the command-line options, and creates a corresponding Options object.

preserve(String) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

preserve(DistCpOptions.FileAttribute) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

preserve(FileSystem, Path, CopyListingFileStatus, EnumSet<DistCpOptions.FileAttribute>, boolean) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Preserve attribute on file matching that of the file status being sent as argument.

PRESERVE_STATUS_DEFAULT - Static variable in enum org.apache.hadoop.tools.DistCpOptionSwitch

processItem(WorkRequest<T>) - Method in interface org.apache.hadoop.tools.util.WorkRequestProcessor

Work processor.

ProducerConsumer<T,R> - Class in org.apache.hadoop.tools.util

ProducerConsumer class encapsulates input and output queues and a thread-pool of Workers that loop on WorkRequest<T> inputQueue and for each consumed WorkRequest Workers invoke WorkRequestProcessor.processItem() and output resulting WorkReport<R> to the outputQueue.

ProducerConsumer(int) - Constructor for class org.apache.hadoop.tools.util.ProducerConsumer

ProducerConsumer maintains input and output queues and a thread-pool of workers.

publish(Configuration, String, T) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Utility to publish a value to a configuration.

put(WorkRequest<T>) - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Blocking put workRequest to ProducerConsumer input queue.

R

RAW_NONE_PATH - Static variable in class org.apache.hadoop.tools.DistCpConstants

read() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

read(byte[]) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

read(byte[], int, int) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

readFields(DataInput) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

RegexCopyFilter - Class in org.apache.hadoop.tools

A CopyFilter which compares Java Regex Patterns to each Path to determine whether a file should be copied.

RegexCopyFilter(String) - Constructor for class org.apache.hadoop.tools.RegexCopyFilter

Constructor, sets up a File object to read filter patterns from and the List to store the patterns.

RetriableCommand - Class in org.apache.hadoop.tools.util

This class represents commands that be retried on failure, in a configurable manner.

RetriableCommand(String) - Constructor for class org.apache.hadoop.tools.util.RetriableCommand

Constructor.

RetriableCommand(String, RetryPolicy) - Constructor for class org.apache.hadoop.tools.util.RetriableCommand

Constructor.

RetriableDirectoryCreateCommand - Class in org.apache.hadoop.tools.mapred

This class extends Retriable command to implement the creation of directories with retries on failure.

RetriableDirectoryCreateCommand(String) - Constructor for class org.apache.hadoop.tools.mapred.RetriableDirectoryCreateCommand

Constructor, taking a description of the action.

RetriableFileCopyCommand - Class in org.apache.hadoop.tools.mapred

This class extends RetriableCommand to implement the copy of files, with retries on failure.

RetriableFileCopyCommand(String, CopyMapper.FileAction) - Constructor for class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand

Constructor, taking a description of the action.

RetriableFileCopyCommand(boolean, String, CopyMapper.FileAction) - Constructor for class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand

Create a RetriableFileCopyCommand.

RetriableFileCopyCommand(boolean, String, CopyMapper.FileAction, boolean) - Constructor for class org.apache.hadoop.tools.mapred.RetriableFileCopyCommand

Create a RetriableFileCopyCommand.

RetriableFileCopyCommand.CopyReadException - Exception in org.apache.hadoop.tools.mapred

Special subclass of IOException.

run(String[]) - Method in class org.apache.hadoop.tools.DistCp

Implementation of Tool::run().

S

seek(long) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

seekToNewSource(long) - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

setAclEntries(List<AclEntry>) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

Sets optional ACL entries.

setChunkLength(long) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

setChunkOffset(long) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

setCommitDirectory(Job, Path) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

Setter for the final directory for DistCp (where files copied will be moved, atomically.)

setCredentials(Credentials) - Method in class org.apache.hadoop.tools.CopyListing

set Credentials store, on which FS delegatin token will be cached

setFilters(List<Pattern>) - Method in class org.apache.hadoop.tools.RegexCopyFilter

Sets the list of filters to exclude files from copy.

setPreserveRawXattrs(boolean) - Method in class org.apache.hadoop.tools.DistCpContext

setRetryPolicy(RetryPolicy) - Method in class org.apache.hadoop.tools.util.RetriableCommand

Fluent-interface to change the RetryHandler.

setSourcePaths(List<Path>) - Method in class org.apache.hadoop.tools.DistCpContext

setTargetPathExists(boolean) - Method in class org.apache.hadoop.tools.DistCpContext

setup(Mapper<Text, CopyListingFileStatus, Text, Text>.Context) - Method in class org.apache.hadoop.tools.mapred.CopyMapper

Implementation of the Mapper::setup() method.

setWorkingDirectory(Job, Path) - Static method in class org.apache.hadoop.tools.mapred.CopyOutputFormat

Setter for the working directory for DistCp (where files will be copied before they are moved to the final commit-directory.)

setXAttrs(Map<String, byte[]>) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

Sets optional xAttrs.

shouldAppend() - Method in class org.apache.hadoop.tools.DistCpContext

shouldAppend() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldAtomicCommit() - Method in class org.apache.hadoop.tools.DistCpContext

shouldAtomicCommit() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldBlock() - Method in class org.apache.hadoop.tools.DistCpContext

shouldBlock() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldCopy(Path) - Method in class org.apache.hadoop.tools.CopyFilter

Predicate to determine if a file can be excluded from copy.

shouldCopy(Path) - Method in class org.apache.hadoop.tools.RegexCopyFilter

shouldCopy(Path) - Method in class org.apache.hadoop.tools.SimpleCopyListing

Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME

shouldCopy(Path) - Method in class org.apache.hadoop.tools.TrueCopyFilter

shouldDeleteMissing() - Method in class org.apache.hadoop.tools.DistCpContext

shouldDeleteMissing() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldDirectWrite() - Method in class org.apache.hadoop.tools.DistCpContext

shouldDirectWrite() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldIgnoreFailures() - Method in class org.apache.hadoop.tools.DistCpContext

shouldIgnoreFailures() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldOverwrite() - Method in class org.apache.hadoop.tools.DistCpContext

shouldOverwrite() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldPreserve(DistCpOptions.FileAttribute) - Method in class org.apache.hadoop.tools.DistCpContext

shouldPreserve(DistCpOptions.FileAttribute) - Method in class org.apache.hadoop.tools.DistCpOptions

Checks if the input attribute should be preserved or not.

shouldPreserveRawXattrs() - Method in class org.apache.hadoop.tools.DistCpContext

shouldSkipCRC() - Method in class org.apache.hadoop.tools.DistCpContext

shouldSkipCRC() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldSyncFolder() - Method in class org.apache.hadoop.tools.DistCpContext

shouldSyncFolder() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldUseDiff() - Method in class org.apache.hadoop.tools.DistCpContext

shouldUseDiff() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldUseRdiff() - Method in class org.apache.hadoop.tools.DistCpContext

shouldUseRdiff() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldUseSnapshotDiff() - Method in class org.apache.hadoop.tools.DistCpContext

shouldUseSnapshotDiff() - Method in class org.apache.hadoop.tools.DistCpOptions

shouldVerboseLog() - Method in class org.apache.hadoop.tools.DistCpOptions

shutdown() - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Shutdown ProducerConsumer worker thread-pool without waiting for completion of any pending work.

SimpleCopyListing - Class in org.apache.hadoop.tools

The SimpleCopyListing is responsible for making the exhaustive list of all files/directories under its specified list of input-paths.

SimpleCopyListing(Configuration, Credentials) - Constructor for class org.apache.hadoop.tools.SimpleCopyListing

Protected constructor, to initialize configuration.

SimpleCopyListing(Configuration, Credentials, int, int, boolean) - Constructor for class org.apache.hadoop.tools.SimpleCopyListing

SimpleCopyListing(Configuration, Credentials, DistCpSync) - Constructor for class org.apache.hadoop.tools.SimpleCopyListing

sortListing(Configuration, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Sort sequence file containing FileStatus and Text as key and value respectively.

sortListing(Configuration, Path, Path) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Sort sequence file containing FileStatus and Text as key and value respectively, saving the result to the output path, which will be deleted first.

SOURCE_SORTED_FILE - Static variable in class org.apache.hadoop.tools.DistCpConstants

Filename of sorted files in when tracking saves them.

SPLIT_RATIO_DEFAULT - Static variable in class org.apache.hadoop.tools.DistCpConstants

splitLargeFile() - Method in class org.apache.hadoop.tools.DistCpContext

SUCCESS - Static variable in class org.apache.hadoop.tools.DistCpConstants

Constants for DistCp return code to shell / consumer of ToolRunner's run

T

take() - Method in class org.apache.hadoop.tools.util.ProducerConsumer

Blocking take from ProducerConsumer output queue that can be interrupted.

TARGET_LISTING_FILE - Static variable in class org.apache.hadoop.tools.DistCpConstants

Filename of unsorted target listing.

TARGET_SORTED_FILE - Static variable in class org.apache.hadoop.tools.DistCpConstants

Filename of sorted target listing.

ThrottledInputStream - Class in org.apache.hadoop.tools.util

The ThrottleInputStream provides bandwidth throttling on a specified InputStream.

ThrottledInputStream(InputStream) - Constructor for class org.apache.hadoop.tools.util.ThrottledInputStream

ThrottledInputStream(InputStream, float) - Constructor for class org.apache.hadoop.tools.util.ThrottledInputStream

toCopyListingFileStatus(FileSystem, FileStatus, boolean, boolean, boolean, int) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Converts FileStatus to a list of CopyListingFileStatus.

toCopyListingFileStatusHelper(FileSystem, FileStatus, boolean, boolean, boolean, long, long) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Converts a FileStatus to a CopyListingFileStatus.

toString() - Method in class org.apache.hadoop.tools.CopyListingFileStatus

toString() - Method in class org.apache.hadoop.tools.DistCpContext

toString() - Method in class org.apache.hadoop.tools.DistCpOptions

Utility to easily string-ify Options, for logging.

toString() - Method in enum org.apache.hadoop.tools.DistCpOptionSwitch

toString() - Method in class org.apache.hadoop.tools.util.ThrottledInputStream

TrueCopyFilter - Class in org.apache.hadoop.tools

A CopyFilter which always returns true.

TrueCopyFilter() - Constructor for class org.apache.hadoop.tools.TrueCopyFilter

U

UNIFORMSIZE - Static variable in class org.apache.hadoop.tools.DistCpConstants

UniformSizeInputFormat - Class in org.apache.hadoop.tools.mapred

UniformSizeInputFormat extends the InputFormat class, to produce input-splits for DistCp.

UniformSizeInputFormat() - Constructor for class org.apache.hadoop.tools.mapred.UniformSizeInputFormat

UNKNOWN_ERROR - Static variable in class org.apache.hadoop.tools.DistCpConstants

unpackAttributes(String) - Static method in class org.apache.hadoop.tools.util.DistCpUtils

Unpacks preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve

usage() - Static method in class org.apache.hadoop.tools.OptionsParser

V

validatePaths(DistCpContext) - Method in class org.apache.hadoop.tools.CopyListing

Validate input and output paths

validatePaths(DistCpContext) - Method in class org.apache.hadoop.tools.FileBasedCopyListing

Validate input and output paths

validatePaths(DistCpContext) - Method in class org.apache.hadoop.tools.GlobbedCopyListing

Validate input and output paths

validatePaths(DistCpContext) - Method in class org.apache.hadoop.tools.SimpleCopyListing

valueOf(String) - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum org.apache.hadoop.tools.mapred.CopyMapper.Counter

Returns the enum constant of this type with the specified name.

values() - Static method in enum org.apache.hadoop.tools.DistCpOptions.FileAttribute

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum org.apache.hadoop.tools.DistCpOptionSwitch

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum org.apache.hadoop.tools.mapred.CopyMapper.Counter

Returns an array containing the constants of this enum type, in the order they are declared.

W

waitForJobCompletion(Job) - Method in class org.apache.hadoop.tools.DistCp

Wait for the given job to complete.

withAppend(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withAtomicCommit(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withAtomicWorkPath(Path) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withBlocking(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withBlocksPerChunk(int) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withCopyBufferSize(int) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withCopyStrategy(String) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withCRC(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withDeleteMissing(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withDirectWrite(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withFiltersFile(String) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withIgnoreFailures(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withLogPath(Path) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withMapBandwidth(float) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withNumListstatusThreads(int) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withOverwrite(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withSyncFolder(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withTrackMissing(Path) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withUseDiff(String, String) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withUseRdiff(String, String) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

withVerboseLog(boolean) - Method in class org.apache.hadoop.tools.DistCpOptions.Builder

WorkReport<T> - Class in org.apache.hadoop.tools.util

WorkReport<T> is a simple container for items of class T and its corresponding retry counter that indicates how many times this item was previously attempted to be processed.

WorkReport(T, int, boolean) - Constructor for class org.apache.hadoop.tools.util.WorkReport

WorkReport(T, int, boolean, Exception) - Constructor for class org.apache.hadoop.tools.util.WorkReport

WorkRequest<T> - Class in org.apache.hadoop.tools.util

WorkRequest<T> is a simple container for items of class T and its corresponding retry counter that indicates how many times this item was previously attempted to be processed.

WorkRequest(T) - Constructor for class org.apache.hadoop.tools.util.WorkRequest

WorkRequest(T, int) - Constructor for class org.apache.hadoop.tools.util.WorkRequest

WorkRequestProcessor<T,R> - Interface in org.apache.hadoop.tools.util

Interface for ProducerConsumer worker loop.

write(DataOutput) - Method in class org.apache.hadoop.tools.CopyListingFileStatus

X

XATTRS_NOT_SUPPORTED - Static variable in class org.apache.hadoop.tools.DistCpConstants

XAttrsNotSupportedException(String) - Constructor for exception org.apache.hadoop.tools.CopyListing.XAttrsNotSupportedException

A B C D E F G H I M N O P R S T U V W X