org.apache.hadoop.tools

Enum DistCpOptionSwitch

java.lang.Object

java.lang.Enum<DistCpOptionSwitch>

org.apache.hadoop.tools.DistCpOptionSwitch

All Implemented Interfaces:

Serializable, Comparable<DistCpOptionSwitch>

public enum DistCpOptionSwitch
extends Enum<DistCpOptionSwitch>

Enumeration mapping configuration keys to distcp command line options.

Enum Constant Summary

Enum Constants

Enum Constant and Description
APPEND
ATOMIC_COMMIT Copy all the source files and commit them atomically to the target This is typically useful in cases where there is a process polling for availability of a file/dir.
BANDWIDTH Specify bandwidth per map in MB, accepts bandwidth as a fraction
BLOCKING Should DisctpExecution be blocking
BLOCKS_PER_CHUNK
COPY_BUFFER_SIZE Configurable copy buffer size.
COPY_STRATEGY Copy strategy is use.
DELETE_MISSING Deletes missing files in target that are missing from source.
DIFF
DIRECT_WRITE Write directly to the final location, avoiding the creation and rename of temporary files.
FILE_LIMIT
FILTERS Path containing a list of strings, which when found in the path of a file to be copied excludes that file from the copy job.
IGNORE_FAILURES Ignores any failures during copy, and continues with rest.
LOG_PATH Log path where distcp output logs are written to
MAX_MAPS Max number of maps to use during copy.
NUM_LISTSTATUS_THREADS Number of threads for building source file listing (before map-reduce phase, max one listStatus per thread at a time).
OVERWRITE Overwrite target-files unconditionally.
PRESERVE_STATUS Preserves status of file/path in the target.
RDIFF
SIZE_LIMIT
SKIP_CRC Skip CRC checks between source and target, when determining what files need to be copied.
SOURCE_FILE_LISTING Source file listing can be provided to DistCp in a file.
SYNC_FOLDERS Update target location by copying only files that are missing in the target.
TRACK_MISSING Track missing files in target that are missing from source This allows for other applications to complete the synchronization, possibly with object-store-specific delete algorithms.
VERBOSE_LOG Log additional info (path, size) in the SKIP/COPY log.
WORK_PATH Work path to be used only in conjunction in Atomic commit

Field Summary

Fields

Modifier and Type	Field and Description
static String	PRESERVE_STATUS_DEFAULT

Method Summary

Modifier and Type	Method and Description
static void	addToConf(org.apache.hadoop.conf.Configuration conf, DistCpOptionSwitch option) Helper function to set an option to hadoop configuration object
static void	addToConf(org.apache.hadoop.conf.Configuration conf, DistCpOptionSwitch option, String value) Helper function to add an option to hadoop configuration object
String	getConfigLabel() Get Configuration label for the option
org.apache.commons.cli.Option	getOption() Get CLI Option corresponding to the distcp option
String	getSwitch() Get Switch symbol
String	toString()
static DistCpOptionSwitch	valueOf(String name) Returns the enum constant of this type with the specified name.
static DistCpOptionSwitch[]	values() Returns an array containing the constants of this enum type, in the order they are declared.

Methods inherited from class java.lang.Enum

clone, compareTo, equals, finalize, getDeclaringClass, hashCode, name, ordinal, valueOf

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Enum Constant Detail

IGNORE_FAILURES

public static final DistCpOptionSwitch IGNORE_FAILURES

Ignores any failures during copy, and continues with rest. Logs failures in a file

PRESERVE_STATUS

public static final DistCpOptionSwitch PRESERVE_STATUS

Preserves status of file/path in the target. Default behavior with -p, is to preserve replication, block size, user, group, permission, checksum type and timestamps on the target file. Note that when preserving checksum type, block size is also preserved. If any of the optional switches are present among rbugpcaxt, then only the corresponding file attribute is preserved.

SYNC_FOLDERS

public static final DistCpOptionSwitch SYNC_FOLDERS

Update target location by copying only files that are missing in the target. This can be used to periodically sync two folders across source and target. Typically used with DELETE_MISSING Incompatible with ATOMIC_COMMIT

DELETE_MISSING

public static final DistCpOptionSwitch DELETE_MISSING

Deletes missing files in target that are missing from source. This allows the target to be in sync with the source contents Typically used in conjunction with SYNC_FOLDERS Incompatible with ATOMIC_COMMIT

TRACK_MISSING

@InterfaceStability.Unstable
public static final DistCpOptionSwitch TRACK_MISSING

Track missing files in target that are missing from source This allows for other applications to complete the synchronization, possibly with object-store-specific delete algorithms. Typically used in conjunction with SYNC_FOLDERS Incompatible with ATOMIC_COMMIT

NUM_LISTSTATUS_THREADS

public static final DistCpOptionSwitch NUM_LISTSTATUS_THREADS

Number of threads for building source file listing (before map-reduce phase, max one listStatus per thread at a time).

MAX_MAPS

public static final DistCpOptionSwitch MAX_MAPS

Max number of maps to use during copy. DistCp will split work as equally as possible among these maps

SOURCE_FILE_LISTING

public static final DistCpOptionSwitch SOURCE_FILE_LISTING

Source file listing can be provided to DistCp in a file. This allows DistCp to copy random list of files from source and copy them to target

ATOMIC_COMMIT

public static final DistCpOptionSwitch ATOMIC_COMMIT

Copy all the source files and commit them atomically to the target This is typically useful in cases where there is a process polling for availability of a file/dir. This option is incompatible with SYNC_FOLDERS and DELETE_MISSING

WORK_PATH

public static final DistCpOptionSwitch WORK_PATH

Work path to be used only in conjunction in Atomic commit

LOG_PATH

public static final DistCpOptionSwitch LOG_PATH

Log path where distcp output logs are written to

VERBOSE_LOG

public static final DistCpOptionSwitch VERBOSE_LOG

Log additional info (path, size) in the SKIP/COPY log.

COPY_STRATEGY

public static final DistCpOptionSwitch COPY_STRATEGY

Copy strategy is use. This could be dynamic or uniform size etc. DistCp would use an appropriate input format based on this.

SKIP_CRC

public static final DistCpOptionSwitch SKIP_CRC

Skip CRC checks between source and target, when determining what files need to be copied.

OVERWRITE

public static final DistCpOptionSwitch OVERWRITE

Overwrite target-files unconditionally.

APPEND

public static final DistCpOptionSwitch APPEND

DIFF

public static final DistCpOptionSwitch DIFF

RDIFF

public static final DistCpOptionSwitch RDIFF

BLOCKING

public static final DistCpOptionSwitch BLOCKING

Should DisctpExecution be blocking

FILE_LIMIT

public static final DistCpOptionSwitch FILE_LIMIT

SIZE_LIMIT

public static final DistCpOptionSwitch SIZE_LIMIT

BLOCKS_PER_CHUNK

public static final DistCpOptionSwitch BLOCKS_PER_CHUNK

COPY_BUFFER_SIZE

public static final DistCpOptionSwitch COPY_BUFFER_SIZE

Configurable copy buffer size.

BANDWIDTH

public static final DistCpOptionSwitch BANDWIDTH

Specify bandwidth per map in MB, accepts bandwidth as a fraction

FILTERS

public static final DistCpOptionSwitch FILTERS

Path containing a list of strings, which when found in the path of a file to be copied excludes that file from the copy job.

DIRECT_WRITE

public static final DistCpOptionSwitch DIRECT_WRITE

Write directly to the final location, avoiding the creation and rename of temporary files. This is typically useful in cases where the target filesystem implementation does not support atomic rename operations, such as with the S3AFileSystem which translates file renames to potentially very expensive copy-then-delete operations.

Field Detail

PRESERVE_STATUS_DEFAULT

public static final String PRESERVE_STATUS_DEFAULT

See Also:

Constant Field Values

Method Detail

values

public static DistCpOptionSwitch[] values()

Returns an array containing the constants of this enum type, in the order they are declared. This method may be used to iterate over the constants as follows:

for (DistCpOptionSwitch c : DistCpOptionSwitch.values())
    System.out.println(c);

Returns:

an array containing the constants of this enum type, in the order they are declared

valueOf

public static DistCpOptionSwitch valueOf(String name)

Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)

Parameters:

name - the name of the enum constant to be returned.

Returns:

the enum constant with the specified name

Throws:

IllegalArgumentException - if this enum type has no constant with the specified name

NullPointerException - if the argument is null

getConfigLabel

public String getConfigLabel()

Get Configuration label for the option

Returns:

configuration label name

getOption

public org.apache.commons.cli.Option getOption()

Get CLI Option corresponding to the distcp option

Returns:

option

getSwitch

public String getSwitch()

Get Switch symbol

Returns:

switch symbol char

toString

public String toString()

Overrides:

toString in class Enum<DistCpOptionSwitch>

addToConf

public static void addToConf(org.apache.hadoop.conf.Configuration conf,
                             DistCpOptionSwitch option,
                             String value)

Helper function to add an option to hadoop configuration object

Parameters:

conf - - Configuration object to include the option

option - - Option to add

value - - Value

addToConf

public static void addToConf(org.apache.hadoop.conf.Configuration conf,
                             DistCpOptionSwitch option)

Helper function to set an option to hadoop configuration object

Parameters:

conf - - Configuration object to include the option

option - - Option to add