org.apache.hadoop.tools

Class GlobbedCopyListing

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.CopyListing

org.apache.hadoop.tools.GlobbedCopyListing

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class GlobbedCopyListing
extends CopyListing

GlobbedCopyListing implements the CopyListing interface, to create the copy listing-file by "globbing" all specified source paths (wild-cards and all.)

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.tools.CopyListing

CopyListing.AclsNotSupportedException, CopyListing.XAttrsNotSupportedException

Constructor Summary

Constructors

Constructor and Description
GlobbedCopyListing(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.security.Credentials credentials) Constructor, to initialize the configuration.

Method Summary

Modifier and Type	Method and Description
void	doBuildListing(org.apache.hadoop.fs.Path pathToListingFile, DistCpContext context) Implementation of CopyListing::buildListing().
protected long	getBytesToCopy() Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
protected long	getNumberOfPaths() Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
protected void	validatePaths(DistCpContext context) Validate input and output paths

Methods inherited from class org.apache.hadoop.tools.CopyListing

buildListing, getCopyListing, getCredentials, getFileListingKey, getFileListingValue, setCredentials

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GlobbedCopyListing

public GlobbedCopyListing(org.apache.hadoop.conf.Configuration configuration,
                          org.apache.hadoop.security.Credentials credentials)

Constructor, to initialize the configuration.

Parameters:

configuration - The input Configuration object.

credentials - Credentials object on which the FS delegation tokens are cached. If null delegation token caching is skipped

Method Detail

validatePaths

protected void validatePaths(DistCpContext context)
                      throws IOException,
                             org.apache.hadoop.tools.CopyListing.InvalidInputException

Validate input and output paths

Specified by:

validatePaths in class CopyListing

Parameters:

context - - Distcp context

Throws:

IOException - any Exception with FS

org.apache.hadoop.tools.CopyListing.InvalidInputException

doBuildListing

public void doBuildListing(org.apache.hadoop.fs.Path pathToListingFile,
                           DistCpContext context)
                    throws IOException

Implementation of CopyListing::buildListing(). Creates the copy listing by "globbing" all source-paths.

Specified by:

doBuildListing in class CopyListing

Parameters:

pathToListingFile - The location at which the copy-listing file is to be created.

context - The distcp context with associated input options.

Throws:

IOException

getBytesToCopy

protected long getBytesToCopy()

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

Specified by:

getBytesToCopy in class CopyListing

Returns:

total bytes to copy

getNumberOfPaths

protected long getNumberOfPaths()

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

Specified by:

getNumberOfPaths in class CopyListing

Returns:

Total number of paths to distcp