org.apache.hadoop.fs.azure

Class AzureNativeFileSystemStore

java.lang.Object

org.apache.hadoop.fs.azure.AzureNativeFileSystemStore

@InterfaceAudience.Private
public class AzureNativeFileSystemStore
extends Object

Core implementation of Windows Azure Filesystem for Hadoop. Provides the bridging logic between Hadoop's abstract filesystem and Azure Storage

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_DOWNLOAD_BLOCK_SIZE
static int	DEFAULT_UPLOAD_BLOCK_SIZE
static String	KEY_ATOMIC_RENAME_DIRECTORIES Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.
static String	KEY_PAGE_BLOB_DIRECTORIES Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs.
static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
AzureNativeFileSystemStore()

Method Summary

Modifier and Type	Method and Description
SelfRenewingLease	acquireLease(String key) Get a lease on the blob identified by key.
void	changePermissionStatus(String key, PermissionStatus newPermission) Changes the permission status on the given key.
void	close()
void	delete(String key)
void	delete(String key, SelfRenewingLease lease)
void	dump() Diagnostic method to dump state to the console.
protected void	finalize()
static String	getAccountKeyFromConfiguration(String accountName, Configuration conf)
BandwidthGaugeUpdater	getBandwidthGaugeUpdater()
String	getLinkInFileMetadata(String key) If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file.
void	initialize(URI uri, Configuration conf, AzureFileSystemInstrumentation instrumentation) Method for the URI and configuration object necessary to create a storage session with an Azure session.
boolean	isAtomicRenameKey(String key) Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.
boolean	isKeyForDirectorySet(String key, Set<String> dirSet)
boolean	isPageBlobKey(String key) Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.
org.apache.hadoop.fs.azure.PartialListing	list(String prefix, int maxListingCount, int maxListingDepth)
org.apache.hadoop.fs.azure.PartialListing	list(String prefix, int maxListingCount, int maxListingDepth, String priorLastKey)
org.apache.hadoop.fs.azure.PartialListing	listAll(String prefix, int maxListingCount, int maxListingDepth, String priorLastKey)
void	purge(String prefix) Delete all keys with the given prefix.
void	rename(String srcKey, String dstKey)
void	rename(String srcKey, String dstKey, boolean acquireLease, SelfRenewingLease existingLease)
DataInputStream	retrieve(String key)
DataInputStream	retrieve(String key, long startByteOffset)
org.apache.hadoop.fs.azure.FileMetadata	retrieveMetadata(String key)
void	storeEmptyFolder(String key, PermissionStatus permissionStatus)
void	storeEmptyLinkFile(String key, String tempBlobKey, PermissionStatus permissionStatus) Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.
DataOutputStream	storefile(String key, PermissionStatus permissionStatus)
void	updateFolderLastModifiedTime(String key, Date lastModified, SelfRenewingLease folderLease)
void	updateFolderLastModifiedTime(String key, SelfRenewingLease folderLease)

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

KEY_PAGE_BLOB_DIRECTORIES

public static final String KEY_PAGE_BLOB_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs. Entries should be plain directory names (i.e. not URIs) with no leading or trailing slashes. Delimit the entries with commas.

See Also:

Constant Field Values

KEY_ATOMIC_RENAME_DIRECTORIES

public static final String KEY_ATOMIC_RENAME_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.

See Also:

Constant Field Values

DEFAULT_DOWNLOAD_BLOCK_SIZE

public static final int DEFAULT_DOWNLOAD_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_UPLOAD_BLOCK_SIZE

public static final int DEFAULT_UPLOAD_BLOCK_SIZE

See Also:

Constant Field Values

Constructor Detail

AzureNativeFileSystemStore

public AzureNativeFileSystemStore()

Method Detail

getBandwidthGaugeUpdater

public BandwidthGaugeUpdater getBandwidthGaugeUpdater()

initialize

public void initialize(URI uri,
                       Configuration conf,
                       AzureFileSystemInstrumentation instrumentation)
                throws IllegalArgumentException,
                       AzureException,
                       IOException

Method for the URI and configuration object necessary to create a storage session with an Azure session. It parses the scheme to ensure it matches the storage protocol supported by this file system.

Parameters:

uri - - URI for target storage blob.

conf - - reference to configuration object.

instrumentation - - the metrics source that will keep track of operations here.

Throws:

IllegalArgumentException - if URI or job object is null, or invalid scheme.

AzureException

IOException

getAccountKeyFromConfiguration

public static String getAccountKeyFromConfiguration(String accountName,
                                                    Configuration conf)
                                             throws KeyProviderException

Throws:

KeyProviderException

isPageBlobKey

public boolean isPageBlobKey(String key)

Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.

isAtomicRenameKey

public boolean isAtomicRenameKey(String key)

Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.

isKeyForDirectorySet

public boolean isKeyForDirectorySet(String key,
                                    Set<String> dirSet)

storefile

public DataOutputStream storefile(String key,
                                  PermissionStatus permissionStatus)
                           throws AzureException

Throws:

AzureException

storeEmptyFolder

public void storeEmptyFolder(String key,
                             PermissionStatus permissionStatus)
                      throws AzureException

Throws:

AzureException

storeEmptyLinkFile

public void storeEmptyLinkFile(String key,
                               String tempBlobKey,
                               PermissionStatus permissionStatus)
                        throws AzureException

Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.

Throws:

AzureException

getLinkInFileMetadata

public String getLinkInFileMetadata(String key)
                             throws AzureException

If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file. Otherwise, returns null.

Throws:

AzureException

retrieveMetadata

public org.apache.hadoop.fs.azure.FileMetadata retrieveMetadata(String key)
                                                         throws IOException

Throws:

IOException

retrieve

public DataInputStream retrieve(String key)
                         throws AzureException,
                                IOException

Throws:

AzureException

IOException

retrieve

public DataInputStream retrieve(String key,
                                long startByteOffset)
                         throws AzureException,
                                IOException

Throws:

AzureException

IOException

list

public org.apache.hadoop.fs.azure.PartialListing list(String prefix,
                                                      int maxListingCount,
                                                      int maxListingDepth)
                                               throws IOException

Throws:

IOException

list

public org.apache.hadoop.fs.azure.PartialListing list(String prefix,
                                                      int maxListingCount,
                                                      int maxListingDepth,
                                                      String priorLastKey)
                                               throws IOException

Throws:

IOException

listAll

public org.apache.hadoop.fs.azure.PartialListing listAll(String prefix,
                                                         int maxListingCount,
                                                         int maxListingDepth,
                                                         String priorLastKey)
                                                  throws IOException

Throws:

IOException

delete

public void delete(String key,
                   SelfRenewingLease lease)
            throws IOException

Throws:

IOException

delete

public void delete(String key)
            throws IOException

Throws:

IOException

rename

public void rename(String srcKey,
                   String dstKey)
            throws IOException

Throws:

IOException

rename

public void rename(String srcKey,
                   String dstKey,
                   boolean acquireLease,
                   SelfRenewingLease existingLease)
            throws IOException

Throws:

IOException

changePermissionStatus

public void changePermissionStatus(String key,
                                   PermissionStatus newPermission)
                            throws AzureException

Changes the permission status on the given key.

Throws:

AzureException

purge

public void purge(String prefix)
           throws IOException

Delete all keys with the given prefix. Used for testing.

Throws:

IOException

acquireLease

public SelfRenewingLease acquireLease(String key)
                               throws AzureException

Get a lease on the blob identified by key. This lease will be renewed indefinitely by a background thread.

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(String key,
                                         Date lastModified,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(String key,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

dump

public void dump()
          throws IOException

Diagnostic method to dump state to the console.

Throws:

IOException

close

public void close()

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable