org.apache.hadoop.fs.aliyun.oss

Class AliyunOSSFileSystemStore

java.lang.Object

org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore

public class AliyunOSSFileSystemStore
extends Object

Core implementation of Aliyun OSS Filesystem for Hadoop. Provides the bridging logic between Hadoop's abstract filesystem and Aliyun OSS.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
AliyunOSSFileSystemStore()

Method Summary

Modifier and Type	Method and Description
void	abortMultipartUpload(String key, String uploadId) Abort the specific multipart upload.
void	close() Close OSS client properly.
com.aliyun.oss.model.CompleteMultipartUploadResult	completeMultipartUpload(String key, String uploadId, List<com.aliyun.oss.model.PartETag> partETags) Complete the specific multipart upload.
boolean	copyFile(String srcKey, long srcLen, String dstKey) Copy an object from source key to destination key.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	createLocatedFileStatusIterator(String prefix, int maxListingLength, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.PathFilter filter, FileStatusAcceptor acceptor, String delimiter)
void	deleteDirs(String key) Delete a directory from Aliyun OSS.
void	deleteObject(String key) Delete an object, and update write operation statistics.
void	deleteObjects(List<String> keysToDelete) Delete a list of keys, and update write operation statistics.
com.aliyun.oss.model.ObjectMetadata	getObjectMetadata(String key) Return metadata of a given object key.
String	getUploadId(String key) Initiate multipart upload.
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf, String user, org.apache.hadoop.fs.FileSystem.Statistics stat)
com.aliyun.oss.model.ObjectListing	listObjects(String prefix, int maxListingLength, String marker, boolean recursive) list objects.
void	purge(String prefix) Clean up all objects matching the prefix.
InputStream	retrieve(String key, long byteStart, long byteEnd) Retrieve a part of an object.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	singleStatusRemoteIterator(org.apache.hadoop.fs.FileStatus fileStatus, org.apache.hadoop.fs.BlockLocation[] locations)
void	storeEmptyFile(String key) Upload an empty file as an OSS object, using single upload.
void	uploadObject(String key, File file) Upload a file as an OSS object, using single upload.
com.aliyun.oss.model.PartETag	uploadPart(File file, String key, String uploadId, int idx)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AliyunOSSFileSystemStore

public AliyunOSSFileSystemStore()

Method Detail

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       String user,
                       org.apache.hadoop.fs.FileSystem.Statistics stat)
                throws IOException

Throws:

IOException

deleteObject

public void deleteObject(String key)

Delete an object, and update write operation statistics.

Parameters:

key - key to blob to delete.

deleteObjects

public void deleteObjects(List<String> keysToDelete)
                   throws IOException

Delete a list of keys, and update write operation statistics.

Parameters:

keysToDelete - collection of keys to delete.

Throws:

IOException - if failed to delete objects.

deleteDirs

public void deleteDirs(String key)
                throws IOException

Delete a directory from Aliyun OSS.

Parameters:

key - directory key to delete.

Throws:

IOException - if failed to delete directory.

getObjectMetadata

public com.aliyun.oss.model.ObjectMetadata getObjectMetadata(String key)

Return metadata of a given object key.

Parameters:

key - object key.

Returns:

return null if key does not exist.

storeEmptyFile

public void storeEmptyFile(String key)
                    throws IOException

Upload an empty file as an OSS object, using single upload.

Parameters:

key - object key.

Throws:

IOException - if failed to upload object.

copyFile

public boolean copyFile(String srcKey,
                        long srcLen,
                        String dstKey)

Copy an object from source key to destination key.

Parameters:

srcKey - source key.

srcLen - source file length.

dstKey - destination key.

Returns:

true if file is successfully copied.

uploadObject

public void uploadObject(String key,
                         File file)
                  throws IOException

Upload a file as an OSS object, using single upload.

Parameters:

key - object key.

file - local file to upload.

Throws:

IOException - if failed to upload object.

listObjects

public com.aliyun.oss.model.ObjectListing listObjects(String prefix,
                                                      int maxListingLength,
                                                      String marker,
                                                      boolean recursive)

list objects.

Parameters:

prefix - prefix.

maxListingLength - max no. of entries

marker - last key in any previous search.

recursive - whether to list directory recursively.

Returns:

a list of matches.

retrieve

public InputStream retrieve(String key,
                            long byteStart,
                            long byteEnd)

Retrieve a part of an object.

Parameters:

key - the object name that is being retrieved from the Aliyun OSS.

byteStart - start position.

byteEnd - end position.

Returns:

This method returns null if the key is not found.

close

public void close()

Close OSS client properly.

purge

public void purge(String prefix)
           throws IOException

Clean up all objects matching the prefix.

Parameters:

prefix - Aliyun OSS object prefix.

Throws:

IOException - if failed to clean up objects.

singleStatusRemoteIterator

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> singleStatusRemoteIterator(org.apache.hadoop.fs.FileStatus fileStatus,
                                                                                                              org.apache.hadoop.fs.BlockLocation[] locations)

createLocatedFileStatusIterator

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> createLocatedFileStatusIterator(String prefix,
                                                                                                                   int maxListingLength,
                                                                                                                   org.apache.hadoop.fs.FileSystem fs,
                                                                                                                   org.apache.hadoop.fs.PathFilter filter,
                                                                                                                   FileStatusAcceptor acceptor,
                                                                                                                   String delimiter)

uploadPart

public com.aliyun.oss.model.PartETag uploadPart(File file,
                                                String key,
                                                String uploadId,
                                                int idx)
                                         throws IOException

Throws:

IOException

getUploadId

public String getUploadId(String key)

Initiate multipart upload.

completeMultipartUpload

public com.aliyun.oss.model.CompleteMultipartUploadResult completeMultipartUpload(String key,
                                                                                  String uploadId,
                                                                                  List<com.aliyun.oss.model.PartETag> partETags)

Complete the specific multipart upload.

abortMultipartUpload

public void abortMultipartUpload(String key,
                                 String uploadId)

Abort the specific multipart upload.