alluxio.underfs.s3

Class S3UnderFileSystem

java.lang.Object

alluxio.underfs.UnderFileSystem

alluxio.underfs.s3.S3UnderFileSystem

@ThreadSafe
public final class S3UnderFileSystem
extends UnderFileSystem

S3 FS UnderFileSystem implementation based on the jets3t library.

Nested Class Summary

Nested classes/interfaces inherited from class alluxio.underfs.UnderFileSystem

UnderFileSystem.SpaceType

Field Summary

Fields inherited from class alluxio.underfs.UnderFileSystem

LISTING_LENGTH, mProperties, mUri

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	S3UnderFileSystem(AlluxioURI uri, org.jets3t.service.S3Service s3Service, String bucketName, String bucketPrefix, short bucketMode, String accountOwner) Constructor for S3UnderFileSystem.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	connectFromMaster(String hostname)
void	connectFromWorker(String hostname)
OutputStream	create(String path)
OutputStream	create(String path, CreateOptions options)
static S3UnderFileSystem	createInstance(AlluxioURI uri) Constructs a new instance of S3UnderFileSystem.
boolean	delete(String path, boolean recursive)
boolean	exists(String path)
long	getBlockSizeByte(String path) Gets the block size in bytes.
Object	getConf()
List<String>	getFileLocations(String path)
List<String>	getFileLocations(String path, long offset)
long	getFileSize(String path)
String	getGroup(String path)
short	getMode(String path)
long	getModificationTimeMs(String path)
String	getOwner(String path)
long	getSpace(String path, UnderFileSystem.SpaceType type)
String	getUnderFSType()
boolean	isFile(String path)
String[]	list(String path)
boolean	mkdirs(String path, boolean createParent)
boolean	mkdirs(String path, MkdirsOptions options)
InputStream	open(String path)
InputStream	openAtPosition(String path, long pos) Opens a S3 object at given position and returns the opened input stream.
boolean	rename(String src, String dst)
void	setConf(Object conf)
void	setMode(String path, short mode)
void	setOwner(String path, String user, String group)
boolean	supportsFlush()

Methods inherited from class alluxio.underfs.UnderFileSystem

configureProperties, get, get, getProperties, isHadoopUnderFS, listRecursive, parse, providesStorage, resolveUri, setProperties

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3UnderFileSystem

protected S3UnderFileSystem(AlluxioURI uri,
                 org.jets3t.service.S3Service s3Service,
                 String bucketName,
                 String bucketPrefix,
                 short bucketMode,
                 String accountOwner)

Constructor for S3UnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

s3Service - Jets3t S3 client

bucketName - bucket name of user's configured Alluxio bucket

bucketPrefix - prefix of the bucket

bucketMode - the permission mode that the account owner has to the bucket

accountOwner - the name of the account owner

Method Detail

createInstance

public static S3UnderFileSystem createInstance(AlluxioURI uri)
                                        throws org.jets3t.service.ServiceException

Constructs a new instance of S3UnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

Returns:

the created S3UnderFileSystem instance

Throws:

org.jets3t.service.ServiceException - when a connection to S3 could not be created

getUnderFSType

public String getUnderFSType()

Specified by:

getUnderFSType in class UnderFileSystem

close

public void close()
           throws IOException

Specified by:

close in class UnderFileSystem

Throws:

IOException

connectFromMaster

public void connectFromMaster(String hostname)

Specified by:

connectFromMaster in class UnderFileSystem

connectFromWorker

public void connectFromWorker(String hostname)

Specified by:

connectFromWorker in class UnderFileSystem

create

public OutputStream create(String path)
                    throws IOException

Specified by:

create in class UnderFileSystem

Throws:

IOException

create

public OutputStream create(String path,
                  CreateOptions options)
                    throws IOException

Specified by:

create in class UnderFileSystem

Throws:

IOException

delete

public boolean delete(String path,
             boolean recursive)
               throws IOException

Specified by:

delete in class UnderFileSystem

Throws:

IOException

exists

public boolean exists(String path)
               throws IOException

Specified by:

exists in class UnderFileSystem

Throws:

IOException

getBlockSizeByte

public long getBlockSizeByte(String path)
                      throws IOException

Gets the block size in bytes. There is no concept of a block in S3 and the maximum size of one put is 5 GB, and the max size of a multi-part upload is 5 TB. This method defaults to the default user block size in Alluxio.

Specified by:

getBlockSizeByte in class UnderFileSystem

Parameters:

path - the file name

Returns:

the default Alluxio user block size

Throws:

IOException - this implementation will not throw this exception, but subclasses may

getConf

public Object getConf()

Specified by:

getConf in class UnderFileSystem

getFileLocations

public List<String> getFileLocations(String path)
                              throws IOException

Specified by:

getFileLocations in class UnderFileSystem

Throws:

IOException

getFileLocations

public List<String> getFileLocations(String path,
                            long offset)
                              throws IOException

Specified by:

getFileLocations in class UnderFileSystem

Throws:

IOException

getFileSize

public long getFileSize(String path)
                 throws IOException

Specified by:

getFileSize in class UnderFileSystem

Throws:

IOException

getModificationTimeMs

public long getModificationTimeMs(String path)
                           throws IOException

Specified by:

getModificationTimeMs in class UnderFileSystem

Throws:

IOException

getSpace

public long getSpace(String path,
            UnderFileSystem.SpaceType type)
              throws IOException

Specified by:

getSpace in class UnderFileSystem

Throws:

IOException

isFile

public boolean isFile(String path)
               throws IOException

Specified by:

isFile in class UnderFileSystem

Throws:

IOException

list

public String[] list(String path)
              throws IOException

Specified by:

list in class UnderFileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(String path,
             boolean createParent)
               throws IOException

Specified by:

mkdirs in class UnderFileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(String path,
             MkdirsOptions options)
               throws IOException

Specified by:

mkdirs in class UnderFileSystem

Throws:

IOException

open

public InputStream open(String path)
                 throws IOException

Specified by:

open in class UnderFileSystem

Throws:

IOException

openAtPosition

public InputStream openAtPosition(String path,
                         long pos)
                           throws IOException

Opens a S3 object at given position and returns the opened input stream.

Parameters:

path - the S3 object path

pos - the position to open at

Returns:

the opened input stream

Throws:

IOException - if failed to open file at position

rename

public boolean rename(String src,
             String dst)
               throws IOException

Specified by:

rename in class UnderFileSystem

Throws:

IOException

setConf

public void setConf(Object conf)

Specified by:

setConf in class UnderFileSystem

setOwner

public void setOwner(String path,
            String user,
            String group)

Specified by:

setOwner in class UnderFileSystem

setMode

public void setMode(String path,
           short mode)
             throws IOException

Specified by:

setMode in class UnderFileSystem

Throws:

IOException

getOwner

public String getOwner(String path)
                throws IOException

Specified by:

getOwner in class UnderFileSystem

Throws:

IOException

getGroup

public String getGroup(String path)
                throws IOException

Specified by:

getGroup in class UnderFileSystem

Throws:

IOException

getMode

public short getMode(String path)
              throws IOException

Specified by:

getMode in class UnderFileSystem

Throws:

IOException

supportsFlush

public boolean supportsFlush()

Overrides:

supportsFlush in class UnderFileSystem