alluxio.underfs.s3

Class S3UnderFileSystem

java.lang.Object

alluxio.underfs.UnderFileSystem

alluxio.underfs.s3.S3UnderFileSystem

@ThreadSafe
public class S3UnderFileSystem
extends UnderFileSystem

S3 FS UnderFileSystem implementation based on the jets3t library.

Nested Class Summary

Nested classes/interfaces inherited from class alluxio.underfs.UnderFileSystem

UnderFileSystem.SpaceType, UnderFileSystem.UnderFSType

Field Summary

Fields inherited from class alluxio.underfs.UnderFileSystem

mConfiguration, mProperties, mUri

Constructor Summary

Constructors

Constructor and Description
S3UnderFileSystem(AlluxioURI uri, Configuration conf) Constructs a new instance of S3UnderFileSystem.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	connectFromMaster(Configuration conf, String hostname)
void	connectFromWorker(Configuration conf, String hostname)
OutputStream	create(String path)
OutputStream	create(String path, int blockSizeByte)
OutputStream	create(String path, short replication, int blockSizeByte)
boolean	delete(String path, boolean recursive)
boolean	exists(String path)
long	getBlockSizeByte(String path) Gets the block size in bytes.
Object	getConf()
List<String>	getFileLocations(String path)
List<String>	getFileLocations(String path, long offset)
long	getFileSize(String path)
long	getModificationTimeMs(String path)
long	getSpace(String path, UnderFileSystem.SpaceType type)
UnderFileSystem.UnderFSType	getUnderFSType()
boolean	isFile(String path)
String[]	list(String path)
boolean	mkdirs(String path, boolean createParent)
InputStream	open(String path)
InputStream	openAtPosition(String path, long pos) Opens a S3 object at given position and returns the opened input stream.
boolean	rename(String src, String dst)
void	setConf(Object conf)
void	setOwner(String path, String user, String group)
void	setPermission(String path, String posixPerm)

Methods inherited from class alluxio.underfs.UnderFileSystem

configureProperties, get, get, getProperties, isHadoopUnderFS, listRecursive, parse, providesStorage, resolveUri, setProperties

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3UnderFileSystem

public S3UnderFileSystem(AlluxioURI uri,
                 Configuration conf)
                  throws org.jets3t.service.ServiceException

Constructs a new instance of S3UnderFileSystem.

Parameters:

uri - the AlluxioURI for this UFS

conf - the configuration for Alluxio

Throws:

org.jets3t.service.ServiceException - when a connection to S3 could not be created

Method Detail

getUnderFSType

public UnderFileSystem.UnderFSType getUnderFSType()

Specified by:

getUnderFSType in class UnderFileSystem

close

public void close()
           throws IOException

Specified by:

close in class UnderFileSystem

Throws:

IOException

connectFromMaster

public void connectFromMaster(Configuration conf,
                     String hostname)

Specified by:

connectFromMaster in class UnderFileSystem

connectFromWorker

public void connectFromWorker(Configuration conf,
                     String hostname)

Specified by:

connectFromWorker in class UnderFileSystem

create

public OutputStream create(String path)
                    throws IOException

Specified by:

create in class UnderFileSystem

Throws:

IOException

create

public OutputStream create(String path,
                  int blockSizeByte)
                    throws IOException

Specified by:

create in class UnderFileSystem

Throws:

IOException

create

public OutputStream create(String path,
                  short replication,
                  int blockSizeByte)
                    throws IOException

Specified by:

create in class UnderFileSystem

Throws:

IOException

delete

public boolean delete(String path,
             boolean recursive)
               throws IOException

Specified by:

delete in class UnderFileSystem

Throws:

IOException

exists

public boolean exists(String path)
               throws IOException

Specified by:

exists in class UnderFileSystem

Throws:

IOException

getBlockSizeByte

public long getBlockSizeByte(String path)
                      throws IOException

Gets the block size in bytes. There is no concept of a block in S3 and the maximum size of one put is 5 GB, and the max size of a multi-part upload is 5 TB. This method defaults to the default user block size in Alluxio.

Specified by:

getBlockSizeByte in class UnderFileSystem

Parameters:

path - the file name

Returns:

the default Alluxio user block size

Throws:

IOException - this implementation will not throw this exception, but subclasses may

getConf

public Object getConf()

Specified by:

getConf in class UnderFileSystem

getFileLocations

public List<String> getFileLocations(String path)
                              throws IOException

Specified by:

getFileLocations in class UnderFileSystem

Throws:

IOException

getFileLocations

public List<String> getFileLocations(String path,
                            long offset)
                              throws IOException

Specified by:

getFileLocations in class UnderFileSystem

Throws:

IOException

getFileSize

public long getFileSize(String path)
                 throws IOException

Specified by:

getFileSize in class UnderFileSystem

Throws:

IOException

getModificationTimeMs

public long getModificationTimeMs(String path)
                           throws IOException

Specified by:

getModificationTimeMs in class UnderFileSystem

Throws:

IOException

getSpace

public long getSpace(String path,
            UnderFileSystem.SpaceType type)
              throws IOException

Specified by:

getSpace in class UnderFileSystem

Throws:

IOException

isFile

public boolean isFile(String path)
               throws IOException

Specified by:

isFile in class UnderFileSystem

Throws:

IOException

list

public String[] list(String path)
              throws IOException

Specified by:

list in class UnderFileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(String path,
             boolean createParent)
               throws IOException

Specified by:

mkdirs in class UnderFileSystem

Throws:

IOException

open

public InputStream open(String path)
                 throws IOException

Specified by:

open in class UnderFileSystem

Throws:

IOException

openAtPosition

public InputStream openAtPosition(String path,
                         long pos)
                           throws IOException

Opens a S3 object at given position and returns the opened input stream.

Parameters:

path - the S3 object path

pos - the position to open at

Returns:

the opened input stream

Throws:

IOException - if failed to open file at position

rename

public boolean rename(String src,
             String dst)
               throws IOException

Specified by:

rename in class UnderFileSystem

Throws:

IOException

setConf

public void setConf(Object conf)

Specified by:

setConf in class UnderFileSystem

setOwner

public void setOwner(String path,
            String user,
            String group)

Specified by:

setOwner in class UnderFileSystem

setPermission

public void setPermission(String path,
                 String posixPerm)
                   throws IOException

Specified by:

setPermission in class UnderFileSystem

Throws:

IOException