com.upplication.s3fs

Class S3FileSystemProvider

java.lang.Object

java.nio.file.spi.FileSystemProvider

com.upplication.s3fs.S3FileSystemProvider

public class S3FileSystemProvider
extends FileSystemProvider

Spec:

URI: s3://[endpoint]/{bucket}/{key} If endpoint is missing, it's assumed to be the default S3 endpoint (s3.amazonaws.com)

FileSystem roots: /{bucket}/

Treatment of S3 objects: - If a key ends in "/" it's considered a directory *and* a regular file. Otherwise, it's just a regular file. - It is legal for a key "xyz" and "xyz/" to exist at the same time. The latter is treated as a directory. - If a file "a/b/c" exists but there's no "a" or "a/b/", these are considered "implicit" directories. They can be listed, traversed and deleted.

Deviations from FileSystem provider API: - Deleting a file or directory always succeeds, regardless of whether the file/directory existed before the operation was issued i.e. Files.delete() and Files.deleteIfExists() are equivalent.

Future versions of this provider might allow for a strict mode that mimics the semantics of the FileSystem provider API on a best effort basis, at an increased processing cost.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AMAZON_S3_FACTORY_CLASS
static String	CHARSET_KEY

Constructor Summary

Constructors

Constructor and Description
S3FileSystemProvider()

Method Summary

Modifier and Type	Method and Description
protected void	addEnvProperties(Properties props, Map<String,?> env)
void	checkAccess(Path path, AccessMode... modes)
void	close(S3FileSystem fileSystem)
void	copy(Path source, Path target, CopyOption... options)
void	createDirectory(Path dir, FileAttribute<?>... attrs) Deviations from spec: Does not perform atomic check-and-create.
S3FileSystem	createFileSystem(URI uri, Properties props) Create the fileSystem
void	delete(Path path)
protected com.amazonaws.services.s3.AmazonS3	getAmazonS3(URI uri, Properties props)
protected AmazonS3Factory	getAmazonS3Factory(Properties props)
Cache	getCache()
<V extends FileAttributeView> V	getFileAttributeView(Path path, Class<V> type, LinkOption... options)
FileStore	getFileStore(Path path)
S3FileSystem	getFileSystem(URI uri)
FileSystem	getFileSystem(URI uri, Map<String,?> env) Get existing filesystem based on a combination of URI and env settings.
protected String	getFileSystemKey(URI uri, Properties props) get the file system key represented by: the access key @ endpoint.
protected static ConcurrentMap<String,S3FileSystem>	getFilesystems() only 4 testing
Path	getPath(URI uri) Deviation from spec: throws FileSystemNotFoundException if FileSystem hasn't yet been initialized.
String	getScheme()
boolean	isHidden(Path path)
boolean	isOpen(S3FileSystem s3FileSystem)
boolean	isSameFile(Path path1, Path path2)
Properties	loadAmazonProperties() find /amazon.properties in the classpath
void	move(Path source, Path target, CopyOption... options)
SeekableByteChannel	newByteChannel(Path path, Set<? extends OpenOption> options, FileAttribute<?>... attrs)
DirectoryStream<Path>	newDirectoryStream(Path dir, DirectoryStream.Filter<? super Path> filter)
FileChannel	newFileChannel(Path path, Set<? extends OpenOption> options, FileAttribute<?>... attrs)
FileSystem	newFileSystem(URI uri, Map<String,?> env)
InputStream	newInputStream(Path path, OpenOption... options)
protected boolean	overloadPropertiesWithEnv(Properties props, Map<String,?> env, String key)
boolean	overloadPropertiesWithSystemEnv(Properties props, String key) The system envs have preference over the properties files.
boolean	overloadPropertiesWithSystemProps(Properties props, String key)
<A extends BasicFileAttributes> A	readAttributes(Path path, Class<A> type, LinkOption... options)
Map<String,Object>	readAttributes(Path path, String attributes, LinkOption... options)
void	setAttribute(Path path, String attribute, Object value, LinkOption... options)
void	setCache(Cache cache)
String	systemGetEnv(String key) Get the system env with the key param
protected void	validateUri(URI uri)

Methods inherited from class java.nio.file.spi.FileSystemProvider

createLink, createSymbolicLink, deleteIfExists, installedProviders, newAsynchronousFileChannel, newFileSystem, newOutputStream, readSymbolicLink

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHARSET_KEY

public static final String CHARSET_KEY

See Also:

Constant Field Values

AMAZON_S3_FACTORY_CLASS

public static final String AMAZON_S3_FACTORY_CLASS

See Also:

Constant Field Values

Constructor Detail

S3FileSystemProvider

public S3FileSystemProvider()

Method Detail

getScheme

public String getScheme()

Specified by:

getScheme in class FileSystemProvider

newFileSystem

public FileSystem newFileSystem(URI uri,
                                Map<String,?> env)

Specified by:

newFileSystem in class FileSystemProvider

getFileSystemKey

protected String getFileSystemKey(URI uri,
                                  Properties props)

get the file system key represented by: the access key @ endpoint. Example: access-key@s3.amazonaws.com If uri host is empty then s3.amazonaws.com are used as host

Parameters:

uri - URI with the endpoint

props - with the access key property

Returns:

String

validateUri

protected void validateUri(URI uri)

addEnvProperties

protected void addEnvProperties(Properties props,
                                Map<String,?> env)

overloadPropertiesWithEnv

protected boolean overloadPropertiesWithEnv(Properties props,
                                            Map<String,?> env,
                                            String key)

Returns:

true if the key are overloaded by the map parameter

overloadPropertiesWithSystemProps

public boolean overloadPropertiesWithSystemProps(Properties props,
                                                 String key)

Returns:

true if the key are overloaded by a system property

overloadPropertiesWithSystemEnv

public boolean overloadPropertiesWithSystemEnv(Properties props,
                                               String key)

The system envs have preference over the properties files. So we overload it

Parameters:

props - Properties

key - String

Returns:

true if the key are overloaded by a system property

systemGetEnv

public String systemGetEnv(String key)

Get the system env with the key param

Parameters:

key - String

Returns:

String or null

getFileSystem

public FileSystem getFileSystem(URI uri,
                                Map<String,?> env)

Get existing filesystem based on a combination of URI and env settings. Create new filesystem otherwise.

Parameters:

uri - URI of existing, or to be created filesystem.

env - environment settings.

Returns:

new or existing filesystem.

getFileSystem

public S3FileSystem getFileSystem(URI uri)

Specified by:

getFileSystem in class FileSystemProvider

getPath

public Path getPath(URI uri)

Deviation from spec: throws FileSystemNotFoundException if FileSystem hasn't yet been initialized. Call newFileSystem() first. Need credentials. Maybe set credentials after? how? TODO: we can create a new one if the credentials are present by: s3://access-key:secret-key@endpoint.com/

Specified by:

getPath in class FileSystemProvider

newDirectoryStream

public DirectoryStream<Path> newDirectoryStream(Path dir,
                                                DirectoryStream.Filter<? super Path> filter)
                                         throws IOException

Specified by:

newDirectoryStream in class FileSystemProvider

Throws:

IOException

newInputStream

public InputStream newInputStream(Path path,
                                  OpenOption... options)
                           throws IOException

Overrides:

newInputStream in class FileSystemProvider

Throws:

IOException

newByteChannel

public SeekableByteChannel newByteChannel(Path path,
                                          Set<? extends OpenOption> options,
                                          FileAttribute<?>... attrs)
                                   throws IOException

Specified by:

newByteChannel in class FileSystemProvider

Throws:

IOException

newFileChannel

public FileChannel newFileChannel(Path path,
                                  Set<? extends OpenOption> options,
                                  FileAttribute<?>... attrs)
                           throws IOException

Overrides:

newFileChannel in class FileSystemProvider

Throws:

IOException

createDirectory

public void createDirectory(Path dir,
                            FileAttribute<?>... attrs)
                     throws IOException

Deviations from spec: Does not perform atomic check-and-create. Since a directory is just an S3 object, all directories in the hierarchy are created or it already existed.

Specified by:

createDirectory in class FileSystemProvider

Throws:

IOException

delete

public void delete(Path path)
            throws IOException

Specified by:

delete in class FileSystemProvider

Throws:

IOException

copy

public void copy(Path source,
                 Path target,
                 CopyOption... options)
          throws IOException

Specified by:

copy in class FileSystemProvider

Throws:

IOException

move

public void move(Path source,
                 Path target,
                 CopyOption... options)
          throws IOException

Specified by:

move in class FileSystemProvider

Throws:

IOException

isSameFile

public boolean isSameFile(Path path1,
                          Path path2)
                   throws IOException

Specified by:

isSameFile in class FileSystemProvider

Throws:

IOException

isHidden

public boolean isHidden(Path path)
                 throws IOException

Specified by:

isHidden in class FileSystemProvider

Throws:

IOException

getFileStore

public FileStore getFileStore(Path path)
                       throws IOException

Specified by:

getFileStore in class FileSystemProvider

Throws:

IOException

checkAccess

public void checkAccess(Path path,
                        AccessMode... modes)
                 throws IOException

Specified by:

checkAccess in class FileSystemProvider

Throws:

IOException

getFileAttributeView

public <V extends FileAttributeView> V getFileAttributeView(Path path,
                                                            Class<V> type,
                                                            LinkOption... options)

Specified by:

getFileAttributeView in class FileSystemProvider

readAttributes

public <A extends BasicFileAttributes> A readAttributes(Path path,
                                                        Class<A> type,
                                                        LinkOption... options)
                                                 throws IOException

Specified by:

readAttributes in class FileSystemProvider

Throws:

IOException

readAttributes

public Map<String,Object> readAttributes(Path path,
                                         String attributes,
                                         LinkOption... options)
                                  throws IOException

Specified by:

readAttributes in class FileSystemProvider

Throws:

IOException

setAttribute

public void setAttribute(Path path,
                         String attribute,
                         Object value,
                         LinkOption... options)
                  throws IOException

Specified by:

setAttribute in class FileSystemProvider

Throws:

IOException

createFileSystem

public S3FileSystem createFileSystem(URI uri,
                                     Properties props)

Create the fileSystem

Parameters:

uri - URI

props - Properties

Returns:

S3FileSystem never null

getAmazonS3

protected com.amazonaws.services.s3.AmazonS3 getAmazonS3(URI uri,
                                                         Properties props)

getAmazonS3Factory

protected AmazonS3Factory getAmazonS3Factory(Properties props)

loadAmazonProperties

public Properties loadAmazonProperties()

find /amazon.properties in the classpath

Returns:

Properties amazon.properties

close

public void close(S3FileSystem fileSystem)

isOpen

public boolean isOpen(S3FileSystem s3FileSystem)

getFilesystems

protected static ConcurrentMap<String,S3FileSystem> getFilesystems()

only 4 testing

getCache

public Cache getCache()

setCache

public void setCache(Cache cache)