Package org.pipecraft.infra.storage.local

Class LocalDiskBucket

java.lang.Object

org.pipecraft.infra.storage.Bucket<File>

org.pipecraft.infra.storage.local.LocalDiskBucket

public class LocalDiskBucket
extends Bucket<File>

A bucket implementation pointing to a local disk folder. Intended mainly for test purposes. Supported optional operations: - File composing - Exclusive file creation operation (lock functionality) - Uploading data using OutputStream - List-after-write consistency Unsupported optional operations: - Generating read/write signed URLs - Public access for uploaded files

Author:

Eyal Schneider

Field Summary

Fields inherited from class org.pipecraft.infra.storage.Bucket

DEFAULT_RETRIER, DEFAULT_RETRY_INITIAL_SLEEP_SEC, DEFAULT_RETRY_MAX_ATTEMPTS, DEFAULT_RETRY_WAIT_TIME_FACTOR, DONE_FILE_NAME

Method Summary

Modifier and Type	Method	Description
File	compose(List<String> paths, String composedFilePath, boolean removeComprisingFiles)	Composes (concats) remote files.
void	copyToAnotherBucket(String fromKey, String toBucket, String toKey)	Copies a file between this bucket to another bucket
void	delete(File f)	Deletes a single object.
boolean	exists(String key)
URL	generateReadOnlyUrl(String key, int expirationSeconds)	Generates a signed url for reading a specific file.
URL	generateResumableSignedUrlForUpload(String key, String contentType, int expirationSeconds, Long maxContentLengthInBytes, boolean isPublic)	Generates a resumable signed url for a for uploading to a specific file.
URL	generateSignedUrl(String key, String contentType, int expirationSeconds, boolean isPublicRead)	Generates a signed url for an upload on a specific file.
void	get(File meta, File output)	Downloads a file given the remote file's metadata object
SizedInputStream	getAsStream(File f, int chunkSize)	Gets an input stream for a remote file, using a given chunk size.
Long	getLastUpdated(File objMetadata)
long	getLength(File f)
File	getObjectMetadata(String key)	Supplies the metadata of a file, if it exists
OutputStream	getOutputStream(String key, int chunkSize)	Gets an output stream for writing to a remote file.
String	getPath(File f)
Iterator<File>	listObjects(String folderPath, boolean recursive)
void	put(String key, InputStream input, long length, String contentType, boolean isPublic, boolean allowOverride)	Uploads data into a remote file from a given InputStream

Methods inherited from class org.pipecraft.infra.storage.Bucket

copy, copyFolderRecursiveInterruptibly, copyFolderRecursiveInterruptibly, copyInterruptibly, copyInterruptibly, copyToAnotherBucketInterruptibly, copyToAnotherBucketInterruptibly, delete, deleteAllByMetaInterruptibly, deleteAllByMetaInterruptibly, deleteAllByMetaInterruptibly, deleteAllByMetaInterruptibly, deleteAllInterruptibly, deleteAllInterruptibly, deleteAllInterruptibly, deleteAllInterruptibly, deleteFolderRecursiveInterruptibly, deleteFolderRecursiveInterruptibly, deleteFolderRegularFiles, deleteInterruptibly, deleteInterruptibly, deleteInterruptibly, deleteInterruptibly, generateResumableSignedUrlForUpload, generateSignedUrl, get, getAllRegularFiles, getAllRegularFilesByMetaInterruptibly, getAllRegularFilesByMetaInterruptibly, getAllRegularFilesByMetaInterruptibly, getAllRegularFilesByMetaInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAllRegularFilesInterruptibly, getAsStream, getAsStream, getAsStream, getBucketName, getFromJson, getInterruptibly, getInterruptibly, getInterruptibly, getInterruptibly, getLastFile, getObjectMetadata, getObjectMetadata, getOutputStream, getSliced, getSliced, isFile, isFilePath, isFolderPath, listFiles, listFiles, listFiles, listFilesRecursive, listFilesRecursive, listFilesRecursive, listFolders, listObjects, listObjects, moveFolderRecursive, moveFolderRecursive, moveInterruptibly, moveInterruptibly, normalizeFolderPath, put, putAllInterruptibly, putAllInterruptibly, putAllRecursiveInterruptibly, putAllRecursiveInterruptibly, putDoneFile, putEmptyFile, putFile, putFileInterruptibly, putFileInterruptibly, putLockFile, putPrivate, putPrivate, putPublic, putPublic, putUniquePrivate, putUniquePrivate, putUniquePrivate, putUniquePublic, putUniquePublic, putUniquePublic, validateNotFolderPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

put

public void put(String key,
                InputStream input,
                long length,
                String contentType,
                boolean isPublic,
                boolean allowOverride)
         throws IOException

Description copied from class: Bucket

Uploads data into a remote file from a given InputStream

Specified by:

put in class Bucket<File>

Parameters:

key - full path of remote file, relative to the bucket

input - InputStream source. Buffering is not required, and is assumed to be added by the implementation. The stream is closed by this method.

length - size in bytes of input stream. Not all implementations require this field, and it depends on the cloud library requirements. Please read the specific implementation details.

contentType - content mime-type (e.g. "image/jpeg"). Not mandatory, may be null.

isPublic - true to set public file access, false for private. Not all implementations support public, so this parameter may be ignored.

allowOverride - When false, if the remote file already exists the call will fail with FileAlreadyExistsException. This is an optional feature. Implementations may throw UnsupportedOperationException when it's set to true.

Throws:

IOException - in case of an actual IO error, or if the given key is illegal. A path ending with '/' is considered to be illegal here, because it follows the folder naming convention.

FileAlreadyExistsException - In case that allowOverride is on and supported, and the target file already exists

copyToAnotherBucket

public void copyToAnotherBucket(String fromKey,
                                String toBucket,
                                String toKey)
                         throws IOException

Description copied from class: Bucket

Copies a file between this bucket to another bucket

Specified by:

copyToAnotherBucket in class Bucket<File>

Parameters:

fromKey - The path of the source file, relative to this bucket

toBucket - The name of the target bucket (may be the current bucket name)

toKey - The path of the target file, relative to the target bucket. Overridden if exists.

Throws:

IOException

delete

public void delete(File f)
            throws IOException

Description copied from class: Bucket

Deletes a single object. Nothing happens and no exception is thrown in case the file doesn't exist.

Specified by:

delete in class Bucket<File>

Parameters:

f - The metadata object pointing to the remote file to be deleted

Throws:

IOException

get

public void get(File meta,
                File output)
         throws IOException

Description copied from class: Bucket

Downloads a file given the remote file's metadata object

Specified by:

get in class Bucket<File>

Parameters:

meta - The metadata object pointing to the file to download

output - The target file to write to (overriding write). The file may not exist, but the folder must exist.

Throws:

FileNotFoundException - In case that the required object wasn't found in the bucket

IOException - In case of IO error while reading the file or writing to local file. This includes non-existing folder in the local file path.

getAsStream

public SizedInputStream getAsStream(File f,
                                    int chunkSize)
                             throws IOException

Description copied from class: Bucket

Gets an input stream for a remote file, using a given chunk size. The chunk size indicates how many bytes to fetch in each request, and may be ignored by some implementations.

Specified by:

getAsStream in class Bucket<File>

Parameters:

f - The remote object metadata

chunkSize - The size (in bytes) of each chunk read from the storage at once, or 0 for using the default one. This parameter may be ignored in implementations where it's not relevant.

Returns:

The input stream to read the requested resource from. Note that this stream is a SizedInputStream, therefore it provides the data size. The stream is aware of any RuntimeException thrown by the underlying cloud library, and convents it to a proper IOException. The returned stream isn't buffered.

Throws:

IOException

FileNotFoundException - In case that the key wasn't found in the bucket

getOutputStream

public OutputStream getOutputStream(String key,
                                    int chunkSize)
                             throws IOException

Description copied from class: Bucket

Gets an output stream for writing to a remote file. The file is assigned private access permissions. Note that since this is an optional feature, implementations may throw UnsupportedOperationException

Specified by:

getOutputStream in class Bucket<File>

Parameters:

key - The remote target file path, relative to the bucket. May exist or not. Overridden by this method if exists.

chunkSize - The size (in bytes) of each written chunk, or 0 for using the default one.

Returns:

The output stream to write to. Not buffered. There is no guarantee that the file is written unless the writing completes with no errors, and the stream is closed by the caller. The written data will override the existing remote file, if exists, in an atomic manner. The output stream is not allowed to throw runtime exceptions for IO errors while writing - only IOExceptions are allowed.

Throws:

IOException - in case of a write error, or if the supplied key has a form of a folder rather than a file

exists

public boolean exists(String key)
               throws IOException

Specified by:

exists in class Bucket<File>

Parameters:

key - file path, relative to the bucket

Returns:

true if and only if the file exists in the bucket. In case the key point to a folder (either existing or not), false is returned.

Throws:

IOException - upon failure to inspect the bucket

listObjects

public Iterator<File> listObjects(String folderPath,
                                  boolean recursive)
                           throws IOException

Specified by:

listObjects in class Bucket<File>

Parameters:

folderPath - A folder path, relative to the bucket. Treated as a folder path - the caller may or may not add '/' to it.

recursive - Indicates whether to to return objects recursively beneath the given folder

Returns:

The set of all objects (files and folders) under the given path, as an iterator. In case that the path doesn't exist, returns an empty iterator. No particular order is guaranteed. In case of an IO error during iteration, any runtime exception is allowed, but UncheckedIOException is recommended. We recommend implementations to return a lazy iterator (with pages in the background) in order to be more memory friendly.

Throws:

IOException - upon failure to list the bucket

getObjectMetadata

public File getObjectMetadata(String key)
                       throws IOException

Description copied from class: Bucket

Supplies the metadata of a file, if it exists

Specified by:

getObjectMetadata in class Bucket<File>

Parameters:

key - the path for the file, relative to the current bucket

Returns:

the metadata of the required object

Throws:

IOException - upon failure to retrieve the metadata for the specified file

FileNotFoundException - if the provided path refers to a non existing remote file, or if it refers to a folder

getPath

public String getPath(File f)

Specified by:

getPath in class Bucket<File>

Parameters:

f - A metadata of an object under the bucket

Returns:

The path of the object represented by this metadata (relative to the bucket). In case that the metadata object represents a folder (and only in this case), the path should be terminated with '/'. In case the metadata indicates the object doesn't belong to the current bucket, IllegalArgumentException is thrown.

getLength

public long getLength(File f)

Specified by:

getLength in class Bucket<File>

Parameters:

f - A metadata of an object under the bucket or another

Returns:

The size in bytes of the object as specified by the metadata. In case the metadata object refers to a folder, 0 should be returned.

getLastUpdated

public Long getLastUpdated(File objMetadata)

Specified by:

getLastUpdated in class Bucket<File>

Parameters:

objMetadata - A metadata of an object under the bucket or another

Returns:

The time the file was last updated, as milliseconds since epoch In case the metadata object refers to a folder, null should be returned.

generateSignedUrl

public URL generateSignedUrl(String key,
                             String contentType,
                             int expirationSeconds,
                             boolean isPublicRead)

Description copied from class: Bucket

Generates a signed url for an upload on a specific file. This generated url will expire within the required number of seconds.
This generated url only allows the client to use PUT operation on it.

Specified by:

generateSignedUrl in class Bucket<File>

Parameters:

key - the target key, relative to the bucket

contentType - the key's content type. this content type must be supplied by the client when uploading to the url.

expirationSeconds - number of seconds in which the sign url is valid

isPublicRead - true to set public file access, false for private

Returns:

a signed url for the required key

generateReadOnlyUrl

public URL generateReadOnlyUrl(String key,
                               int expirationSeconds)

Description copied from class: Bucket

Generates a signed url for reading a specific file. This generated url will expire within the required number of seconds.
This generated url only allows the client read (GET) from this url.

Specified by:

generateReadOnlyUrl in class Bucket<File>

Parameters:

key - the remote file path, relative to the bucket

expirationSeconds - number of seconds during which the sign url is valid

Returns:

a signed url for the required file

generateResumableSignedUrlForUpload

public URL generateResumableSignedUrlForUpload(String key,
                                               String contentType,
                                               int expirationSeconds,
                                               Long maxContentLengthInBytes,
                                               boolean isPublic)
                                        throws IOException

Description copied from class: Bucket

Generates a resumable signed url for a for uploading to a specific file. This generated url will expire within the required number of seconds.
This generated url only allows the client to use PUT operation on it.

The upload file is limited only to the declared size in byte if maxContentLengthInBytes is not null.

Specified by:

generateResumableSignedUrlForUpload in class Bucket<File>

Parameters:

key - the target file the sign url points to

contentType - the key's content type. this content type must be supplied by the client when uploading to the url.

expirationSeconds - number of seconds in which the sign url is valid

maxContentLengthInBytes - if not null then limits the uploaded content bytes.

isPublic - true to set public file access, false for private.

Returns:

a signed url for the required key

Throws:

IOException - if could not sign url

compose

public File compose(List<String> paths,
                    String composedFilePath,
                    boolean removeComprisingFiles)
             throws IOException

Description copied from class: Bucket

Composes (concats) remote files. This operation is done remotely, and typically performed in an efficient manner by creating a virtual file rather than copying data. Important notes: 1) Be careful when composing files, since not all file formats (specially compressed ones) are valid after concatenation. 2) The final file is created atomically, but remote intermediate files may be created in the target folder during the operation. The intermediate files are removed (best effort) once the operation terminates, either successfully or not.

Specified by:

compose in class Bucket<File>

Parameters:

paths - The non-empty list of paths (relative to the bucket) of files to compose, in the required order

composedFilePath - The target path (relative to the bucket) of the composed file

removeComprisingFiles - Whether to remove the files that were concatenated. Deletion is done only after successful composing of all files.

Returns:

The path to the composed object, relative to the bucket. May be one of the input files (for achieving kind of append functionality for example). Overridden if already exists.

Throws:

IOException - In case of IO error, or if the supplied target file has a form of a folder rather than a file