org.apache.hadoop.hdfs.web

Class WebHdfsFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.hdfs.web.WebHdfsFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, DelegationTokenRenewer.Renewable

Direct Known Subclasses:

SWebHdfsFileSystem

public class WebHdfsFileSystem
extends FileSystem
implements DelegationTokenRenewer.Renewable

A FileSystem for HDFS over the web.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	WebHdfsFileSystem.ReadRunner This class will allow retries to occur for both open and read operations.
class	WebHdfsFileSystem.WebHdfsInputStream This class is used for opening, reading, and seeking files while using the WebHdfsFileSystem.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	CANT_FALLBACK_TO_INSECURE_MSG
protected org.apache.hadoop.hdfs.web.URLConnectionFactory	connectionFactory Default connection factory may be overridden in tests to use smaller timeout values
static org.apache.commons.logging.Log	LOG
static String	PATH_PREFIX Http URI: http://namenode:port/{PATH_PREFIX}/path/to/file
static String	SCHEME File System URI: {SCHEME}://namenode:port/path/to/file
static Text	TOKEN_KIND Delegation token kind
protected Text	tokenServiceName
static int	VERSION WebHdfs version.

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
WebHdfsFileSystem()

Method Summary

Modifier and Type	Method and Description
void	access(Path path, FsAction mode)
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress)
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<?> token)
protected URI	canonicalizeUri(URI uri)
void	close()
void	concat(Path trg, Path[] srcs)
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress)
Path	createSnapshot(Path path, String snapshotName)
void	createSymlink(Path destination, Path f, boolean createParent) Create a symlink pointing to the destination path.
boolean	delete(Path f, boolean recursive)
void	deleteSnapshot(Path path, String snapshotName)
AclStatus	getAclStatus(Path f)
String	getCanonicalServiceName()
URI	getCanonicalUri()
ContentSummary	getContentSummary(Path p)
long	getDefaultBlockSize()
int	getDefaultPort()
short	getDefaultReplication()
protected org.apache.hadoop.security.token.Token<?>	getDelegationToken()
org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>	getDelegationToken(String renewer)
BlockLocation[]	getFileBlockLocations(FileStatus status, long offset, long length)
BlockLocation[]	getFileBlockLocations(Path p, long offset, long length)
MD5MD5CRC32FileChecksum	getFileChecksum(Path p)
FileStatus	getFileStatus(Path f)
Path	getHomeDirectory()
static String	getHomeDirectoryString(UserGroupInformation ugi)
org.apache.hadoop.security.token.Token<?>	getRenewToken()
String	getScheme() Return the protocol scheme for the FileSystem.
protected Text	getTokenKind()
protected String	getTransportScheme() return the underlying transport protocol (http / https).
URI	getUri()
Path	getWorkingDirectory()
byte[]	getXAttr(Path p, String name)
Map<String,byte[]>	getXAttrs(Path p)
Map<String,byte[]>	getXAttrs(Path p, List<String> names)
void	initialize(URI uri, Configuration conf)
static boolean	isEnabled(Configuration conf, org.apache.commons.logging.Log log) Is WebHDFS enabled in conf?
FileStatus[]	listStatus(Path f)
List<String>	listXAttrs(Path p)
boolean	mkdirs(Path f, FsPermission permission)
void	modifyAclEntries(Path path, List<AclEntry> aclSpec)
FSDataInputStream	open(Path f, int bufferSize)
void	removeAcl(Path path)
void	removeAclEntries(Path path, List<AclEntry> aclSpec)
void	removeDefaultAcl(Path path)
void	removeXAttr(Path p, String name)
boolean	rename(Path src, Path dst)
void	rename(Path src, Path dst, Options.Rename... options)
void	renameSnapshot(Path path, String snapshotOldName, String snapshotNewName)
long	renewDelegationToken(org.apache.hadoop.security.token.Token<?> token)
void	setAcl(Path p, List<AclEntry> aclSpec)
<T extends org.apache.hadoop.security.token.TokenIdentifier> void	setDelegationToken(org.apache.hadoop.security.token.Token<T> token)
void	setOwner(Path p, String owner, String group)
void	setPermission(Path p, FsPermission permission)
boolean	setReplication(Path p, short replication)
void	setRetryPolicy(org.apache.hadoop.io.retry.RetryPolicy rp)
void	setTimes(Path p, long mtime, long atime)
void	setWorkingDirectory(Path dir)
void	setXAttr(Path p, String name, byte[] value, EnumSet<XAttrSetFlag> flag)
boolean	truncate(Path f, long newLength)

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, areSymlinksEnabled, cancelDeleteOnExit, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, delete, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, get, getAllStatistics, getBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, makeQualified, mkdirs, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, resolveLink, resolvePath, setDefaultUri, setDefaultUri, setVerifyChecksum, setWriteChecksum, setXAttr, startLocalOutput, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

SCHEME

public static final String SCHEME

File System URI: {SCHEME}://namenode:port/path/to/file

See Also:

Constant Field Values

VERSION

public static final int VERSION

WebHdfs version.

See Also:

Constant Field Values

PATH_PREFIX

public static final String PATH_PREFIX

Http URI: http://namenode:port/{PATH_PREFIX}/path/to/file

See Also:

Constant Field Values

connectionFactory

protected org.apache.hadoop.hdfs.web.URLConnectionFactory connectionFactory

Default connection factory may be overridden in tests to use smaller timeout values

TOKEN_KIND

public static final Text TOKEN_KIND

Delegation token kind

CANT_FALLBACK_TO_INSECURE_MSG

public static final String CANT_FALLBACK_TO_INSECURE_MSG

See Also:

Constant Field Values

tokenServiceName

protected Text tokenServiceName

Constructor Detail

WebHdfsFileSystem

public WebHdfsFileSystem()

Method Detail

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class FileSystem

Returns:

webhdfs

getTransportScheme

protected String getTransportScheme()

return the underlying transport protocol (http / https).

getTokenKind

protected Text getTokenKind()

initialize

public void initialize(URI uri,
                       Configuration conf)
                throws IOException

Overrides:

initialize in class FileSystem

Throws:

IOException

getCanonicalUri

public URI getCanonicalUri()

Overrides:

getCanonicalUri in class FileSystem

isEnabled

public static boolean isEnabled(Configuration conf,
                                org.apache.commons.logging.Log log)

Is WebHDFS enabled in conf?

getDelegationToken

protected org.apache.hadoop.security.token.Token<?> getDelegationToken()
                                                                throws IOException

Throws:

IOException

getDefaultPort

public int getDefaultPort()

Overrides:

getDefaultPort in class FileSystem

getUri

public URI getUri()

Specified by:

getUri in class FileSystem

canonicalizeUri

protected URI canonicalizeUri(URI uri)

Overrides:

canonicalizeUri in class FileSystem

getHomeDirectoryString

public static String getHomeDirectoryString(UserGroupInformation ugi)

Returns:

the home directory.

getHomeDirectory

public Path getHomeDirectory()

Overrides:

getHomeDirectory in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class FileSystem

setWorkingDirectory

public void setWorkingDirectory(Path dir)

Specified by:

setWorkingDirectory in class FileSystem

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Specified by:

getFileStatus in class FileSystem

Throws:

IOException

getAclStatus

public AclStatus getAclStatus(Path f)
                       throws IOException

Overrides:

getAclStatus in class FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(Path f,
                      FsPermission permission)
               throws IOException

Specified by:

mkdirs in class FileSystem

Throws:

IOException

createSymlink

public void createSymlink(Path destination,
                          Path f,
                          boolean createParent)
                   throws IOException

Create a symlink pointing to the destination path.

Overrides:

createSymlink in class FileSystem

Throws:

IOException

See Also:

Hdfs.createSymlink(Path, Path, boolean)

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Specified by:

rename in class FileSystem

Throws:

IOException

rename

public void rename(Path src,
                   Path dst,
                   Options.Rename... options)
            throws IOException

Overrides:

rename in class FileSystem

Throws:

IOException

setXAttr

public void setXAttr(Path p,
                     String name,
                     byte[] value,
                     EnumSet<XAttrSetFlag> flag)
              throws IOException

Overrides:

setXAttr in class FileSystem

Throws:

IOException

getXAttr

public byte[] getXAttr(Path p,
                       String name)
                throws IOException

Overrides:

getXAttr in class FileSystem

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(Path p)
                             throws IOException

Overrides:

getXAttrs in class FileSystem

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(Path p,
                                    List<String> names)
                             throws IOException

Overrides:

getXAttrs in class FileSystem

Throws:

IOException

listXAttrs

public List<String> listXAttrs(Path p)
                        throws IOException

Overrides:

listXAttrs in class FileSystem

Throws:

IOException

removeXAttr

public void removeXAttr(Path p,
                        String name)
                 throws IOException

Overrides:

removeXAttr in class FileSystem

Throws:

IOException

setOwner

public void setOwner(Path p,
                     String owner,
                     String group)
              throws IOException

Overrides:

setOwner in class FileSystem

Throws:

IOException

setPermission

public void setPermission(Path p,
                          FsPermission permission)
                   throws IOException

Overrides:

setPermission in class FileSystem

Throws:

IOException

modifyAclEntries

public void modifyAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Overrides:

modifyAclEntries in class FileSystem

Throws:

IOException

removeAclEntries

public void removeAclEntries(Path path,
                             List<AclEntry> aclSpec)
                      throws IOException

Overrides:

removeAclEntries in class FileSystem

Throws:

IOException

removeDefaultAcl

public void removeDefaultAcl(Path path)
                      throws IOException

Overrides:

removeDefaultAcl in class FileSystem

Throws:

IOException

removeAcl

public void removeAcl(Path path)
               throws IOException

Overrides:

removeAcl in class FileSystem

Throws:

IOException

setAcl

public void setAcl(Path p,
                   List<AclEntry> aclSpec)
            throws IOException

Overrides:

setAcl in class FileSystem

Throws:

IOException

createSnapshot

public Path createSnapshot(Path path,
                           String snapshotName)
                    throws IOException

Overrides:

createSnapshot in class FileSystem

Throws:

IOException

deleteSnapshot

public void deleteSnapshot(Path path,
                           String snapshotName)
                    throws IOException

Overrides:

deleteSnapshot in class FileSystem

Throws:

IOException

renameSnapshot

public void renameSnapshot(Path path,
                           String snapshotOldName,
                           String snapshotNewName)
                    throws IOException

Overrides:

renameSnapshot in class FileSystem

Throws:

IOException

setReplication

public boolean setReplication(Path p,
                              short replication)
                       throws IOException

Overrides:

setReplication in class FileSystem

Throws:

IOException

setTimes

public void setTimes(Path p,
                     long mtime,
                     long atime)
              throws IOException

Overrides:

setTimes in class FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class FileSystem

getDefaultReplication

public short getDefaultReplication()

Overrides:

getDefaultReplication in class FileSystem

concat

public void concat(Path trg,
                   Path[] srcs)
            throws IOException

Overrides:

concat in class FileSystem

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                                 FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 Progressable progress)
                          throws IOException

Specified by:

append in class FileSystem

Throws:

IOException

truncate

public boolean truncate(Path f,
                        long newLength)
                 throws IOException

Overrides:

truncate in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in class FileSystem

Throws:

IOException

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Specified by:

listStatus in class FileSystem

Throws:

IOException

getDelegationToken

public org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier> getDelegationToken(String renewer)
                                                                                                                                      throws IOException

Overrides:

getDelegationToken in class FileSystem

Throws:

IOException

getRenewToken

public org.apache.hadoop.security.token.Token<?> getRenewToken()

Specified by:

getRenewToken in interface DelegationTokenRenewer.Renewable

setDelegationToken

public <T extends org.apache.hadoop.security.token.TokenIdentifier> void setDelegationToken(org.apache.hadoop.security.token.Token<T> token)

Specified by:

setDelegationToken in interface DelegationTokenRenewer.Renewable

renewDelegationToken

public long renewDelegationToken(org.apache.hadoop.security.token.Token<?> token)
                          throws IOException

Throws:

IOException

cancelDelegationToken

public void cancelDelegationToken(org.apache.hadoop.security.token.Token<?> token)
                           throws IOException

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus status,
                                             long offset,
                                             long length)
                                      throws IOException

Overrides:

getFileBlockLocations in class FileSystem

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(Path p,
                                             long offset,
                                             long length)
                                      throws IOException

Overrides:

getFileBlockLocations in class FileSystem

Throws:

IOException

access

public void access(Path path,
                   FsAction mode)
            throws IOException

Overrides:

access in class FileSystem

Throws:

IOException

getContentSummary

public ContentSummary getContentSummary(Path p)
                                 throws IOException

Overrides:

getContentSummary in class FileSystem

Throws:

IOException

getFileChecksum

public MD5MD5CRC32FileChecksum getFileChecksum(Path p)
                                        throws IOException

Overrides:

getFileChecksum in class FileSystem

Throws:

IOException

getCanonicalServiceName

public String getCanonicalServiceName()

Overrides:

getCanonicalServiceName in class FileSystem

setRetryPolicy

public void setRetryPolicy(org.apache.hadoop.io.retry.RetryPolicy rp)