org.apache.hadoop.contrib.bkjournal

Class BookKeeperJournalManager

java.lang.Object

org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager

All Implemented Interfaces:

Closeable, AutoCloseable, Storage.FormatConfirmable, JournalManager

public class BookKeeperJournalManager
extends Object
implements JournalManager

BookKeeper Journal Manager To use, add the following to hdfs-site.xml.

 
 <property>
   <name>dfs.namenode.edits.dir</name>
   <value>bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal</value>
 </property>

 <property>
   <name>dfs.namenode.edits.journal-plugin.bookkeeper</name>
   <value>org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager</value>
 </property>
 
 

The URI format for bookkeeper is bookkeeper://[zkEnsemble]/[rootZnode] [zookkeeper ensemble] is a list of semi-colon separated, zookeeper host:port pairs. In the example above there are 3 servers, in the ensemble, zk1, zk2 & zk3, each one listening on port 2181. [root znode] is the path of the zookeeper znode, under which the editlog information will be stored. Other configuration options are:

• dfs.namenode.bookkeeperjournal.output-buffer-size Number of bytes a bookkeeper journal stream will buffer before forcing a flush. Default is 1024.
• dfs.namenode.bookkeeperjournal.ensemble-size Number of bookkeeper servers in edit log ledger ensembles. This is the number of bookkeeper servers which need to be available for the ledger to be writable. Default is 3.
• dfs.namenode.bookkeeperjournal.quorum-size Number of bookkeeper servers in the write quorum. This is the number of bookkeeper servers which must have acknowledged the write of an entry before it is considered written. Default is 2.
• dfs.namenode.bookkeeperjournal.digestPw Password to use when creating ledgers.
• dfs.namenode.bookkeeperjournal.zk.session.timeout Session timeout for Zookeeper client from BookKeeper Journal Manager. Hadoop recommends that, this value should be less than the ZKFC session timeout value. Default value is 3000.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.namenode.JournalManager

JournalManager.CorruptionException

Field Summary

Fields

Modifier and Type	Field and Description
static String	BKJM_BOOKKEEPER_ACK_QUORUM_SIZE
static int	BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_DEFAULT
static String	BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_SEC
static String	BKJM_BOOKKEEPER_DIGEST_PW
static String	BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT
static String	BKJM_BOOKKEEPER_ENSEMBLE_SIZE
static int	BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT
static String	BKJM_BOOKKEEPER_QUORUM_SIZE
static int	BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT
static int	BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_DEFAULT
static String	BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_SEC
static int	BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_DEFAULT
static String	BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_MS
static String	BKJM_OUTPUT_BUFFER_SIZE
static int	BKJM_OUTPUT_BUFFER_SIZE_DEFAULT
static String	BKJM_ZK_LEDGERS_AVAILABLE_PATH
static String	BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT
static String	BKJM_ZK_SESSION_TIMEOUT
static int	BKJM_ZK_SESSION_TIMEOUT_DEFAULT

Constructor Summary

Constructors

Constructor and Description
BookKeeperJournalManager(Configuration conf, URI uri, NamespaceInfo nsInfo) Construct a Bookkeeper journal manager.

Method Summary

Modifier and Type	Method and Description
boolean	canRollBack(StorageInfo storage, StorageInfo prevStorage, int targetLayoutVersion)
void	close()
void	discardSegments(long startTxId)
void	doFinalize()
void	doPreUpgrade()
void	doRollback()
void	doUpgrade(Storage storage)
void	finalizeLogSegment(long firstTxId, long lastTxId) Finalize a log segment.
void	format(NamespaceInfo ns)
long	getJournalCTime()
boolean	hasSomeData()
void	purgeLogsOlderThan(long minTxIdToKeep)
void	recoverUnfinalizedSegments()
void	selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk)
void	setOutputBufferCapacity(int size) Set the amount of memory that this stream should use to buffer edits.
EditLogOutputStream	startLogSegment(long txId, int layoutVersion) Start a new log segment in a BookKeeper ledger.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hdfs.server.common.Storage.FormatConfirmable

toString

Field Detail

BKJM_OUTPUT_BUFFER_SIZE

public static final String BKJM_OUTPUT_BUFFER_SIZE

See Also:

Constant Field Values

BKJM_OUTPUT_BUFFER_SIZE_DEFAULT

public static final int BKJM_OUTPUT_BUFFER_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ENSEMBLE_SIZE

public static final String BKJM_BOOKKEEPER_ENSEMBLE_SIZE

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT

public static final int BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_QUORUM_SIZE

public static final String BKJM_BOOKKEEPER_QUORUM_SIZE

See Also:

Constant Field Values

BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT

public static final int BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_DIGEST_PW

public static final String BKJM_BOOKKEEPER_DIGEST_PW

See Also:

Constant Field Values

BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT

public static final String BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT

See Also:

Constant Field Values

BKJM_ZK_SESSION_TIMEOUT

public static final String BKJM_ZK_SESSION_TIMEOUT

See Also:

Constant Field Values

BKJM_ZK_SESSION_TIMEOUT_DEFAULT

public static final int BKJM_ZK_SESSION_TIMEOUT_DEFAULT

See Also:

Constant Field Values

BKJM_ZK_LEDGERS_AVAILABLE_PATH

public static final String BKJM_ZK_LEDGERS_AVAILABLE_PATH

See Also:

Constant Field Values

BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT

public static final String BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_MS

public static final String BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_MS

See Also:

Constant Field Values

BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_DEFAULT

public static final int BKJM_BOOKKEEPER_SPECULATIVE_READ_TIMEOUT_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_SEC

public static final String BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_SEC

See Also:

Constant Field Values

BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_DEFAULT

public static final int BKJM_BOOKKEEPER_READ_ENTRY_TIMEOUT_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ACK_QUORUM_SIZE

public static final String BKJM_BOOKKEEPER_ACK_QUORUM_SIZE

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_SEC

public static final String BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_SEC

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_DEFAULT

public static final int BKJM_BOOKKEEPER_ADD_ENTRY_TIMEOUT_DEFAULT

See Also:

Constant Field Values

Constructor Detail

BookKeeperJournalManager

public BookKeeperJournalManager(Configuration conf,
                                URI uri,
                                NamespaceInfo nsInfo)
                         throws IOException

Construct a Bookkeeper journal manager.

Throws:

IOException

Method Detail

format

public void format(NamespaceInfo ns)
            throws IOException

Specified by:

format in interface JournalManager

Throws:

IOException

hasSomeData

public boolean hasSomeData()
                    throws IOException

Specified by:

hasSomeData in interface Storage.FormatConfirmable

Throws:

IOException

startLogSegment

public EditLogOutputStream startLogSegment(long txId,
                                           int layoutVersion)
                                    throws IOException

Start a new log segment in a BookKeeper ledger. First ensure that we have the write lock for this journal. Then create a ledger and stream based on that ledger. The ledger id is written to the inprogress znode, so that in the case of a crash, a recovery process can find the ledger we were writing to when we crashed.

Specified by:

startLogSegment in interface JournalManager

Parameters:

txId - First transaction id to be written to the stream

Throws:

IOException

finalizeLogSegment

public void finalizeLogSegment(long firstTxId,
                               long lastTxId)
                        throws IOException

Finalize a log segment. If the journal manager is currently writing to a ledger, ensure that this is the ledger of the log segment being finalized. Otherwise this is the recovery case. In the recovery case, ensure that the firstTxId of the ledger matches firstTxId for the segment we are trying to finalize.

Specified by:

finalizeLogSegment in interface JournalManager

Throws:

IOException

selectInputStreams

public void selectInputStreams(Collection<EditLogInputStream> streams,
                               long fromTxId,
                               boolean inProgressOk)
                        throws IOException

Throws:

IOException

recoverUnfinalizedSegments

public void recoverUnfinalizedSegments()
                                throws IOException

Specified by:

recoverUnfinalizedSegments in interface JournalManager

Throws:

IOException

purgeLogsOlderThan

public void purgeLogsOlderThan(long minTxIdToKeep)
                        throws IOException

Throws:

IOException

discardSegments

public void discardSegments(long startTxId)
                     throws IOException

Specified by:

discardSegments in interface JournalManager

Throws:

IOException

doPreUpgrade

public void doPreUpgrade()
                  throws IOException

Specified by:

doPreUpgrade in interface JournalManager

Throws:

IOException

doUpgrade

public void doUpgrade(Storage storage)
               throws IOException

Specified by:

doUpgrade in interface JournalManager

Throws:

IOException

getJournalCTime

public long getJournalCTime()
                     throws IOException

Specified by:

getJournalCTime in interface JournalManager

Throws:

IOException

doFinalize

public void doFinalize()
                throws IOException

Specified by:

doFinalize in interface JournalManager

Throws:

IOException

canRollBack

public boolean canRollBack(StorageInfo storage,
                           StorageInfo prevStorage,
                           int targetLayoutVersion)
                    throws IOException

Specified by:

canRollBack in interface JournalManager

Throws:

IOException

doRollback

public void doRollback()
                throws IOException

Specified by:

doRollback in interface JournalManager

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface JournalManager

Throws:

IOException

setOutputBufferCapacity

public void setOutputBufferCapacity(int size)

Set the amount of memory that this stream should use to buffer edits. Setting this will only affect future output stream. Streams which have currently be created won't be affected.

Specified by:

setOutputBufferCapacity in interface JournalManager