org.apache.hadoop.hbase.regionserver.wal

Class HLogSplitter

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.HLogSplitter

@InterfaceAudience.Private
public class HLogSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HLogSplitter.MutationReplay A struct used by getMutationsFromWALEntry

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected boolean	distributedLogReplay
protected String	failedServerName
protected org.apache.hadoop.fs.FileSystem	fs
protected Map<String,Long>	lastFlushedSequenceIds
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
protected org.apache.hadoop.fs.Path	rootDir
protected LastSequenceId	sequenceIdChecker
protected AtomicReference<Throwable>	thrown

Method Summary

Methods

Modifier and Type	Method and Description
protected HLog.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path logfile, org.apache.hadoop.conf.Configuration conf) Create a new HLog.Writer for writing log splits.
static void	finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf) Completes the work done by splitLogFile by archiving logs
static List<HLogSplitter.MutationReplay>	getMutationsFromWALEntry(AdminProtos.WALEntry entry, CellScanner cells, Pair<HLogKey,WALEdit> logEntry, boolean addLogReplayTag) This function is used to construct mutations from a WALEntry.
protected HLog.Reader	getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus file, org.apache.hadoop.conf.Configuration conf, boolean skipErrors, CancelableProgressable reporter) Create a new HLog.Reader for reading logs to split.
protected HLog.Reader	getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path curLogFile, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter) Create a new HLog.Reader for reading logs to split.
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
static boolean	splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, ZooKeeperWatcher zkw, ZooKeeperProtos.SplitLogTask.RecoveryMode mode) Splits a HLog file into region's recovered-edits directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rootDir

protected final org.apache.hadoop.fs.Path rootDir

fs

protected final org.apache.hadoop.fs.FileSystem fs

conf

protected final org.apache.hadoop.conf.Configuration conf

thrown

protected AtomicReference<Throwable> thrown

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

distributedLogReplay

protected boolean distributedLogReplay

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

failedServerName

protected String failedServerName

Method Detail

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                   org.apache.hadoop.fs.FileStatus logfile,
                   org.apache.hadoop.fs.FileSystem fs,
                   org.apache.hadoop.conf.Configuration conf,
                   CancelableProgressable reporter,
                   LastSequenceId idChecker,
                   ZooKeeperWatcher zkw,
                   ZooKeeperProtos.SplitLogTask.RecoveryMode mode)
                            throws IOException

Splits a HLog file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

If the log file has N regions then N recovered.edits files will be produced.

Parameters:

rootDir -

logfile -

fs -

conf -

reporter -

idChecker -

zkw - ZooKeeperWatcher if it's null, we will back to the old-style log splitting where we dump out recoved.edits files for regions to replay on.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir,
                                    org.apache.hadoop.fs.Path logDir,
                                    org.apache.hadoop.fs.Path oldLogDir,
                                    org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.conf.Configuration conf)
                                             throws IOException

Throws:

IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:

logfile -

conf -

Throws:

IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.FileStatus file,
                    org.apache.hadoop.conf.Configuration conf,
                    boolean skipErrors,
                    CancelableProgressable reporter)
                         throws IOException,
                                org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

Create a new HLog.Reader for reading logs to split.

Parameters:

fs -

file -

conf -

Returns:

A new Reader instance

Throws:

IOException

CorruptedLogFileException

org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

createWriter

protected HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path logfile,
                       org.apache.hadoop.conf.Configuration conf)
                            throws IOException

Create a new HLog.Writer for writing log splits.

Throws:

IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path curLogFile,
                    org.apache.hadoop.conf.Configuration conf,
                    CancelableProgressable reporter)
                         throws IOException

Create a new HLog.Reader for reading logs to split.

Throws:

IOException

getMutationsFromWALEntry

public static List<HLogSplitter.MutationReplay> getMutationsFromWALEntry(AdminProtos.WALEntry entry,
                                                         CellScanner cells,
                                                         Pair<HLogKey,WALEdit> logEntry,
                                                         boolean addLogReplayTag)
                                                                  throws IOException

This function is used to construct mutations from a WALEntry. It also reconstructs HLogKey & WALEdit from the passed in WALEntry

Parameters:

entry -

cells -

logEntry - pair of HLogKey and WALEdit instance stores HLogKey and WALEdit instances extracted from the passed in WALEntry.

addLogReplayTag -

Returns:

list of Pair to be replayed

Throws:

IOException