org.apache.hadoop.hdfs.server.datanode

类 DataNode

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hdfs.server.datanode.DataNode

所有已实现的接口:

Runnable, Configurable, ClientDatanodeProtocol, FSConstants, DataNodeMXBean, InterDatanodeProtocol, VersionedProtocol

public class DataNode
extends Configured
implements InterDatanodeProtocol, ClientDatanodeProtocol, FSConstants, Runnable, DataNodeMXBean

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.hadoop.hdfs.protocol.FSConstants

FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction

字段概要

字段

限定符和类型	字段和说明
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner	blockScanner
Daemon	blockScannerThread
FSDatasetInterface	data
static String	DATA_DIR_KEY
static String	DATA_DIR_PERMISSION_KEY
static String	DN_CLIENTTRACE_FORMAT
DatanodeRegistration	dnRegistration
static String	EMPTY_DEL_HINT
Server	ipcServer
static org.apache.commons.logging.Log	LOG
DatanodeProtocol	namenode
static int	PKT_HEADER_LEN Header size for a packet

从接口继承的字段 org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

versionID

从接口继承的字段 org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

versionID

从接口继承的字段 org.apache.hadoop.hdfs.protocol.FSConstants

BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_DATA_SOCKET_SIZE, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE

方法概要

方法

限定符和类型	方法和说明
protected void	checkDiskError() Check if there is a disk failure and if so, handle the error
protected void	checkDiskError(Exception e) Check if there is no space in disk
static DataNode	createDataNode(String[] args, Configuration conf) Instantiate & Start a single datanode daemon and wait for it to finish.
static DataNode	createDataNode(String[] args, Configuration conf, SecureDataNodeStarter.SecureResources resources) Instantiate & Start a single datanode daemon and wait for it to finish.
static InterDatanodeProtocol	createInterDataNodeProtocolProxy(DatanodeInfo info, Configuration conf, int socketTimeout, boolean connectToDnViaHostname)
static InetSocketAddress	createSocketAddr(String target) 已过时。
Long	getBalancerBandwidth() Get current value of the max balancer bandwidth in bytes per second.
Block	getBlockInfo(Block block) Returns a block object that contains the specified block object from the specified Datanode.
BlockLocalPathInfo	getBlockLocalPathInfo(Block block, Token<BlockTokenIdentifier> token) Retrieves the path names of the block file and metadata file stored on the local file system.
BlockMetaDataInfo	getBlockMetaDataInfo(Block block)
static DataNode	getDataNode() Return the DataNode object
FSDatasetInterface	getFSDataset() This method is used for testing.
String	getHostName()
String	getHttpPort() Gets the http port.
static InetSocketAddress	getInfoAddr(Configuration conf) Determine the http server's effective addr
String	getNamenode() Return the namenode's identifier
InetSocketAddress	getNameNodeAddr()
String	getNamenodeAddress() Gets the namenode IP address.
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
String	getRpcPort() Gets the rpc port.
InetSocketAddress	getSelfAddr()
static InetSocketAddress	getStreamingAddr(Configuration conf)
String	getVersion() Gets the version of Hadoop.
String	getVolumeInfo() Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values
int	getXceiverCount() Number of concurrent xceivers per node.
static DataNode	instantiateDataNode(String[] args, Configuration conf) Instantiate a single datanode object.
static DataNode	instantiateDataNode(String[] args, Configuration conf, SecureDataNodeStarter.SecureResources resources) Instantiate a single datanode object.
static void	main(String[] args)
static DataNode	makeInstance(String[] dataDirs, Configuration conf, SecureDataNodeStarter.SecureResources resources) Make an instance of DataNode after ensuring that at least one of the given data directories (and their parent directories, if necessary) can be created.
protected Socket	newSocket() Creates either NIO or regular depending on socketWriteTimeout.
protected void	notifyNamenodeReceivedBlock(Block block, String delHint)
void	offerService() Main loop for the DataNode.
LocatedBlock	recoverBlock(Block block, boolean keepLength, DatanodeInfo[] targets) Start generation-stamp recovery for specified block
Daemon	recoverBlocks(Block[] blocks, DatanodeInfo[][] targets)
void	run() No matter what kind of exception we get, keep retrying to offerService().
static void	runDatanodeDaemon(DataNode dn) Start a single datanode daemon and wait for it to finish.
void	scheduleBlockReport(long delay) This methods arranges for the data node to send the block report at the next heartbeat.
static void	secureMain(String[] args, SecureDataNodeStarter.SecureResources resources)
static void	setNewStorageID(DatanodeRegistration dnReg)
void	shutdown() Shut down this instance of the datanode.
BlockRecoveryInfo	startBlockRecovery(Block block) Begin recovery on a block - this interrupts writers and returns the necessary metadata for recovery to begin.
String	toString()
void	unRegisterMXBean()
void	updateBlock(Block oldblock, Block newblock, boolean finalize) Update the block to the new generation stamp and length.

从类继承的方法 org.apache.hadoop.conf.Configured

getConf, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

LOG

public static final org.apache.commons.logging.Log LOG

DN_CLIENTTRACE_FORMAT

public static final String DN_CLIENTTRACE_FORMAT

另请参阅:

常量字段值

namenode

public DatanodeProtocol namenode

data

public FSDatasetInterface data

dnRegistration

public DatanodeRegistration dnRegistration

EMPTY_DEL_HINT

public static final String EMPTY_DEL_HINT

另请参阅:

常量字段值

blockScanner

public org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner

blockScannerThread

public Daemon blockScannerThread

DATA_DIR_KEY

public static final String DATA_DIR_KEY

另请参阅:

常量字段值

DATA_DIR_PERMISSION_KEY

public static final String DATA_DIR_PERMISSION_KEY

另请参阅:

常量字段值

ipcServer

public Server ipcServer

PKT_HEADER_LEN

public static final int PKT_HEADER_LEN

Header size for a packet

另请参阅:

常量字段值

方法详细资料

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)
                                          throws IOException

已过时。

Use NetUtils.createSocketAddr(String) instead.

抛出:

IOException

unRegisterMXBean

public void unRegisterMXBean()

getInfoAddr

public static InetSocketAddress getInfoAddr(Configuration conf)

Determine the http server's effective addr

newSocket

protected Socket newSocket()
                    throws IOException

Creates either NIO or regular depending on socketWriteTimeout.

抛出:

IOException

getDataNode

public static DataNode getDataNode()

Return the DataNode object

createInterDataNodeProtocolProxy

public static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeInfo info,
                                                     Configuration conf,
                                                     int socketTimeout,
                                                     boolean connectToDnViaHostname)
                                                              throws IOException

抛出:

IOException

getNameNodeAddr

public InetSocketAddress getNameNodeAddr()

getSelfAddr

public InetSocketAddress getSelfAddr()

getNamenode

public String getNamenode()

Return the namenode's identifier

setNewStorageID

public static void setNewStorageID(DatanodeRegistration dnReg)

shutdown

public void shutdown()

Shut down this instance of the datanode. Returns only after shutdown is complete. This method can only be called by the offerService thread. Otherwise, deadlock might occur.

checkDiskError

protected void checkDiskError(Exception e)
                       throws IOException

Check if there is no space in disk

参数:

e - that caused this checkDiskError call

抛出:

IOException

checkDiskError

protected void checkDiskError()

Check if there is a disk failure and if so, handle the error

getXceiverCount

public int getXceiverCount()

Number of concurrent xceivers per node.

指定者:

getXceiverCount 在接口中 DataNodeMXBean

offerService

public void offerService()
                  throws Exception

Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

抛出:

Exception

notifyNamenodeReceivedBlock

protected void notifyNamenodeReceivedBlock(Block block,
                               String delHint)

run

public void run()

No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

指定者:

run 在接口中 Runnable

runDatanodeDaemon

public static void runDatanodeDaemon(DataNode dn)
                              throws IOException

Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

抛出:

IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           Configuration conf)
                                    throws IOException

Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

抛出:

IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           Configuration conf,
                           SecureDataNodeStarter.SecureResources resources)
                                    throws IOException

Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

参数:

resources - Secure resources needed to run under Kerberos

抛出:

IOException

createDataNode

public static DataNode createDataNode(String[] args,
                      Configuration conf)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

抛出:

IOException

createDataNode

public static DataNode createDataNode(String[] args,
                      Configuration conf,
                      SecureDataNodeStarter.SecureResources resources)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting. LimitedPrivate for creating secure datanodes

抛出:

IOException

makeInstance

public static DataNode makeInstance(String[] dataDirs,
                    Configuration conf,
                    SecureDataNodeStarter.SecureResources resources)
                             throws IOException

Make an instance of DataNode after ensuring that at least one of the given data directories (and their parent directories, if necessary) can be created.

参数:

dataDirs - List of directories, where the new DataNode instance should keep its files.

conf - Configuration instance to use.

resources - Secure resources needed to run under Kerberos

返回:

DataNode instance for given list of data dirs and conf, or null if no directory from this directory list can be created.

抛出:

IOException

toString

public String toString()

覆盖:

toString 在类中 Object

scheduleBlockReport

public void scheduleBlockReport(long delay)

This methods arranges for the data node to send the block report at the next heartbeat.

getFSDataset

public FSDatasetInterface getFSDataset()

This method is used for testing. Examples are adding and deleting blocks directly. The most common usage will be when the data node's storage is similated.

返回:

the fsdataset that stores the blocks

secureMain

public static void secureMain(String[] args,
              SecureDataNodeStarter.SecureResources resources)

main

public static void main(String[] args)

getBlockMetaDataInfo

public BlockMetaDataInfo getBlockMetaDataInfo(Block block)
                                       throws IOException

指定者:

getBlockMetaDataInfo 在接口中 InterDatanodeProtocol

返回:

the BlockMetaDataInfo of a block; null if the block is not found

抛出:

IOException

startBlockRecovery

public BlockRecoveryInfo startBlockRecovery(Block block)
                                     throws IOException

从接口复制的说明: InterDatanodeProtocol

Begin recovery on a block - this interrupts writers and returns the necessary metadata for recovery to begin.

指定者:

startBlockRecovery 在接口中 InterDatanodeProtocol

返回:

the BlockRecoveryInfo for a block

抛出:

IOException

recoverBlocks

public Daemon recoverBlocks(Block[] blocks,
                   DatanodeInfo[][] targets)

updateBlock

public void updateBlock(Block oldblock,
               Block newblock,
               boolean finalize)
                 throws IOException

Update the block to the new generation stamp and length.

指定者:

updateBlock 在接口中 InterDatanodeProtocol

抛出:

IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                      long clientVersion)
                        throws IOException

Return protocol version corresponding to protocol interface.

指定者:

getProtocolVersion 在接口中 VersionedProtocol

参数:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

返回:

the version that the server will speak

抛出:

IOException

getBlockLocalPathInfo

public BlockLocalPathInfo getBlockLocalPathInfo(Block block,
                                       Token<BlockTokenIdentifier> token)
                                         throws IOException

从接口复制的说明: ClientDatanodeProtocol

Retrieves the path names of the block file and metadata file stored on the local file system. In order for this method to work, one of the following should be satisfied:

• The client user must be configured at the datanode to be able to use this method.
• When security is enabled, kerberos authentication must be used to connect to the datanode.

指定者:

getBlockLocalPathInfo 在接口中 ClientDatanodeProtocol

参数:

block - the specified block on the local datanode

token - the block access token.

返回:

the BlockLocalPathInfo of a block

抛出:

IOException - on error

recoverBlock

public LocatedBlock recoverBlock(Block block,
                        boolean keepLength,
                        DatanodeInfo[] targets)
                          throws IOException

Start generation-stamp recovery for specified block

指定者:

recoverBlock 在接口中 ClientDatanodeProtocol

参数:

block - the specified block

keepLength - keep the block length

targets - the list of possible locations of specified block

返回:

either a new generation stamp, or the original generation stamp. Regardless of whether a new generation stamp is returned, a newly generated access token is returned as part of the return value.

抛出:

IOException

getBlockInfo

public Block getBlockInfo(Block block)
                   throws IOException

Returns a block object that contains the specified block object from the specified Datanode.

指定者:

getBlockInfo 在接口中 ClientDatanodeProtocol

参数:

block - the specified block

返回:

the Block object from the specified Datanode

抛出:

IOException - if the block does not exist

getStreamingAddr

public static InetSocketAddress getStreamingAddr(Configuration conf)

getHostName

public String getHostName()

指定者:

getHostName 在接口中 DataNodeMXBean

返回:

the host name

getVersion

public String getVersion()

从接口复制的说明: DataNodeMXBean

Gets the version of Hadoop.

指定者:

getVersion 在接口中 DataNodeMXBean

返回:

the version of Hadoop

getRpcPort

public String getRpcPort()

从接口复制的说明: DataNodeMXBean

Gets the rpc port.

指定者:

getRpcPort 在接口中 DataNodeMXBean

返回:

the rpc port

getHttpPort

public String getHttpPort()

从接口复制的说明: DataNodeMXBean

Gets the http port.

指定者:

getHttpPort 在接口中 DataNodeMXBean

返回:

the http port

getNamenodeAddress

public String getNamenodeAddress()

从接口复制的说明: DataNodeMXBean

Gets the namenode IP address.

指定者:

getNamenodeAddress 在接口中 DataNodeMXBean

返回:

the namenode IP address

getVolumeInfo

public String getVolumeInfo()

Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values

指定者:

getVolumeInfo 在接口中 DataNodeMXBean

返回:

the volume info

getBalancerBandwidth

public Long getBalancerBandwidth()

Get current value of the max balancer bandwidth in bytes per second.

返回:

bandwidth Blanacer bandwidth in bytes per second for this datanode.