org.apache.hadoop.mapred

Class ShuffleHandler

java.lang.Object

org.apache.hadoop.yarn.service.AbstractService

org.apache.hadoop.mapred.ShuffleHandler

All Implemented Interfaces:

org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService, org.apache.hadoop.yarn.service.Service

public class ShuffleHandler
extends org.apache.hadoop.yarn.service.AbstractService
implements org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.yarn.service.Service

org.apache.hadoop.yarn.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_SHUFFLE_CONNECTIONS
static boolean	DEFAULT_SHUFFLE_MANAGE_OS_CACHE
static int	DEFAULT_SHUFFLE_PORT
static int	DEFAULT_SHUFFLE_READAHEAD_BYTES
static int	DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE
static String	MAPREDUCE_SHUFFLE_SERVICEID
static String	MAX_SHUFFLE_CONNECTIONS
protected org.apache.hadoop.mapred.ShuffleHandler.HttpPipelineFactory	pipelineFact
static String	SHUFFLE_MANAGE_OS_CACHE
static String	SHUFFLE_PORT_CONFIG_KEY
static String	SHUFFLE_READAHEAD_BYTES
static String	SUFFLE_SSL_FILE_BUFFER_SIZE_KEY

Constructor Summary

Constructors

Constructor and Description
ShuffleHandler()

Method Summary

Methods

Modifier and Type	Method and Description
static int	deserializeMetaData(ByteBuffer meta) A helper function to deserialize the metadata returned by ShuffleHandler.
ByteBuffer	getMeta()
protected org.apache.hadoop.mapred.ShuffleHandler.Shuffle	getShuffle(org.apache.hadoop.conf.Configuration conf)
void	init(org.apache.hadoop.conf.Configuration conf)
void	initApp(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, ByteBuffer secret)
static ByteBuffer	serializeMetaData(int port) Serialize the shuffle port into a ByteBuffer for use later on.
static ByteBuffer	serializeServiceData(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier> jobToken) A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData.
void	start()
void	stop()
void	stopApp(org.apache.hadoop.yarn.api.records.ApplicationId appId)

Methods inherited from class org.apache.hadoop.yarn.service.AbstractService

getConfig, getName, getServiceState, getStartTime, register, unregister

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.service.Service

getConfig, getName, getServiceState, getStartTime, register, unregister

Field Detail

SHUFFLE_MANAGE_OS_CACHE

public static final String SHUFFLE_MANAGE_OS_CACHE

See Also:

Constant Field Values

DEFAULT_SHUFFLE_MANAGE_OS_CACHE

public static final boolean DEFAULT_SHUFFLE_MANAGE_OS_CACHE

See Also:

Constant Field Values

SHUFFLE_READAHEAD_BYTES

public static final String SHUFFLE_READAHEAD_BYTES

See Also:

Constant Field Values

DEFAULT_SHUFFLE_READAHEAD_BYTES

public static final int DEFAULT_SHUFFLE_READAHEAD_BYTES

See Also:

Constant Field Values

pipelineFact

protected org.apache.hadoop.mapred.ShuffleHandler.HttpPipelineFactory pipelineFact

MAPREDUCE_SHUFFLE_SERVICEID

public static final String MAPREDUCE_SHUFFLE_SERVICEID

See Also:

Constant Field Values

SHUFFLE_PORT_CONFIG_KEY

public static final String SHUFFLE_PORT_CONFIG_KEY

See Also:

Constant Field Values

DEFAULT_SHUFFLE_PORT

public static final int DEFAULT_SHUFFLE_PORT

See Also:

Constant Field Values

SUFFLE_SSL_FILE_BUFFER_SIZE_KEY

public static final String SUFFLE_SSL_FILE_BUFFER_SIZE_KEY

See Also:

Constant Field Values

DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE

public static final int DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE

See Also:

Constant Field Values

MAX_SHUFFLE_CONNECTIONS

public static final String MAX_SHUFFLE_CONNECTIONS

See Also:

Constant Field Values

DEFAULT_MAX_SHUFFLE_CONNECTIONS

public static final int DEFAULT_MAX_SHUFFLE_CONNECTIONS

See Also:

Constant Field Values

Constructor Detail

ShuffleHandler

public ShuffleHandler()

Method Detail

serializeMetaData

public static ByteBuffer serializeMetaData(int port)
                                    throws IOException

Serialize the shuffle port into a ByteBuffer for use later on.

Parameters:

port - the port to be sent to the ApplciationMaster

Returns:

the serialized form of the port.

Throws:

IOException

deserializeMetaData

public static int deserializeMetaData(ByteBuffer meta)
                               throws IOException

A helper function to deserialize the metadata returned by ShuffleHandler.

Parameters:

meta - the metadata returned by the ShuffleHandler

Returns:

the port the Shuffle Handler is listening on to serve shuffle data.

Throws:

IOException

serializeServiceData

public static ByteBuffer serializeServiceData(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier> jobToken)
                                       throws IOException

A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData.

Parameters:

jobToken - the job token to be used for authentication of shuffle data requests.

Returns:

the serialized version of the jobToken.

Throws:

IOException

initApp

public void initApp(String user,
           org.apache.hadoop.yarn.api.records.ApplicationId appId,
           ByteBuffer secret)

Specified by:

initApp in interface org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService

stopApp

public void stopApp(org.apache.hadoop.yarn.api.records.ApplicationId appId)

Specified by:

stopApp in interface org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService

init

public void init(org.apache.hadoop.conf.Configuration conf)

Specified by:

init in interface org.apache.hadoop.yarn.service.Service

Overrides:

init in class org.apache.hadoop.yarn.service.AbstractService

start

public void start()

Specified by:

start in interface org.apache.hadoop.yarn.service.Service

Overrides:

start in class org.apache.hadoop.yarn.service.AbstractService

stop

public void stop()

Specified by:

stop in interface org.apache.hadoop.yarn.service.Service

Overrides:

stop in class org.apache.hadoop.yarn.service.AbstractService

getMeta

public ByteBuffer getMeta()

Specified by:

getMeta in interface org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService

getShuffle

protected org.apache.hadoop.mapred.ShuffleHandler.Shuffle getShuffle(org.apache.hadoop.conf.Configuration conf)