org.apache.tez.dag.app

Class DAGAppMaster

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.tez.dag.app.DAGAppMaster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

Direct Known Subclasses:

LocalDAGAppMaster

public class DAGAppMaster
extends org.apache.hadoop.service.AbstractService

The Tez DAG Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. Tez DAG AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	DAGAppMaster.DAGAppMasterShutdownHandler

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	isLastAMRetry
static int	SHUTDOWN_HOOK_PRIORITY Priority of the DAGAppMaster shutdown hook.
protected DAGAppMaster.DAGAppMasterShutdownHandler	shutdownHandler

Constructor Summary

Constructors

Constructor and Description

DAGAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime, boolean isSession, String workingDirectory, String[] localDirs, String[] logDirs, String clientVersion, org.apache.hadoop.security.Credentials credentials, String jobUserName, DAGProtos.AMPluginDescriptorProto pluginDescriptorProto)

Method Summary

Modifier and Type	Method and Description
protected void	addIfService(Object object, boolean addDispatcher)
protected void	addIfServiceDependency(Object object, Object dependency)
protected ContainerHeartbeatHandler	createContainerHeartbeatHandler(AppContext context, org.apache.hadoop.conf.Configuration conf)
protected ContainerLauncherManager	createContainerLauncherManager(List<NamedEntityDescriptor> containerLauncherDescriptors, boolean isLocal)
protected ContainerSignatureMatcher	createContainerSignatureMatcher()
protected DAG	createDAG(DAGProtos.DAGPlan dagPB)
protected AsyncDispatcher	createDispatcher()
protected HistoryEventHandler	createHistoryEventHandler(AppContext appContext)
protected DAGAppMaster.DAGAppMasterShutdownHandler	createShutdownHandler()
protected TaskCommunicatorManagerInterface	createTaskCommunicatorManager(AppContext context, TaskHeartbeatHandler thh, ContainerHeartbeatHandler chh, List<NamedEntityDescriptor> entityDescriptors)
protected TaskHeartbeatHandler	createTaskHeartbeatHandler(AppContext context, org.apache.hadoop.conf.Configuration conf)
protected TaskSchedulerManager	createTaskSchedulerManager(List<NamedEntityDescriptor> taskSchedulerDescriptors)
ACLManager	getACLManager()
org.apache.hadoop.yarn.api.records.ContainerId	getAppContainerId()
org.apache.hadoop.yarn.api.records.ApplicationId	getAppID()
String	getAppNMHost()
int	getAppNMHttpPort()
int	getAppNMPort()
org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptID()
ContainerLauncherManager	getContainerLauncherManager()
AppContext	getContext()
List<String>	getDiagnostics()
org.apache.hadoop.yarn.event.Dispatcher	getDispatcher()
float	getProgress()
int	getRpcPort()
int	getStartCount()
DAGAppMasterState	getState()
TaskCommunicatorManagerInterface	getTaskCommunicatorManager()
protected TaskSchedulerManager	getTaskSchedulerManager()
String	getWebUIAddress()
protected void	handle(DAGAppMasterEvent event)
static void	initAndStartAppMaster(DAGAppMaster appMaster, org.apache.hadoop.conf.Configuration conf)
protected void	initClientRpcServer()
boolean	isSession()
static void	main(String[] args)
static void	parsePlugin(List<NamedEntityDescriptor> resultList, com.google.common.collect.BiMap<String,Integer> pluginMap, List<DAGProtos.TezNamedEntityDescriptorProto> namedEntityDescriptorProtos, boolean tezYarnEnabled, boolean uberEnabled, UserPayload defaultPayload)
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	serviceStart()
void	serviceStop()
void	setCurrentDAG(DAG currentDAG)
void	shutdownTezAM(String dagKillmessage)
String	submitDAGToAppMaster(DAGProtos.DAGPlan dagPlan, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources)
protected void	sysexit() Exit call.
void	taskAttemptFailed(TezTaskAttemptID attemptID, org.apache.hadoop.yarn.api.records.NodeId nodeId)
void	tryKillDAG(DAG dag, String message)
void	vertexComplete(TezVertexID completedVertexID, Set<org.apache.hadoop.yarn.api.records.NodeId> nodesList)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the DAGAppMaster shutdown hook.

See Also:

Constant Field Values

shutdownHandler

protected DAGAppMaster.DAGAppMasterShutdownHandler shutdownHandler

isLastAMRetry

protected boolean isLastAMRetry

Constructor Detail

DAGAppMaster

public DAGAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                    org.apache.hadoop.yarn.api.records.ContainerId containerId,
                    String nmHost,
                    int nmPort,
                    int nmHttpPort,
                    org.apache.hadoop.yarn.util.Clock clock,
                    long appSubmitTime,
                    boolean isSession,
                    String workingDirectory,
                    String[] localDirs,
                    String[] logDirs,
                    String clientVersion,
                    org.apache.hadoop.security.Credentials credentials,
                    String jobUserName,
                    DAGProtos.AMPluginDescriptorProto pluginDescriptorProto)

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

initClientRpcServer

protected void initClientRpcServer()

createShutdownHandler

protected DAGAppMaster.DAGAppMasterShutdownHandler createShutdownHandler()

createTaskSchedulerManager

protected TaskSchedulerManager createTaskSchedulerManager(List<NamedEntityDescriptor> taskSchedulerDescriptors)

createContainerSignatureMatcher

protected ContainerSignatureMatcher createContainerSignatureMatcher()

createDispatcher

protected AsyncDispatcher createDispatcher()

createHistoryEventHandler

protected HistoryEventHandler createHistoryEventHandler(AppContext appContext)

sysexit

protected void sysexit()

Exit call. Just in a function call to enable testing.

getTaskSchedulerManager

protected TaskSchedulerManager getTaskSchedulerManager()

handle

protected void handle(DAGAppMasterEvent event)

setCurrentDAG

public void setCurrentDAG(DAG currentDAG)

getACLManager

public ACLManager getACLManager()

createDAG

protected DAG createDAG(DAGProtos.DAGPlan dagPB)

addIfService

protected void addIfService(Object object,
                            boolean addDispatcher)

addIfServiceDependency

protected void addIfServiceDependency(Object object,
                                      Object dependency)

createTaskCommunicatorManager

protected TaskCommunicatorManagerInterface createTaskCommunicatorManager(AppContext context,
                                                                         TaskHeartbeatHandler thh,
                                                                         ContainerHeartbeatHandler chh,
                                                                         List<NamedEntityDescriptor> entityDescriptors)
                                                                  throws TezException

Throws:

TezException

createTaskHeartbeatHandler

protected TaskHeartbeatHandler createTaskHeartbeatHandler(AppContext context,
                                                          org.apache.hadoop.conf.Configuration conf)

createContainerHeartbeatHandler

protected ContainerHeartbeatHandler createContainerHeartbeatHandler(AppContext context,
                                                                    org.apache.hadoop.conf.Configuration conf)

createContainerLauncherManager

protected ContainerLauncherManager createContainerLauncherManager(List<NamedEntityDescriptor> containerLauncherDescriptors,
                                                                  boolean isLocal)
                                                           throws UnknownHostException,
                                                                  TezException

Throws:

UnknownHostException

TezException

getAppID

public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

getAttemptID

public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

getStartCount

public int getStartCount()

getContext

public AppContext getContext()

getDispatcher

public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

getContainerLauncherManager

public ContainerLauncherManager getContainerLauncherManager()

getTaskCommunicatorManager

public TaskCommunicatorManagerInterface getTaskCommunicatorManager()

getAppContainerId

public org.apache.hadoop.yarn.api.records.ContainerId getAppContainerId()

getAppNMHost

public String getAppNMHost()

getAppNMPort

public int getAppNMPort()

getAppNMHttpPort

public int getAppNMHttpPort()

getRpcPort

public int getRpcPort()

getState

public DAGAppMasterState getState()

getDiagnostics

public List<String> getDiagnostics()

getProgress

public float getProgress()

shutdownTezAM

public void shutdownTezAM(String dagKillmessage)
                   throws TezException

Throws:

TezException

submitDAGToAppMaster

public String submitDAGToAppMaster(DAGProtos.DAGPlan dagPlan,
                                   Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources)
                            throws TezException

Throws:

TezException

tryKillDAG

public void tryKillDAG(DAG dag,
                       String message)
                throws TezException

Throws:

TezException

serviceStart

public void serviceStart()
                  throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

public void serviceStop()
                 throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

isSession

public boolean isSession()

main

public static void main(String[] args)

initAndStartAppMaster

public static void initAndStartAppMaster(DAGAppMaster appMaster,
                                         org.apache.hadoop.conf.Configuration conf)
                                  throws IOException,
                                         InterruptedException

Throws:

IOException

InterruptedException

getWebUIAddress

public String getWebUIAddress()

parsePlugin

public static void parsePlugin(List<NamedEntityDescriptor> resultList,
                               com.google.common.collect.BiMap<String,Integer> pluginMap,
                               List<DAGProtos.TezNamedEntityDescriptorProto> namedEntityDescriptorProtos,
                               boolean tezYarnEnabled,
                               boolean uberEnabled,
                               UserPayload defaultPayload)

vertexComplete

public void vertexComplete(TezVertexID completedVertexID,
                           Set<org.apache.hadoop.yarn.api.records.NodeId> nodesList)

taskAttemptFailed

public void taskAttemptFailed(TezTaskAttemptID attemptID,
                              org.apache.hadoop.yarn.api.records.NodeId nodeId)