org.apache.flink.runtime.jobmanager

Class JobManager

java.lang.Object

org.apache.flink.runtime.jobmanager.JobManager

All Implemented Interfaces:

VersionedProtocol, JobStatusListener, DeploymentManager, AccumulatorProtocol, ChannelLookupProtocol, ExtendedManagementProtocol, InputSplitProviderProtocol, JobManagementProtocol, JobManagerProtocol

public class JobManager
extends Object
implements DeploymentManager, ExtendedManagementProtocol, InputSplitProviderProtocol, JobManagerProtocol, ChannelLookupProtocol, JobStatusListener, AccumulatorProtocol

In Nephele the job manager is the central component for communication with clients, creating schedules for incoming jobs and supervise their execution. A job manager may only exist once in the system and its address must be known the clients. Task managers can discover the job manager by means of an UDP broadcast and afterwards advertise themselves as new workers for tasks.

Constructor Summary

Constructors

Constructor and Description
JobManager(ExecutionMode executionMode)

Method Summary

Methods

Modifier and Type	Method and Description
JobCancelResult	cancelJob(JobID jobID) Requests to cancel the job specified by the given ID.
void	deploy(JobID jobID, Instance instance, List<ExecutionVertex> verticesToBeDeployed) Deploys the list of vertices on the given Instance.
AccumulatorEvent	getAccumulatorResults(JobID jobID) Get the final accumulator results.
Map<String,Accumulator<?,?>>	getAccumulators(JobID jobID)
ArchiveListener	getArchive()
int	getAvailableSlots() Returns the number of available slots among the registered task managers
List<AbstractEvent>	getEvents(JobID jobID) Retrieves the collected events for the job with the given job ID.
InstanceManager	getInstanceManager()
Map<InstanceConnectionInfo,Instance>	getInstances()
JobProgressResult	getJobProgress(JobID jobID) Retrieves the current status of the job specified by the given ID.
ManagementGraph	getManagementGraph(JobID jobID) Returns current ManagementGraph from eventCollector and, if not current, from archive Retrieves the management graph for the job with the given ID.
NetworkTopology	getNetworkTopology(JobID jobID) Retrieves the current network topology for the job with the given ID.
int	getNumberOfTaskTrackers()
List<RecentJobEvent>	getOldJobs()
List<RecentJobEvent>	getRecentJobs() Retrieves a list of jobs which have either running or have been started recently.
IntegerRecord	getRecommendedPollingInterval() Returns the recommended interval in seconds in which a client is supposed to poll for progress information.
static JobManager	initialize(String[] args)
boolean	isShutDown() Tests whether the job manager has been shut down completely.
void	jobStatusHasChanged(ExecutionGraph executionGraph, InternalJobStatus newJobStatus, String optionalMessage) Called when the status of the job with the given {@li
void	killInstance(StringRecord instanceName) Kills the instance with the given name (i.e.
void	killTask(JobID jobID, ManagementVertexID id) Kills the task with the given vertex ID.
void	logBufferUtilization(JobID jobID) Triggers all task managers involved in processing the job with the given job ID to write the utilization of their read and write buffers to their log files.
ConnectionInfoLookupResponse	lookupConnectionInfo(InstanceConnectionInfo caller, JobID jobID, ChannelID sourceChannelID) Retrieves all recipients of a data for the given sourceChannelID.
static void	main(String[] args) Entry point for the program
RegisterTaskManagerResult	registerTaskManager(InstanceConnectionInfo instanceConnectionInfo, HardwareDescription hardwareDescription, IntegerRecord numberOfSlots) Registers a task manager at the JobManager.
void	reportAccumulatorResult(AccumulatorEvent accumulatorEvent) Report accumulators that were collected in a task.
InputSplitWrapper	requestNextInputSplit(JobID jobID, ExecutionVertexID vertexID, IntegerRecord sequenceNumber) Requests the next split to be consumed by the task with the given execution vertex ID.
void	sendHeartbeat(InstanceConnectionInfo instanceConnectionInfo) Sends a heart beat to the job manager.
void	shutdown()
void	startInfoServer() Starts the Jetty Infoserver for the Jobmanager
JobSubmissionResult	submitJob(JobGraph job) Submits the specified job to the job manager.
void	updateTaskExecutionState(TaskExecutionState executionState) Reports an update of a task's execution state to the job manager.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JobManager

public JobManager(ExecutionMode executionMode)
           throws Exception

Throws:

Exception

Method Detail

shutdown

public void shutdown()

main

public static void main(String[] args)

Entry point for the program

Parameters:

args - arguments from the command line

initialize

public static JobManager initialize(String[] args)
                             throws Exception

Throws:

Exception

submitJob

public JobSubmissionResult submitJob(JobGraph job)
                              throws IOException

Description copied from interface: JobManagementProtocol

Submits the specified job to the job manager.

Specified by:

submitJob in interface JobManagementProtocol

Parameters:

job - the job to be executed

Returns:

a protocol of the job submission including the success status

Throws:

IOException - thrown if an error occurred while transmitting the submit request

getInstanceManager

public InstanceManager getInstanceManager()

sendHeartbeat

public void sendHeartbeat(InstanceConnectionInfo instanceConnectionInfo)

Description copied from interface: JobManagerProtocol

Sends a heart beat to the job manager.

Specified by:

sendHeartbeat in interface JobManagerProtocol

Parameters:

instanceConnectionInfo - the information the job manager requires to connect to the instance's task manager

registerTaskManager

public RegisterTaskManagerResult registerTaskManager(InstanceConnectionInfo instanceConnectionInfo,
                                            HardwareDescription hardwareDescription,
                                            IntegerRecord numberOfSlots)

Description copied from interface: JobManagerProtocol

Registers a task manager at the JobManager.

Specified by:

registerTaskManager in interface JobManagerProtocol

Parameters:

instanceConnectionInfo - the information the job manager requires to connect to the instance's task manager

hardwareDescription - a hardware description with details on the instance's compute resources.

Returns:

whether the task manager was successfully registered

updateTaskExecutionState

public void updateTaskExecutionState(TaskExecutionState executionState)
                              throws IOException

Description copied from interface: JobManagerProtocol

Reports an update of a task's execution state to the job manager.

Specified by:

updateTaskExecutionState in interface JobManagerProtocol

Parameters:

executionState - the new task execution state

Throws:

IOException - thrown if an error occurs during this remote procedure call

cancelJob

public JobCancelResult cancelJob(JobID jobID)
                          throws IOException

Description copied from interface: JobManagementProtocol

Requests to cancel the job specified by the given ID.

Specified by:

cancelJob in interface JobManagementProtocol

Parameters:

jobID - the ID of the job

Returns:

a JobCancelResult containing the result of the cancel request

Throws:

IOException - thrown if an error occurred while transmitting the request

getJobProgress

public JobProgressResult getJobProgress(JobID jobID)
                                 throws IOException

Description copied from interface: JobManagementProtocol

Retrieves the current status of the job specified by the given ID. Consecutive calls of this method may result in duplicate events. The caller must take care of this.

Specified by:

getJobProgress in interface JobManagementProtocol

Parameters:

jobID - the ID of the job

Returns:

a JobProgressResult object including the current job progress

Throws:

IOException - thrown if an error occurred while transmitting the request

lookupConnectionInfo

public ConnectionInfoLookupResponse lookupConnectionInfo(InstanceConnectionInfo caller,
                                                JobID jobID,
                                                ChannelID sourceChannelID)

Description copied from interface: ChannelLookupProtocol

Retrieves all recipients of a data for the given sourceChannelID.

Specified by:

lookupConnectionInfo in interface ChannelLookupProtocol

Parameters:

caller - the InstanceConnectionInfo object of the task manager which calls this method

jobID - the ID of the job the channel ID belongs to

sourceChannelID - the ID of the channel to resolve

Returns:

the lookup response containing the connection info and a return code

getManagementGraph

public ManagementGraph getManagementGraph(JobID jobID)
                                   throws IOException

Returns current ManagementGraph from eventCollector and, if not current, from archive Retrieves the management graph for the job with the given ID.

Specified by:

getManagementGraph in interface ExtendedManagementProtocol

Parameters:

jobID - the ID identifying the job

Returns:

the management graph for the job

Throws:

IOException - thrown if an error occurs while retrieving the management graph

getNetworkTopology

public NetworkTopology getNetworkTopology(JobID jobID)
                                   throws IOException

Description copied from interface: ExtendedManagementProtocol

Retrieves the current network topology for the job with the given ID.

Specified by:

getNetworkTopology in interface ExtendedManagementProtocol

Parameters:

jobID - the ID identifying the job

Returns:

the network topology for the job

Throws:

IOException - thrown if an error occurs while retrieving the network topology

getRecommendedPollingInterval

public IntegerRecord getRecommendedPollingInterval()
                                            throws IOException

Description copied from interface: JobManagementProtocol

Returns the recommended interval in seconds in which a client is supposed to poll for progress information.

Specified by:

getRecommendedPollingInterval in interface JobManagementProtocol

Returns:

the interval in seconds

Throws:

IOException - thrown if an error occurred while transmitting the request

getRecentJobs

public List<RecentJobEvent> getRecentJobs()
                                   throws IOException

Description copied from interface: ExtendedManagementProtocol

Retrieves a list of jobs which have either running or have been started recently.

Specified by:

getRecentJobs in interface ExtendedManagementProtocol

Returns:

a (possibly) empty list of recent jobs

Throws:

IOException - thrown if an error occurs while retrieving the job list

getEvents

public List<AbstractEvent> getEvents(JobID jobID)
                              throws IOException

Description copied from interface: ExtendedManagementProtocol

Retrieves the collected events for the job with the given job ID.

Specified by:

getEvents in interface ExtendedManagementProtocol

Parameters:

jobID - the ID of the job to retrieve the events for

Returns:

a (possibly empty) list of events which occurred for that event and which are not older than the query interval

Throws:

IOException - thrown if an error occurs while retrieving the list of events

killTask

public void killTask(JobID jobID,
            ManagementVertexID id)
              throws IOException

Description copied from interface: ExtendedManagementProtocol

Kills the task with the given vertex ID.

Specified by:

killTask in interface ExtendedManagementProtocol

Parameters:

jobID - the ID of the job the vertex to be killed belongs to

id - the vertex ID which identified the task be killed

Throws:

IOException - thrown if an error occurs while transmitting the kill request

killInstance

public void killInstance(StringRecord instanceName)
                  throws IOException

Description copied from interface: ExtendedManagementProtocol

Kills the instance with the given name (i.e. shuts down its task manager).

Specified by:

killInstance in interface ExtendedManagementProtocol

Parameters:

instanceName - the name of the instance to be killed

Throws:

IOException - thrown if an error occurs while transmitting the kill request

isShutDown

public boolean isShutDown()

Tests whether the job manager has been shut down completely.

Returns:

true if the job manager has been shut down completely, false otherwise

jobStatusHasChanged

public void jobStatusHasChanged(ExecutionGraph executionGraph,
                       InternalJobStatus newJobStatus,
                       String optionalMessage)

Description copied from interface: JobStatusListener

Called when the status of the job with the given {@li

Specified by:

jobStatusHasChanged in interface JobStatusListener

Parameters:

executionGraph - the executionGraph representing the job the event belongs to

newJobStatus - the new job status

optionalMessage - an optional message (possibly null) that can be attached to the state change

logBufferUtilization

public void logBufferUtilization(JobID jobID)
                          throws IOException

Description copied from interface: ExtendedManagementProtocol

Triggers all task managers involved in processing the job with the given job ID to write the utilization of their read and write buffers to their log files. This method is primarily for debugging purposes.

Specified by:

logBufferUtilization in interface ExtendedManagementProtocol

Parameters:

jobID - the ID of the job to print the buffer distribution for

Throws:

IOException - throws if an error occurs while transmitting the request

getAvailableSlots

public int getAvailableSlots()

Description copied from interface: ExtendedManagementProtocol

Returns the number of available slots among the registered task managers

Specified by:

getAvailableSlots in interface ExtendedManagementProtocol

Returns:

number of available slots

deploy

public void deploy(JobID jobID,
          Instance instance,
          List<ExecutionVertex> verticesToBeDeployed)

Description copied from interface: DeploymentManager

Deploys the list of vertices on the given Instance.

Specified by:

deploy in interface DeploymentManager

Parameters:

jobID - the ID of the job the vertices to be deployed belong to

instance - the instance on which the vertices shall be deployed

verticesToBeDeployed - the list of vertices to be deployed

requestNextInputSplit

public InputSplitWrapper requestNextInputSplit(JobID jobID,
                                      ExecutionVertexID vertexID,
                                      IntegerRecord sequenceNumber)
                                        throws IOException

Description copied from interface: InputSplitProviderProtocol

Requests the next split to be consumed by the task with the given execution vertex ID.

Specified by:

requestNextInputSplit in interface InputSplitProviderProtocol

Parameters:

jobID - the ID of the job the task to retrieve the next input split for belongs to

vertexID - the ID of the task to retrieve the next input split for

sequenceNumber - a sequence number, starting at 0 and increased by the task on each request

Returns:

a wrapper containing the next input split. The wrapped input split may also be null in case no more input splits shall be consumed by the task with the given execution vertex ID

Throws:

IOException - thrown if an I/O error occurs while retrieving the new input split

startInfoServer

public void startInfoServer()

Starts the Jetty Infoserver for the Jobmanager

getOldJobs

public List<RecentJobEvent> getOldJobs()
                                throws IOException

Throws:

IOException

getArchive

public ArchiveListener getArchive()

getNumberOfTaskTrackers

public int getNumberOfTaskTrackers()

getInstances

public Map<InstanceConnectionInfo,Instance> getInstances()

reportAccumulatorResult

public void reportAccumulatorResult(AccumulatorEvent accumulatorEvent)
                             throws IOException

Description copied from interface: AccumulatorProtocol

Report accumulators that were collected in a task. Called by Task Manager, after the user code was executed but before the task status update is reported.

Specified by:

reportAccumulatorResult in interface AccumulatorProtocol

Throws:

IOException

getAccumulatorResults

public AccumulatorEvent getAccumulatorResults(JobID jobID)
                                       throws IOException

Description copied from interface: AccumulatorProtocol

Get the final accumulator results. Called by JobClient after the job ended.

Specified by:

getAccumulatorResults in interface AccumulatorProtocol

Throws:

IOException

getAccumulators

public Map<String,Accumulator<?,?>> getAccumulators(JobID jobID)