org.apache.flink.runtime.jobmanager

Class JobManager

java.lang.Object

org.apache.flink.runtime.jobmanager.JobManager

All Implemented Interfaces:

VersionedProtocol, JobStatusListener, AccumulatorProtocol, ChannelLookupProtocol, ExtendedManagementProtocol, InputSplitProviderProtocol, JobManagementProtocol, JobManagerProtocol, ServiceDiscoveryProtocol

public class JobManager
extends Object
implements ExtendedManagementProtocol, InputSplitProviderProtocol, JobManagerProtocol, ChannelLookupProtocol, JobStatusListener, AccumulatorProtocol

The JobManager is the master that coordinates the distributed execution. It receives jobs from clients, tracks the distributed execution.

Constructor Summary

Constructors

Constructor and Description
JobManager(ExecutionMode executionMode)

Method Summary

Methods

Modifier and Type	Method and Description
JobCancelResult	cancelJob(JobID jobID) Requests to cancel the job specified by the given ID.
AccumulatorEvent	getAccumulatorResults(JobID jobID) Get the final accumulator results.
Map<String,Accumulator<?,?>>	getAccumulators(JobID jobID)
ArchiveListener	getArchive()
int	getBlobServerPort() Returns the network port of the job manager's BLOB server.
Map<JobID,ExecutionGraph>	getCurrentJobs()
List<AbstractEvent>	getEvents(JobID jobID) Retrieves the collected events for the job with the given job ID.
InstanceManager	getInstanceManager()
Map<InstanceID,Instance>	getInstances()
JobProgressResult	getJobProgress(JobID jobID) Retrieves the current status of the job specified by the given ID.
int	getNumberOfSlotsAvailableToScheduler()
int	getNumberOfTaskManagers()
List<RecentJobEvent>	getOldJobs()
ExecutionGraph	getRecentExecutionGraph(JobID jobID)
List<RecentJobEvent>	getRecentJobs() Retrieves a list of jobs which have either running or have been started recently.
IntegerRecord	getRecommendedPollingInterval() Returns the recommended interval in seconds in which a client is supposed to poll for progress information.
int	getTotalNumberOfRegisteredSlots() Returns the number of available slots among the registered task managers
static JobManager	initialize(String[] args)
boolean	isShutDown() Tests whether the job manager has been shut down completely.
void	jobStatusHasChanged(ExecutionGraph executionGraph, JobStatus newJobStatus, String optionalMessage) Called when the status of the job changed.
ConnectionInfoLookupResponse	lookupConnectionInfo(InstanceConnectionInfo caller, JobID jobID, ChannelID sourceChannelID) Retrieves all recipients of a data for the given sourceChannelID.
static void	main(String[] args) Entry point for the program
InstanceID	registerTaskManager(InstanceConnectionInfo instanceConnectionInfo, HardwareDescription hardwareDescription, int numberOfSlots) Registers a task manager at the JobManager.
void	reportAccumulatorResult(AccumulatorEvent accumulatorEvent) Report accumulators that were collected in a task.
InputSplitWrapper	requestNextInputSplit(JobID jobID, JobVertexID vertexId, ExecutionAttemptID executionAttempt)
boolean	sendHeartbeat(InstanceID taskManagerId) Sends a heart beat to the job manager.
void	shutdown()
void	startInfoServer() Starts the Jetty Infoserver for the Jobmanager
JobSubmissionResult	submitJob(JobGraph job) Submits the specified job to the job manager.
boolean	updateTaskExecutionState(TaskExecutionState executionState) Reports an update of a task's execution state to the job manager.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JobManager

public JobManager(ExecutionMode executionMode)
           throws Exception

Throws:

Exception

Method Detail

shutdown

public void shutdown()

submitJob

public JobSubmissionResult submitJob(JobGraph job)
                              throws IOException

Description copied from interface: JobManagementProtocol

Submits the specified job to the job manager.

Specified by:

submitJob in interface JobManagementProtocol

Parameters:

job - the job to be executed

Returns:

a protocol of the job submission including the success status

Throws:

IOException - thrown if an error occurred while transmitting the submit request

cancelJob

public JobCancelResult cancelJob(JobID jobID)
                          throws IOException

Description copied from interface: JobManagementProtocol

Requests to cancel the job specified by the given ID.

Specified by:

cancelJob in interface JobManagementProtocol

Parameters:

jobID - the ID of the job

Returns:

a JobCancelResult containing the result of the cancel request

Throws:

IOException - thrown if an error occurred while transmitting the request

updateTaskExecutionState

public boolean updateTaskExecutionState(TaskExecutionState executionState)
                                 throws IOException

Description copied from interface: JobManagerProtocol

Reports an update of a task's execution state to the job manager. This method returns true, if the state was correctly registered. It it returns false, the calling task manager should cancel its execution of the task.

Specified by:

updateTaskExecutionState in interface JobManagerProtocol

Parameters:

executionState - The new task execution state.

Returns:

True if everything is all right, false if the caller should cancel the task execution.

Throws:

IOException - Thrown, if an error occurs during this remote procedure call

requestNextInputSplit

public InputSplitWrapper requestNextInputSplit(JobID jobID,
                                      JobVertexID vertexId,
                                      ExecutionAttemptID executionAttempt)
                                        throws IOException

Specified by:

requestNextInputSplit in interface InputSplitProviderProtocol

Throws:

IOException

jobStatusHasChanged

public void jobStatusHasChanged(ExecutionGraph executionGraph,
                       JobStatus newJobStatus,
                       String optionalMessage)

Description copied from interface: JobStatusListener

Called when the status of the job changed.

Specified by:

jobStatusHasChanged in interface JobStatusListener

Parameters:

executionGraph - The executionGraph representing the job.

newJobStatus - The new job status.

optionalMessage - An optional message (possibly null) that can be attached to the state change.

getJobProgress

public JobProgressResult getJobProgress(JobID jobID)
                                 throws IOException

Description copied from interface: JobManagementProtocol

Retrieves the current status of the job specified by the given ID. Consecutive calls of this method may result in duplicate events. The caller must take care of this.

Specified by:

getJobProgress in interface JobManagementProtocol

Parameters:

jobID - the ID of the job

Returns:

a JobProgressResult object including the current job progress

Throws:

IOException - thrown if an error occurred while transmitting the request

lookupConnectionInfo

public ConnectionInfoLookupResponse lookupConnectionInfo(InstanceConnectionInfo caller,
                                                JobID jobID,
                                                ChannelID sourceChannelID)

Description copied from interface: ChannelLookupProtocol

Retrieves all recipients of a data for the given sourceChannelID.

Specified by:

lookupConnectionInfo in interface ChannelLookupProtocol

Parameters:

caller - the InstanceConnectionInfo object of the task manager which calls this method

jobID - the ID of the job the channel ID belongs to

sourceChannelID - the ID of the channel to resolve

Returns:

the lookup response containing the connection info and a return code

isShutDown

public boolean isShutDown()

Tests whether the job manager has been shut down completely.

Returns:

true if the job manager has been shut down completely, false otherwise

getInstanceManager

public InstanceManager getInstanceManager()

getRecommendedPollingInterval

public IntegerRecord getRecommendedPollingInterval()
                                            throws IOException

Description copied from interface: JobManagementProtocol

Returns the recommended interval in seconds in which a client is supposed to poll for progress information.

Specified by:

getRecommendedPollingInterval in interface JobManagementProtocol

Returns:

the interval in seconds

Throws:

IOException - thrown if an error occurred while transmitting the request

getRecentJobs

public List<RecentJobEvent> getRecentJobs()
                                   throws IOException

Description copied from interface: ExtendedManagementProtocol

Retrieves a list of jobs which have either running or have been started recently.

Specified by:

getRecentJobs in interface ExtendedManagementProtocol

Returns:

a (possibly) empty list of recent jobs

Throws:

IOException - thrown if an error occurs while retrieving the job list

getEvents

public List<AbstractEvent> getEvents(JobID jobID)
                              throws IOException

Description copied from interface: ExtendedManagementProtocol

Retrieves the collected events for the job with the given job ID.

Specified by:

getEvents in interface ExtendedManagementProtocol

Parameters:

jobID - the ID of the job to retrieve the events for

Returns:

a (possibly empty) list of events which occurred for that event and which are not older than the query interval

Throws:

IOException - thrown if an error occurs while retrieving the list of events

getTotalNumberOfRegisteredSlots

public int getTotalNumberOfRegisteredSlots()

Description copied from interface: ExtendedManagementProtocol

Returns the number of available slots among the registered task managers

Specified by:

getTotalNumberOfRegisteredSlots in interface ExtendedManagementProtocol

Returns:

number of available slots

getNumberOfSlotsAvailableToScheduler

public int getNumberOfSlotsAvailableToScheduler()

Specified by:

getNumberOfSlotsAvailableToScheduler in interface ExtendedManagementProtocol

startInfoServer

public void startInfoServer()

Starts the Jetty Infoserver for the Jobmanager

getOldJobs

public List<RecentJobEvent> getOldJobs()
                                throws IOException

Throws:

IOException

getArchive

public ArchiveListener getArchive()

getNumberOfTaskManagers

public int getNumberOfTaskManagers()

getInstances

public Map<InstanceID,Instance> getInstances()

reportAccumulatorResult

public void reportAccumulatorResult(AccumulatorEvent accumulatorEvent)
                             throws IOException

Description copied from interface: AccumulatorProtocol

Report accumulators that were collected in a task. Called by Task Manager, after the user code was executed but before the task status update is reported.

Specified by:

reportAccumulatorResult in interface AccumulatorProtocol

Throws:

IOException

getAccumulatorResults

public AccumulatorEvent getAccumulatorResults(JobID jobID)
                                       throws IOException

Description copied from interface: AccumulatorProtocol

Get the final accumulator results. Called by JobClient after the job ended.

Specified by:

getAccumulatorResults in interface AccumulatorProtocol

Throws:

IOException

getAccumulators

public Map<String,Accumulator<?,?>> getAccumulators(JobID jobID)

getCurrentJobs

public Map<JobID,ExecutionGraph> getCurrentJobs()

getRecentExecutionGraph

public ExecutionGraph getRecentExecutionGraph(JobID jobID)
                                       throws IOException

Throws:

IOException

sendHeartbeat

public boolean sendHeartbeat(InstanceID taskManagerId)

Description copied from interface: JobManagerProtocol

Sends a heart beat to the job manager.

Specified by:

sendHeartbeat in interface JobManagerProtocol

Parameters:

taskManagerId - The ID identifying the task manager.

registerTaskManager

public InstanceID registerTaskManager(InstanceConnectionInfo instanceConnectionInfo,
                             HardwareDescription hardwareDescription,
                             int numberOfSlots)

Description copied from interface: JobManagerProtocol

Registers a task manager at the JobManager.

Specified by:

registerTaskManager in interface JobManagerProtocol

Parameters:

instanceConnectionInfo - the information the job manager requires to connect to the instance's task manager

hardwareDescription - a hardware description with details on the instance's compute resources.

numberOfSlots - The number of task slots that the TaskManager provides.

Returns:

The ID under which the TaskManager is registered. Null, if the JobManager does not register the TaskManager.

main

public static void main(String[] args)

Entry point for the program

Parameters:

args - arguments from the command line

initialize

public static JobManager initialize(String[] args)
                             throws Exception

Throws:

Exception

getBlobServerPort

public int getBlobServerPort()

Description copied from interface: ServiceDiscoveryProtocol

Returns the network port of the job manager's BLOB server.

Specified by:

getBlobServerPort in interface ServiceDiscoveryProtocol

Returns:

the port of the job manager's BLOB server or -1 if the job manager does not run a BLOB server