org.apache.flink.runtime.scheduler

Class SchedulerBase

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, SchedulerNG, org.apache.flink.util.AutoCloseableAsync

Direct Known Subclasses:

DefaultScheduler

public abstract class SchedulerBase
extends Object
implements SchedulerNG, CheckpointScheduling

Base class which can be used to implement SchedulerNG.

Field Summary

Fields

Modifier and Type	Field and Description
protected ExecutionVertexVersioner	executionVertexVersioner
protected InputsLocationsRetriever	inputsLocationsRetriever
protected StateLocationRetriever	stateLocationRetriever

Constructor Summary

Constructors

Constructor and Description

SchedulerBase(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, ExecutionVertexVersioner executionVertexVersioner, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory)

Method Summary

Modifier and Type	Method and Description
void	acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
protected void	archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)
protected void	archiveGlobalFailure(Throwable failure)
void	cancel()
protected abstract void	cancelAllPendingSlotRequestsInternal()
CompletableFuture<Void>	closeAsync()
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation.
static VertexParallelismStore	computeVertexParallelismStore(JobGraph jobGraph) Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.
void	declineCheckpoint(DeclineCheckpoint decline)
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(OperatorID operator, CoordinationRequest request) Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.
void	deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId, OperatorID operatorId, OperatorEvent evt) Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.
protected void	failJob(Throwable cause, long timestamp)
static int	getDefaultMaxParallelism(JobVertex vertex) Get a default value to use for a given vertex's max parallelism if none was specified.
ExecutionGraph	getExecutionGraph() ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler.
ExecutionJobVertex	getExecutionJobVertex(JobVertexID jobVertexId)
ExecutionVertex	getExecutionVertex(ExecutionVertexID executionVertexId)
protected Optional<ExecutionVertexID>	getExecutionVertexId(ExecutionAttemptID executionAttemptId)
protected ExecutionVertexID	getExecutionVertexIdOrThrow(ExecutionAttemptID executionAttemptId)
protected JobGraph	getJobGraph()
CompletableFuture<org.apache.flink.api.common.JobStatus>	getJobTerminationFuture()
protected org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor	getMainThreadExecutor()
protected abstract long	getNumberOfRestarts()
protected ResultPartitionAvailabilityChecker	getResultPartitionAvailabilityChecker()
protected SchedulingTopology	getSchedulingTopology()
void	notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, org.apache.flink.queryablestate.KvStateID kvStateId, InetSocketAddress kvStateServerAddress)
void	notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName)
void	notifyPartitionDataAvailable(ResultPartitionID partitionId)
protected void	notifyPartitionDataAvailableInternal(IntermediateResultPartitionID resultPartitionId)
void	reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID attemptId, long id, CheckpointMetrics metrics)
ExecutionGraphInfo	requestJob()
JobDetails	requestJobDetails()
org.apache.flink.api.common.JobStatus	requestJobStatus()
KvStateLocation	requestKvStateLocation(org.apache.flink.api.common.JobID jobId, String registrationName)
SerializedInputSplit	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt)
ExecutionState	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId)
protected void	resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void	restoreState(Set<ExecutionVertexID> vertices, boolean isGlobalRecovery)
protected void	setGlobalFailureCause(Throwable cause, long timestamp)
void	startCheckpointScheduler() Starts the periodic scheduling if possible.
void	startScheduling()
protected abstract void	startSchedulingInternal()
void	stopCheckpointScheduler() Stops the periodic scheduling if possible.
CompletableFuture<String>	stopWithSavepoint(String targetDirectory, boolean terminate)
protected void	transitionExecutionGraphState(org.apache.flink.api.common.JobStatus current, org.apache.flink.api.common.JobStatus newState)
protected void	transitionToRunning()
protected void	transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob)
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
boolean	updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)
protected void	updateTaskExecutionStateInternal(ExecutionVertexID executionVertexId, TaskExecutionStateTransition taskExecutionState)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

handleGlobalFailure, updateTaskExecutionState

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Field Detail

stateLocationRetriever

protected final StateLocationRetriever stateLocationRetriever

inputsLocationsRetriever

protected final InputsLocationsRetriever inputsLocationsRetriever

executionVertexVersioner

protected final ExecutionVertexVersioner executionVertexVersioner

Constructor Detail

SchedulerBase

public SchedulerBase(org.slf4j.Logger log,
                     JobGraph jobGraph,
                     Executor ioExecutor,
                     org.apache.flink.configuration.Configuration jobMasterConfiguration,
                     ClassLoader userCodeLoader,
                     CheckpointsCleaner checkpointsCleaner,
                     CheckpointRecoveryFactory checkpointRecoveryFactory,
                     JobManagerJobMetricGroup jobManagerJobMetricGroup,
                     ExecutionVertexVersioner executionVertexVersioner,
                     long initializationTimestamp,
                     org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                     JobStatusListener jobStatusListener,
                     ExecutionGraphFactory executionGraphFactory)
              throws Exception

Throws:

Exception

Method Detail

getDefaultMaxParallelism

public static int getDefaultMaxParallelism(JobVertex vertex)

Get a default value to use for a given vertex's max parallelism if none was specified.

Parameters:

vertex - the vertex to compute a default max parallelism for

Returns:

the computed max parallelism

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices,
                                                                   java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation.

Parameters:

vertices - the vertices to compute parallelism for

defaultMaxParallelismFunc - a function for computing a default max parallelism if none is specified on a given vertex

Returns:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.

Parameters:

vertices - the vertices to compute parallelism for

Returns:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(JobGraph jobGraph)

Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.

Parameters:

jobGraph - the job graph to retrieve vertices from

Returns:

the computed parallelism store

resetForNewExecutions

protected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)

restoreState

protected void restoreState(Set<ExecutionVertexID> vertices,
                            boolean isGlobalRecovery)
                     throws Exception

Throws:

Exception

transitionToScheduled

protected void transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)

setGlobalFailureCause

protected void setGlobalFailureCause(@Nullable
                                     Throwable cause,
                                     long timestamp)

getMainThreadExecutor

protected org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor getMainThreadExecutor()

failJob

protected void failJob(Throwable cause,
                       long timestamp)

getSchedulingTopology

protected final SchedulingTopology getSchedulingTopology()

getResultPartitionAvailabilityChecker

protected final ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()

transitionToRunning

protected final void transitionToRunning()

getExecutionVertexId

protected Optional<ExecutionVertexID> getExecutionVertexId(ExecutionAttemptID executionAttemptId)

getExecutionVertexIdOrThrow

protected ExecutionVertexID getExecutionVertexIdOrThrow(ExecutionAttemptID executionAttemptId)

getExecutionVertex

public ExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)

getExecutionJobVertex

public ExecutionJobVertex getExecutionJobVertex(JobVertexID jobVertexId)

getJobGraph

protected JobGraph getJobGraph()

getNumberOfRestarts

protected abstract long getNumberOfRestarts()

cancelAllPendingSlotRequestsInternal

protected abstract void cancelAllPendingSlotRequestsInternal()

transitionExecutionGraphState

protected void transitionExecutionGraphState(org.apache.flink.api.common.JobStatus current,
                                             org.apache.flink.api.common.JobStatus newState)

getExecutionGraph

@VisibleForTesting
public ExecutionGraph getExecutionGraph()

ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler. ExecutionGraph is expected to be used only for state check. Yet at the moment, before all the actions are factored out from ExecutionGraph and its sub-components, some actions may still be performed directly on it.

startScheduling

public final void startScheduling()

Specified by:

startScheduling in interface SchedulerNG

startSchedulingInternal

protected abstract void startSchedulingInternal()

closeAsync

public CompletableFuture<Void> closeAsync()

Specified by:

closeAsync in interface org.apache.flink.util.AutoCloseableAsync

cancel

public void cancel()

Specified by:

cancel in interface SchedulerNG

getJobTerminationFuture

public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobTerminationFuture()

Specified by:

getJobTerminationFuture in interface SchedulerNG

archiveGlobalFailure

protected final void archiveGlobalFailure(Throwable failure)

archiveFromFailureHandlingResult

protected final void archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)

updateTaskExecutionState

public final boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

Specified by:

updateTaskExecutionState in interface SchedulerNG

updateTaskExecutionStateInternal

protected void updateTaskExecutionStateInternal(ExecutionVertexID executionVertexId,
                                                TaskExecutionStateTransition taskExecutionState)

requestNextInputSplit

public SerializedInputSplit requestNextInputSplit(JobVertexID vertexID,
                                                  ExecutionAttemptID executionAttempt)
                                           throws IOException

Specified by:

requestNextInputSplit in interface SchedulerNG

Throws:

IOException

requestPartitionState

public ExecutionState requestPartitionState(IntermediateDataSetID intermediateResultId,
                                            ResultPartitionID resultPartitionId)
                                     throws PartitionProducerDisposedException

Specified by:

requestPartitionState in interface SchedulerNG

Throws:

PartitionProducerDisposedException

notifyPartitionDataAvailable

public final void notifyPartitionDataAvailable(ResultPartitionID partitionId)

Specified by:

notifyPartitionDataAvailable in interface SchedulerNG

notifyPartitionDataAvailableInternal

protected void notifyPartitionDataAvailableInternal(IntermediateResultPartitionID resultPartitionId)

requestJob

public ExecutionGraphInfo requestJob()

Specified by:

requestJob in interface SchedulerNG

requestJobStatus

public org.apache.flink.api.common.JobStatus requestJobStatus()

Specified by:

requestJobStatus in interface SchedulerNG

requestJobDetails

public JobDetails requestJobDetails()

Specified by:

requestJobDetails in interface SchedulerNG

requestKvStateLocation

public KvStateLocation requestKvStateLocation(org.apache.flink.api.common.JobID jobId,
                                              String registrationName)
                                       throws UnknownKvStateLocation,
                                              FlinkJobNotFoundException

Specified by:

requestKvStateLocation in interface SchedulerNG

Throws:

UnknownKvStateLocation

FlinkJobNotFoundException

notifyKvStateRegistered

public void notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId,
                                    JobVertexID jobVertexId,
                                    KeyGroupRange keyGroupRange,
                                    String registrationName,
                                    org.apache.flink.queryablestate.KvStateID kvStateId,
                                    InetSocketAddress kvStateServerAddress)
                             throws FlinkJobNotFoundException

Specified by:

notifyKvStateRegistered in interface SchedulerNG

Throws:

FlinkJobNotFoundException

notifyKvStateUnregistered

public void notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId,
                                      JobVertexID jobVertexId,
                                      KeyGroupRange keyGroupRange,
                                      String registrationName)
                               throws FlinkJobNotFoundException

Specified by:

notifyKvStateUnregistered in interface SchedulerNG

Throws:

FlinkJobNotFoundException

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

Specified by:

updateAccumulators in interface SchedulerNG

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(String targetDirectory,
                                                  boolean cancelJob)

Specified by:

triggerSavepoint in interface SchedulerNG

stopCheckpointScheduler

public void stopCheckpointScheduler()

Description copied from interface: CheckpointScheduling

Stops the periodic scheduling if possible.

Specified by:

stopCheckpointScheduler in interface CheckpointScheduling

startCheckpointScheduler

public void startCheckpointScheduler()

Description copied from interface: CheckpointScheduling

Starts the periodic scheduling if possible.

Specified by:

startCheckpointScheduler in interface CheckpointScheduling

acknowledgeCheckpoint

public void acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  TaskStateSnapshot checkpointState)

Specified by:

acknowledgeCheckpoint in interface SchedulerNG

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

Specified by:

declineCheckpoint in interface SchedulerNG

reportCheckpointMetrics

public void reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID,
                                    ExecutionAttemptID attemptId,
                                    long id,
                                    CheckpointMetrics metrics)

Specified by:

reportCheckpointMetrics in interface SchedulerNG

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(@Nullable
                                                   String targetDirectory,
                                                   boolean terminate)

Specified by:

stopWithSavepoint in interface SchedulerNG

deliverOperatorEventToCoordinator

public void deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId,
                                              OperatorID operatorId,
                                              OperatorEvent evt)
                                       throws org.apache.flink.util.FlinkException

Description copied from interface: SchedulerNG

Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.

Failure semantics: If the task manager sends an event for a non-running task or a non-existing operator coordinator, then respond with an exception to the call. If task and coordinator exist, then we assume that the call from the TaskManager was valid, and any bubbling exception needs to cause a job failure

Specified by:

deliverOperatorEventToCoordinator in interface SchedulerNG

Throws:

org.apache.flink.util.FlinkException - Thrown, if the task is not running or no operator/coordinator exists for the given ID.

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operator,
                                                                                       CoordinationRequest request)
                                                                                throws org.apache.flink.util.FlinkException

Description copied from interface: SchedulerNG

Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.

Specified by:

deliverCoordinationRequestToCoordinator in interface SchedulerNG

Returns:

A future containing the response.

Throws:

org.apache.flink.util.FlinkException - Thrown, if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.