org.apache.tez.dag.app.dag.speculation.legacy

Interface TaskRuntimeEstimator

All Known Implementing Classes:

LegacyTaskRuntimeEstimator, SimpleExponentialTaskRuntimeEstimator

public interface TaskRuntimeEstimator

Estimate the runtime for tasks of a given vertex.

Method Summary

Modifier and Type	Method and Description
long	attemptEnrolledTime(TezTaskAttemptID attemptID)
void	contextualize(org.apache.hadoop.conf.Configuration conf, Vertex vertex)
void	enrollAttempt(TezTaskAttemptID id, long timestamp)
long	estimatedRuntime(TezTaskAttemptID id) Estimate a task attempt's total runtime.
default boolean	hasStagnatedProgress(TezTaskAttemptID id, long timeStamp) Returns true if the estimator has no updates records for a threshold time window.
long	newAttemptEstimatedRuntime() Estimates how long a new attempt on this task will take if we start one now
long	runtimeEstimateVariance(TezTaskAttemptID id) Computes the width of the error band of our estimate of the task runtime as returned by estimatedRuntime(TezTaskAttemptID)
long	thresholdRuntime(TezTaskID id) Find a maximum reasonable execution wallclock time.
void	updateAttempt(TezTaskAttemptID taId, TaskAttemptState reportedState, long timestamp)

Method Detail

enrollAttempt

void enrollAttempt(TezTaskAttemptID id,
                   long timestamp)

attemptEnrolledTime

long attemptEnrolledTime(TezTaskAttemptID attemptID)

updateAttempt

void updateAttempt(TezTaskAttemptID taId,
                   TaskAttemptState reportedState,
                   long timestamp)

contextualize

void contextualize(org.apache.hadoop.conf.Configuration conf,
                   Vertex vertex)

thresholdRuntime

long thresholdRuntime(TezTaskID id)

Find a maximum reasonable execution wallclock time. Includes the time already elapsed. Find a maximum reasonable execution time. Includes the time already elapsed. If the projected total execution time for this task ever exceeds its reasonable execution time, we may speculate it.

Parameters:

id - the TezTaskID of the task we are asking about

Returns:

the task's maximum reasonable runtime, or MAX_VALUE if we don't have enough information to rule out any runtime, however long.

estimatedRuntime

long estimatedRuntime(TezTaskAttemptID id)

Estimate a task attempt's total runtime. Includes the time already elapsed.

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

Returns:

our best estimate of the attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

newAttemptEstimatedRuntime

long newAttemptEstimatedRuntime()

Estimates how long a new attempt on this task will take if we start one now

Returns:

our best estimate of a new attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

runtimeEstimateVariance

long runtimeEstimateVariance(TezTaskAttemptID id)

Computes the width of the error band of our estimate of the task runtime as returned by estimatedRuntime(TezTaskAttemptID)

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

Returns:

our best estimate of the attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

hasStagnatedProgress

default boolean hasStagnatedProgress(TezTaskAttemptID id,
                                     long timeStamp)

Returns true if the estimator has no updates records for a threshold time window. This helps to identify task attempts that are stalled at the beginning of execution.

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

timeStamp - the time of the report we compare with

Returns:

true if the task attempt has no progress for a given time window