org.apache.tez.dag.app.dag.speculation.legacy

Class SimpleExponentialTaskRuntimeEstimator

java.lang.Object

org.apache.tez.dag.app.dag.speculation.legacy.SimpleExponentialTaskRuntimeEstimator

All Implemented Interfaces:

TaskRuntimeEstimator

public class SimpleExponentialTaskRuntimeEstimator
extends Object

A task Runtime Estimator based on exponential smoothing.

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<Task>	doneTasks
protected Map<TezTaskAttemptID,Long>	startTimes
protected DataStatistics	taskStatistics
protected Vertex	vertex

Constructor Summary

Constructors

Constructor and Description
SimpleExponentialTaskRuntimeEstimator()

Method Summary

Modifier and Type	Method and Description
long	attemptEnrolledTime(TezTaskAttemptID attemptID)
void	contextualize(org.apache.hadoop.conf.Configuration conf, Vertex vertex)
protected DataStatistics	dataStatisticsForTask(TezTaskID taskID)
void	enrollAttempt(TezTaskAttemptID id, long timestamp)
long	estimatedRuntime(TezTaskAttemptID id) Estimate a task attempt's total runtime.
boolean	hasStagnatedProgress(TezTaskAttemptID id, long timeStamp) Returns true if the estimator has no updates records for a threshold time window.
long	newAttemptEstimatedRuntime() Estimates how long a new attempt on this task will take if we start one now
long	runtimeEstimateVariance(TezTaskAttemptID id) Computes the width of the error band of our estimate of the task runtime as returned by TaskRuntimeEstimator.estimatedRuntime(TezTaskAttemptID)
long	thresholdRuntime(TezTaskID taskID) Find a maximum reasonable execution wallclock time.
void	updateAttempt(TezTaskAttemptID attemptID, TaskAttemptState state, long timestamp)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

vertex

protected Vertex vertex

startTimes

protected final Map<TezTaskAttemptID,Long> startTimes

taskStatistics

protected final DataStatistics taskStatistics

doneTasks

protected final Set<Task> doneTasks

Constructor Detail

SimpleExponentialTaskRuntimeEstimator

public SimpleExponentialTaskRuntimeEstimator()

Method Detail

contextualize

public void contextualize(org.apache.hadoop.conf.Configuration conf,
                          Vertex vertex)

Specified by:

contextualize in interface TaskRuntimeEstimator

estimatedRuntime

public long estimatedRuntime(TezTaskAttemptID id)

Description copied from interface: TaskRuntimeEstimator

Estimate a task attempt's total runtime. Includes the time already elapsed.

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

Returns:

our best estimate of the attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

newAttemptEstimatedRuntime

public long newAttemptEstimatedRuntime()

Description copied from interface: TaskRuntimeEstimator

Estimates how long a new attempt on this task will take if we start one now

Specified by:

newAttemptEstimatedRuntime in interface TaskRuntimeEstimator

Returns:

our best estimate of a new attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

hasStagnatedProgress

public boolean hasStagnatedProgress(TezTaskAttemptID id,
                                    long timeStamp)

Description copied from interface: TaskRuntimeEstimator

Returns true if the estimator has no updates records for a threshold time window. This helps to identify task attempts that are stalled at the beginning of execution.

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

timeStamp - the time of the report we compare with

Returns:

true if the task attempt has no progress for a given time window

runtimeEstimateVariance

public long runtimeEstimateVariance(TezTaskAttemptID id)

Description copied from interface: TaskRuntimeEstimator

Computes the width of the error band of our estimate of the task runtime as returned by TaskRuntimeEstimator.estimatedRuntime(TezTaskAttemptID)

Parameters:

id - the TezTaskAttemptID of the attempt we are asking about

Returns:

our best estimate of the attempt's runtime, or -1 if we don't have enough information yet to produce an estimate.

updateAttempt

public void updateAttempt(TezTaskAttemptID attemptID,
                          TaskAttemptState state,
                          long timestamp)

Specified by:

updateAttempt in interface TaskRuntimeEstimator

enrollAttempt

public void enrollAttempt(TezTaskAttemptID id,
                          long timestamp)

Specified by:

enrollAttempt in interface TaskRuntimeEstimator

attemptEnrolledTime

public long attemptEnrolledTime(TezTaskAttemptID attemptID)

Specified by:

attemptEnrolledTime in interface TaskRuntimeEstimator

dataStatisticsForTask

protected DataStatistics dataStatisticsForTask(TezTaskID taskID)

thresholdRuntime

public long thresholdRuntime(TezTaskID taskID)

Description copied from interface: TaskRuntimeEstimator

Find a maximum reasonable execution wallclock time. Includes the time already elapsed. Find a maximum reasonable execution time. Includes the time already elapsed. If the projected total execution time for this task ever exceeds its reasonable execution time, we may speculate it.

Specified by:

thresholdRuntime in interface TaskRuntimeEstimator

Parameters:

taskID - the TezTaskID of the task we are asking about

Returns:

the task's maximum reasonable runtime, or MAX_VALUE if we don't have enough information to rule out any runtime, however long.