org.apache.hadoop.mapred.gridmix
Class StressJobFactory

java.lang.Object
  org.apache.hadoop.mapred.gridmix.StressJobFactory

public class StressJobFactory
extends Object

Nested Class Summary

protected static class

JobFactory.FilterJobStory

Field Summary

protected org.apache.hadoop.conf.Configuration	conf
static String	CONF_MAX_JOB_TRACKER_RATIO
static String	CONF_MAX_MAPSLOT_SHARE_PER_JOB
static String	CONF_MAX_REDUCESLOT_SHARE_PER_JOB
static String	CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO
static String	CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO
protected IOException	error
protected JobCreator	jobCreator
protected org.apache.hadoop.tools.rumen.JobStoryProducer	jobProducer
protected ReentrantLock	lock
static org.apache.commons.logging.Log	LOG
protected int	numJobsInTrace
protected float	rateFactor
protected Thread	rThread
protected org.apache.hadoop.fs.Path	scratch
protected AtomicInteger	sequence
protected CountDownLatch	startFlag
protected org.apache.hadoop.mapred.gridmix.JobSubmitter	submitter
protected UserResolver	userResolver

Constructor Summary

StressJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter, org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer, org.apache.hadoop.fs.Path scratch, org.apache.hadoop.conf.Configuration conf, CountDownLatch startFlag, UserResolver resolver)
Creating a new instance does not start the thread.

Method Summary

void	abort() Interrupt the reader thread.
void	add(Void ignored) Add is disabled.
protected void	checkLoadAndGetSlotsToBackfill() We try to use some light-weight mechanism to determine cluster load.
Thread	createReaderThread()
IOException	error() Obtain the error that caused the thread to exit unexpectedly.
protected org.apache.hadoop.tools.rumen.JobStory	getNextJobFiltered()
void	join(long millis) Wait for the reader thread to exhaust the job trace.
void	shutdown() Interrupt the reader thread.
void	start() Start the reader thread, wait for latch if necessary.
void	update(org.apache.hadoop.mapred.gridmix.Statistics.ClusterStats item) STRESS Once you get the notification from StatsCollector.Collect the clustermetrics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO

public static final String CONF_OVERLOAD_MAPTASK_MAPSLOT_RATIO

See Also:

Constant Field Values

CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO

public static final String CONF_OVERLOAD_REDUCETASK_REDUCESLOT_RATIO

See Also:

Constant Field Values

CONF_MAX_MAPSLOT_SHARE_PER_JOB

public static final String CONF_MAX_MAPSLOT_SHARE_PER_JOB

See Also:

Constant Field Values

CONF_MAX_REDUCESLOT_SHARE_PER_JOB

public static final String CONF_MAX_REDUCESLOT_SHARE_PER_JOB

See Also:

Constant Field Values

CONF_MAX_JOB_TRACKER_RATIO

public static final String CONF_MAX_JOB_TRACKER_RATIO

See Also:

Constant Field Values

scratch

protected final org.apache.hadoop.fs.Path scratch

rateFactor

protected final float rateFactor

conf

protected final org.apache.hadoop.conf.Configuration conf

rThread

protected final Thread rThread

sequence

protected final AtomicInteger sequence

submitter

protected final org.apache.hadoop.mapred.gridmix.JobSubmitter submitter

startFlag

protected final CountDownLatch startFlag

userResolver

protected final UserResolver userResolver

jobCreator

protected final JobCreator jobCreator

error

protected volatile IOException error

jobProducer

protected final org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer

lock

protected final ReentrantLock lock

numJobsInTrace

protected int numJobsInTrace

Constructor Detail

StressJobFactory

public StressJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter,
                        org.apache.hadoop.tools.rumen.JobStoryProducer jobProducer,
                        org.apache.hadoop.fs.Path scratch,
                        org.apache.hadoop.conf.Configuration conf,
                        CountDownLatch startFlag,
                        UserResolver resolver)
                 throws IOException

Creating a new instance does not start the thread.

Parameters:

submitter - Component to which deserialized jobs are passed

jobProducer - Stream of job traces with which to construct a ZombieJobProducer

scratch - Directory into which to write output from simulated jobs

conf - Config passed to all jobs to be submitted

startFlag - Latch released from main to start pipeline

Throws:

IOException

Method Detail

createReaderThread

public Thread createReaderThread()

update

public void update(org.apache.hadoop.mapred.gridmix.Statistics.ClusterStats item)

STRESS Once you get the notification from StatsCollector.Collect the clustermetrics. Update current loadStatus with new load status of JT.

Parameters:

item -

checkLoadAndGetSlotsToBackfill

protected void checkLoadAndGetSlotsToBackfill()
                                       throws IOException,
                                              InterruptedException

We try to use some light-weight mechanism to determine cluster load.

Throws:

IOException

InterruptedException

start

public void start()

Start the reader thread, wait for latch if necessary.

getNextJobFiltered

protected org.apache.hadoop.tools.rumen.JobStory getNextJobFiltered()
                                                             throws IOException

Throws:

IOException

error

public IOException error()

Obtain the error that caused the thread to exit unexpectedly.

add

public void add(Void ignored)

Add is disabled.

Throws:

UnsupportedOperationException

join

public void join(long millis)
          throws InterruptedException

Wait for the reader thread to exhaust the job trace.

Throws:

InterruptedException

shutdown

public void shutdown()

Interrupt the reader thread.

abort

public void abort()

Interrupt the reader thread. This requires no special consideration, as the thread has no pending work queue.