org.apache.tez.common

Class TezCommonUtils

java.lang.Object

org.apache.tez.common.TezCommonUtils

@InterfaceAudience.Private
public class TezCommonUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.permission.FsPermission	TEZ_AM_DIR_PERMISSION
static org.apache.hadoop.fs.permission.FsPermission	TEZ_AM_FILE_PERMISSION
static String	TEZ_SYSTEM_SUB_DIR

Constructor Summary

Constructors

Constructor and Description
TezCommonUtils()

Method Summary

Modifier and Type	Method and Description
static void	addAdditionalLocalResources(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalLrs, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> originalLRs, String logContext)
static com.google.protobuf.ByteString	compressByteArrayToByteString(byte[] inBytes)
static com.google.protobuf.ByteString	compressByteArrayToByteString(byte[] inBytes, Deflater deflater)
static ByteBuffer	convertJobTokenToBytes(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
static org.apache.hadoop.fs.FSDataOutputStream	createFileForAM(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filePath) Create a file with TEZ_AM_FILE_PERMISSION permission and returns OutputStream
static org.apache.hadoop.fs.Path	createTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf, String strAppId) This function returns the staging directory for TEZ system.
static byte[]	decompressByteStringToByteArray(com.google.protobuf.ByteString byteString)
static byte[]	decompressByteStringToByteArray(com.google.protobuf.ByteString byteString, Inflater inflater)
static long	getAMClientHeartBeatPollIntervalMillis(org.apache.hadoop.conf.Configuration conf, long heartbeatIntervalMillis, int buckets) Helper function to get the poll interval for client-AM heartbeats.
static long	getAMClientHeartBeatTimeoutMillis(org.apache.hadoop.conf.Configuration conf) Helper function to get the heartbeat interval for client-AM heartbeats See TezConfiguration.TEZ_AM_CLIENT_HEARTBEAT_TIMEOUT_SECS for more details.
static org.apache.hadoop.fs.Path	getAttemptRecoveryPath(org.apache.hadoop.fs.Path recoveryPath, int attemptID) Returns a path to store app attempt specific recovery details
static String	getCredentialsInfo(org.apache.hadoop.security.Credentials credentials, String identifier)
static org.apache.hadoop.fs.Path	getDAGRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath, String dagID) Returns a path to store DAG specific recovery info
static long	getDAGSessionTimeout(org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.Path	getRecoveryPath(org.apache.hadoop.fs.Path tezSysStagingPath, org.apache.hadoop.conf.Configuration conf) Returns a path to store recovery information
static org.apache.hadoop.fs.Path	getSummaryRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath) Returns a path to store summary info for recovery
static String	getSystemPropertiesToLog(org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.Path	getTezAMJarStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store local resources/session jars
static org.apache.hadoop.fs.Path	getTezBaseStagingPath(org.apache.hadoop.conf.Configuration conf) This function returns the staging directory defined in the config with property name TezConfiguration.TEZ_AM_STAGING_DIR.
static org.apache.hadoop.fs.Path	getTezBinPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store binary plan
static org.apache.hadoop.fs.Path	getTezConfStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store binary configuration
static org.apache.hadoop.fs.Path	getTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf, String strAppId) This function returns the staging directory for TEZ system.
static org.apache.hadoop.fs.Path	getTezTextPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath, String strAppId, String dagPBName) Returns a path to store text plan
static String[]	getTrimmedStrings(String str) Splits a comma separated value String, trimming leading and trailing whitespace on each value.
static void	logCredentials(org.slf4j.Logger log, org.apache.hadoop.security.Credentials credentials, String identifier)
static void	mkDirForAM(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Create a directory with predefined directory permission
static Deflater	newBestCompressionDeflater()
static Deflater	newBestSpeedDeflater()
static Inflater	newInflater()
static org.apache.hadoop.security.Credentials	parseCredentialsBytes(byte[] credentialsBytes)
static ByteBuffer	serializeServiceData(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken) A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData.
static Collection<String>	tokenizeString(String str, String delim)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_AM_DIR_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission TEZ_AM_DIR_PERMISSION

TEZ_AM_FILE_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission TEZ_AM_FILE_PERMISSION

TEZ_SYSTEM_SUB_DIR

public static final String TEZ_SYSTEM_SUB_DIR

See Also:

Constant Field Values

Constructor Detail

TezCommonUtils

public TezCommonUtils()

Method Detail

getTezBaseStagingPath

public static org.apache.hadoop.fs.Path getTezBaseStagingPath(org.apache.hadoop.conf.Configuration conf)

This function returns the staging directory defined in the config with property name TezConfiguration.TEZ_AM_STAGING_DIR. If the property is not defined in the conf, Tez uses the value defined as TezConfiguration.TEZ_AM_STAGING_DIR_DEFAULT. In addition, the function makes sure if the staging directory exists. If not, it creates the directory with permission TEZ_AM_DIR_PERMISSION.

Parameters:

conf - TEZ configuration

Returns:

Fully qualified staging directory

createTezSystemStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path createTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf,
                                                                                              String strAppId)

This function returns the staging directory for TEZ system. Tez creates all its temporary files under this sub-directory. The function creates a sub-directory (TEZ_SYSTEM_SUB_DIR/) under the base staging directory, often provided by user.

Parameters:

conf - Tez configuration

strAppId - Application ID as string

Returns:

TEZ system level staging directory used for Tez internals

getTezSystemStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf,
                                                                                           String strAppId)

This function returns the staging directory for TEZ system. Tez creates all its temporary files under this sub-directory. The function normally doesn't creates any sub-directory under the base staging directory.

Parameters:

conf - Tez configuration

strAppId - Application ID as string

Returns:

TEZ system level staging directory used for Tez internals

getTezConfStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezConfStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store binary configuration

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to configuration

getTezAMJarStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezAMJarStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store local resources/session jars

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to store the session jars

getTezBinPlanStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezBinPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store binary plan

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to store the plan in binary

getTezTextPlanStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezTextPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath,
                                                                                             String strAppId,
                                                                                             String dagPBName)

Returns a path to store text plan

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

strAppId - Application ID

dagPBName - DAG PB Name

Returns:

path to store the plan in text

getRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getRecoveryPath(org.apache.hadoop.fs.Path tezSysStagingPath,
                                                                                   org.apache.hadoop.conf.Configuration conf)
                                                                            throws IOException

Returns a path to store recovery information

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

conf - Tez configuration

Returns:

App recovery path

Throws:

IOException

getAttemptRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getAttemptRecoveryPath(org.apache.hadoop.fs.Path recoveryPath,
                                                                                          int attemptID)

Returns a path to store app attempt specific recovery details

Parameters:

recoveryPath - TEZ recovery directory used for Tez internals

attemptID - Application Attempt Id

Returns:

App attempt specific recovery path

getDAGRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getDAGRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath,
                                                                                      String dagID)

Returns a path to store DAG specific recovery info

Parameters:

attemptRecoverPath - :TEZ system level staging directory used for Tez internals

dagID - DagID as string

Returns:

DAG specific recovery path

getSummaryRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getSummaryRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath)

Returns a path to store summary info for recovery

Parameters:

attemptRecoverPath - TEZ system level staging directory used for Tez internals

Returns:

Summary event path used in recovery

mkDirForAM

public static void mkDirForAM(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path dir)
                       throws IOException

Create a directory with predefined directory permission

Parameters:

fs - Filesystem

dir - directory to be created

Throws:

IOException

createFileForAM

public static org.apache.hadoop.fs.FSDataOutputStream createFileForAM(org.apache.hadoop.fs.FileSystem fs,
                                                                      org.apache.hadoop.fs.Path filePath)
                                                               throws IOException

Create a file with TEZ_AM_FILE_PERMISSION permission and returns OutputStream

Parameters:

fs - Filesystem

filePath - file path to create the file

Returns:

FSDataOutputStream

Throws:

IOException

addAdditionalLocalResources

public static void addAdditionalLocalResources(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalLrs,
                                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> originalLRs,
                                               String logContext)

newBestCompressionDeflater

@InterfaceAudience.Private
public static Deflater newBestCompressionDeflater()

newBestSpeedDeflater

@InterfaceAudience.Private
public static Deflater newBestSpeedDeflater()

newInflater

@InterfaceAudience.Private
public static Inflater newInflater()

compressByteArrayToByteString

@InterfaceAudience.Private
public static com.google.protobuf.ByteString compressByteArrayToByteString(byte[] inBytes)
                                                                                               throws IOException

Throws:

IOException

compressByteArrayToByteString

@InterfaceAudience.Private
public static com.google.protobuf.ByteString compressByteArrayToByteString(byte[] inBytes,
                                                                                                      Deflater deflater)
                                                                                               throws IOException

Throws:

IOException

decompressByteStringToByteArray

@InterfaceAudience.Private
public static byte[] decompressByteStringToByteArray(com.google.protobuf.ByteString byteString)
                                                                         throws IOException

Throws:

IOException

decompressByteStringToByteArray

@InterfaceAudience.Private
public static byte[] decompressByteStringToByteArray(com.google.protobuf.ByteString byteString,
                                                                                Inflater inflater)
                                                                         throws IOException

Throws:

IOException

getCredentialsInfo

public static String getCredentialsInfo(org.apache.hadoop.security.Credentials credentials,
                                        String identifier)

convertJobTokenToBytes

public static ByteBuffer convertJobTokenToBytes(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
                                         throws IOException

Throws:

IOException

parseCredentialsBytes

public static org.apache.hadoop.security.Credentials parseCredentialsBytes(byte[] credentialsBytes)
                                                                    throws IOException

Throws:

IOException

logCredentials

public static void logCredentials(org.slf4j.Logger log,
                                  org.apache.hadoop.security.Credentials credentials,
                                  String identifier)

tokenizeString

public static Collection<String> tokenizeString(String str,
                                                String delim)

getTrimmedStrings

public static String[] getTrimmedStrings(String str)

Splits a comma separated value String, trimming leading and trailing whitespace on each value.

Parameters:

str - a comma separated with values

Returns:

an array of String values

serializeServiceData

public static ByteBuffer serializeServiceData(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
                                       throws IOException

A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData. *NOTE* This is a copy of what is done by the MapReduce ShuffleHandler. Not using that directly to avoid a dependency on mapreduce.

Parameters:

jobToken - the job token to be used for authentication of shuffle data requests.

Returns:

the serialized version of the jobToken.

Throws:

IOException

getSystemPropertiesToLog

public static String getSystemPropertiesToLog(org.apache.hadoop.conf.Configuration conf)

getAMClientHeartBeatTimeoutMillis

public static long getAMClientHeartBeatTimeoutMillis(org.apache.hadoop.conf.Configuration conf)

Helper function to get the heartbeat interval for client-AM heartbeats See TezConfiguration.TEZ_AM_CLIENT_HEARTBEAT_TIMEOUT_SECS for more details.

Parameters:

conf - Configuration object

Returns:

heartbeat interval in milliseconds. -1 implies disabled.

getAMClientHeartBeatPollIntervalMillis

public static long getAMClientHeartBeatPollIntervalMillis(org.apache.hadoop.conf.Configuration conf,
                                                          long heartbeatIntervalMillis,
                                                          int buckets)

Helper function to get the poll interval for client-AM heartbeats.

Parameters:

conf - Configuration object

heartbeatIntervalMillis - Heartbeat interval in milliseconds

buckets - How many times to poll within the provided heartbeat interval

Returns:

poll interval in milliseconds

getDAGSessionTimeout

public static long getDAGSessionTimeout(org.apache.hadoop.conf.Configuration conf)