org.apache.flink.runtime.execution.librarycache

Class LibraryCacheManager

java.lang.Object

org.apache.flink.runtime.execution.librarycache.LibraryCacheManager

public final class LibraryCacheManager
extends Object

For each job graph that is submitted to the system the library cache manager maintains a set of libraries (typically JAR files) which the job requires to run. The library cache manager caches library files in order to avoid unnecessary retransmission of data. It is based on a singleton programming pattern, so there exists at most on library manager at a time.

This class is thread-safe.

Method Summary

Methods

Modifier and Type	Method and Description
static void	addLibrary(JobID jobID, Path name, long size, DataInput in) Reads a library from the given input stream and adds it to the local library cache.
static Path	contains(String cacheName) Checks if the given library is in the local cache.
static ClassLoader	getClassLoader(JobID id) Returns the class loader to the specified vertex.
static String[]	getRequiredJarFiles(JobID id) Returns the names of the required libraries of the specified job.
static void	readLibraryFromStream(DataInput in) Reads library data from the given stream.
static void	register(JobID id, Path[] clientPaths) Registers a job ID with a set of library paths that are required to run the job.
static void	register(JobID id, String[] requiredJarFiles) Registers a job ID with a set of library paths that are required to run the job.
static void	unregister(JobID id) Unregisters a job ID and releases the resources associated with it.
static void	writeLibraryToStream(String libraryFileName, DataOutput out) Writes data from the library with the given file name to the specified stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

register

public static void register(JobID id,
            Path[] clientPaths)
                     throws IOException

Registers a job ID with a set of library paths that are required to run the job. The library paths are given in terms of client paths, so the method first translates the client paths into the corresponding internal cache names. For every registered job the library cache manager creates a class loader that is used to instantiate the job's environment later on.

Parameters:

id - the ID of the job to be registered

clientPaths - the client path's of the required libraries

Throws:

IOException - thrown if the library cache manager could not be instantiated, no mapping between the job ID and a job ID exists or the requested library is not in the cache.

register

public static void register(JobID id,
            String[] requiredJarFiles)
                     throws IOException

Registers a job ID with a set of library paths that are required to run the job. For every registered job the library cache manager creates a class loader that is used to instantiate the job's environment later on.

Parameters:

id - the ID of the job to be registered.

clientPaths - the client path's of the required libraries

Throws:

IOException - thrown if the library cache manager could not be instantiated or one of the requested libraries is not in the cache

unregister

public static void unregister(JobID id)
                       throws IOException

Unregisters a job ID and releases the resources associated with it.

Parameters:

id - the job ID to unregister

Throws:

IOException - thrown if the library cache manager could not be instantiated

contains

public static Path contains(String cacheName)
                     throws IOException

Checks if the given library is in the local cache.

Parameters:

cacheName - The name of the library to be checked for.

Returns:

the path object of the library if it is cached, null otherwise

Throws:

IOException - thrown if the library cache manager could not be instantiated or no access to the file system could be obtained

getClassLoader

public static ClassLoader getClassLoader(JobID id)
                                  throws IOException

Returns the class loader to the specified vertex.

Parameters:

id - the ID of the job to return the class loader for

Returns:

the class loader of requested vertex or null if no class loader has been registered with the given ID.

Throws:

IOException - thrown if the library cache manager could not be instantiated

getRequiredJarFiles

public static String[] getRequiredJarFiles(JobID id)
                                    throws IOException

Returns the names of the required libraries of the specified job.

Parameters:

id - the ID of the job to return the names of required libraries for.

Returns:

the names of the required libraries or null if the specified job ID is unknown

Throws:

IOException - thrown if the library cache manager could not be instantiated

writeLibraryToStream

public static void writeLibraryToStream(String libraryFileName,
                        DataOutput out)
                                 throws IOException

Writes data from the library with the given file name to the specified stream.

Parameters:

libraryFileName - the name of the library

out - the stream to write the data to

Throws:

IOException - thrown if an error occurs while writing the data

readLibraryFromStream

public static void readLibraryFromStream(DataInput in)
                                  throws IOException

Reads library data from the given stream.

Parameters:

in - the stream to read the library data from

Throws:

IOException - throws if an error occurs while reading from the stream

addLibrary

public static void addLibrary(JobID jobID,
              Path name,
              long size,
              DataInput in)
                       throws IOException

Reads a library from the given input stream and adds it to the local library cache. The cache name of the library is determined by the checksum of the received data and cannot be specified manually.

Parameters:

jobID - the ID of the job the library data belongs to

name - the name of the library at the clients host

size - the size of the library to be read from the input stream

in - the data input stream

Throws:

IOException - thrown if the library cache manager could not be instantiated or an error occurred while reading the library data from the input stream