org.apache.flink.runtime.io.disk.iomanager

Class IOManager

java.lang.Object

org.apache.flink.runtime.io.disk.iomanager.IOManager

All Implemented Interfaces:

Thread.UncaughtExceptionHandler

public final class IOManager
extends Object
implements Thread.UncaughtExceptionHandler

The facade for the provided I/O manager services.

Constructor Summary

Constructors

Constructor and Description
IOManager() Constructs a new IOManager, writing channels to the system directory.
IOManager(String tempDir) Constructs a new IOManager.
IOManager(String[] paths) Constructs a new IOManager.

Method Summary

Methods

Modifier and Type	Method and Description
BlockChannelReader	createBlockChannelReader(Channel.ID channelID) Creates a block channel reader that reads blocks from the given channel.
BlockChannelReader	createBlockChannelReader(Channel.ID channelID, int numRequestsToCombine) Creates a block channel reader that reads blocks from the given channel.
BlockChannelReader	createBlockChannelReader(Channel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue) Creates a block channel reader that reads blocks from the given channel.
BlockChannelReader	createBlockChannelReader(Channel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue, int numRequestsToCombine) Creates a block channel reader that reads blocks from the given channel.
BlockChannelWriter	createBlockChannelWriter(Channel.ID channelID) Creates a block channel writer that writes to the given channel.
BlockChannelWriter	createBlockChannelWriter(Channel.ID channelID, int numRequestsToCombine) Creates a block channel writer that writes to the given channel.
BlockChannelWriter	createBlockChannelWriter(Channel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue) Creates a block channel writer that writes to the given channel.
BlockChannelWriter	createBlockChannelWriter(Channel.ID channelID, LinkedBlockingQueue<MemorySegment> returnQueue, int numRequestsToCombine) Creates a block channel writer that writes to the given channel.
BulkBlockChannelReader	createBulkBlockChannelReader(Channel.ID channelID, List<MemorySegment> targetSegments, int numBlocks) Creates a block channel reader that reads all blocks from the given channel directly in one bulk.
Channel.ID	createChannel() Creates a new Channel.ID in one of the temp directories.
Channel.Enumerator	createChannelEnumerator() Creates a new Channel.Enumerator, spreading the channels in a round-robin fashion across the temporary file directories.
boolean	isProperlyShutDown() Utility method to check whether the IO manager has been properly shut down.
void	shutdown() Close method.
void	uncaughtException(Thread t, Throwable e)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOManager

public IOManager()

Constructs a new IOManager, writing channels to the system directory.

IOManager

public IOManager(String tempDir)

Constructs a new IOManager.

Parameters:

path - The base directory path for files underlying channels.

IOManager

public IOManager(String[] paths)

Constructs a new IOManager.

Parameters:

path - the basic directory path for files underlying anonymous channels.

Method Detail

shutdown

public final void shutdown()

Close method. Shuts down the reader and writer threads immediately, not waiting for their pending requests to be served. This method waits until the threads have actually ceased their operation.

isProperlyShutDown

public final boolean isProperlyShutDown()

Utility method to check whether the IO manager has been properly shut down. The IO manager is considered to be properly shut down when it is closed and its threads have ceased operation.

Returns:

True, if the IO manager has properly shut down, false otherwise.

uncaughtException

public void uncaughtException(Thread t,
                     Throwable e)

Specified by:

uncaughtException in interface Thread.UncaughtExceptionHandler

createChannel

public Channel.ID createChannel()

Creates a new Channel.ID in one of the temp directories. Multiple invocations of this method spread the channels evenly across the different directories.

Returns:

A channel to a temporary directory.

createChannelEnumerator

public Channel.Enumerator createChannelEnumerator()

Creates a new Channel.Enumerator, spreading the channels in a round-robin fashion across the temporary file directories.

Returns:

An enumerator for channels.

createBlockChannelWriter

public BlockChannelWriter createBlockChannelWriter(Channel.ID channelID,
                                          LinkedBlockingQueue<MemorySegment> returnQueue)
                                            throws IOException

Creates a block channel writer that writes to the given channel. The writer writes asynchronously (write-behind), accepting write request, carrying them out at some time and returning the written segment to the given queue afterwards.

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the written buffers into.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelWriter

public BlockChannelWriter createBlockChannelWriter(Channel.ID channelID,
                                          LinkedBlockingQueue<MemorySegment> returnQueue,
                                          int numRequestsToCombine)
                                            throws IOException

Creates a block channel writer that writes to the given channel. The writer writes asynchronously (write-behind), accepting write request, carrying them out at some time and returning the written segment to the given queue afterwards.

The writer will collect a specified number of write requests and carry them out in one, effectively writing one block in the size of multiple memory pages. Note that this means that no memory segment will reach the return queue before the given number of requests are collected, so the number of buffers used with the writer should be greater than the number of requests to combine. Ideally, the number of memory segments used is a multiple of the number of requests to combine.

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the written buffers into.

numRequestsToCombine - The number of write requests to combine to one I/O request.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelWriter

public BlockChannelWriter createBlockChannelWriter(Channel.ID channelID)
                                            throws IOException

Creates a block channel writer that writes to the given channel. The writer writes asynchronously (write-behind), accepting write request, carrying them out at some time and returning the written segment its return queue afterwards.

Parameters:

channelID - The descriptor for the channel to write to.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelWriter

public BlockChannelWriter createBlockChannelWriter(Channel.ID channelID,
                                          int numRequestsToCombine)
                                            throws IOException

Creates a block channel writer that writes to the given channel. The writer writes asynchronously (write-behind), accepting write request, carrying them out at some time and returning the written segment its return queue afterwards.

The writer will collect a specified number of write requests and carry them out in one, effectively writing one block in the size of multiple memory pages. Note that this means that no memory segment will reach the return queue before the given number of requests are collected, so the number of buffers used with the writer should be greater than the number of requests to combine. Ideally, the number of memory segments used is a multiple of the number of requests to combine.

Parameters:

channelID - The descriptor for the channel to write to.

numRequestsToCombine - The number of write requests to combine to one I/O request.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelReader

public BlockChannelReader createBlockChannelReader(Channel.ID channelID,
                                          LinkedBlockingQueue<MemorySegment> returnQueue)
                                            throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the given queue.

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the full buffers into.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBlockChannelReader

public BlockChannelReader createBlockChannelReader(Channel.ID channelID,
                                          LinkedBlockingQueue<MemorySegment> returnQueue,
                                          int numRequestsToCombine)
                                            throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the given queue.

The reader will collect a specified number of read requests and carry them out in one, effectively reading one block in the size of multiple memory pages. Note that this means that no memory segment will reach the return queue before the given number of requests are collected, so the number of buffers used with the reader should be greater than the number of requests to combine. Ideally, the number of memory segments used is a multiple of the number of requests to combine.

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the full buffers into.

numRequestsToCombine - The number of read requests to combine to one I/O request.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBlockChannelReader

public BlockChannelReader createBlockChannelReader(Channel.ID channelID)
                                            throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the reader's return queue.

Parameters:

channelID - The descriptor for the channel to write to.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBlockChannelReader

public BlockChannelReader createBlockChannelReader(Channel.ID channelID,
                                          int numRequestsToCombine)
                                            throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the reader's return queue.

The reader will collect a specified number of read requests and carry them out in one, effectively reading one block in the size of multiple memory pages. Note that this means that no memory segment will reach the return queue before the given number of requests are collected, so the number of buffers used with the reader should be greater than the number of requests to combine. Ideally, the number of memory segments used is a multiple of the number of requests to combine.

Parameters:

channelID - The descriptor for the channel to write to.

numRequestsToCombine - The number of write requests to combine to one I/O request.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBulkBlockChannelReader

public BulkBlockChannelReader createBulkBlockChannelReader(Channel.ID channelID,
                                                  List<MemorySegment> targetSegments,
                                                  int numBlocks)
                                                    throws IOException

Creates a block channel reader that reads all blocks from the given channel directly in one bulk. The reader draws segments to read the blocks into from a supplied list, which must contain as many segments as the channel has blocks. After the reader is done, the list with the full segments can be obtained from the reader.

If a channel is not to be read in one bulk, but in multiple smaller batches, a BlockChannelReader should be used.

Parameters:

channelID - The descriptor for the channel to write to.

targetSegments - The list to take the segments from into which to read the data.

numBlocks - The number of blocks in the channel to read.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.