org.apache.flink.runtime.io.network.partition

Class BufferWritingResultPartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultPartition

org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition

All Implemented Interfaces:

AutoCloseable, AvailabilityProvider, ResultPartitionWriter

Direct Known Subclasses:

BoundedBlockingResultPartition, PipelinedResultPartition

public abstract class BufferWritingResultPartition
extends ResultPartition

A ResultPartition which writes buffers directly to ResultSubpartitions. This is in contrast to implementations where records are written to a joint structure, from which the subpartitions draw the data after the write phase is finished, for example the sort-based partitioning.

To avoid confusion: On the read side, all subpartitions return buffers (and backlog) to be transported through the network.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields

Modifier and Type	Field and Description
protected ResultSubpartition[]	subpartitions The subpartitions of this partition.

Fields inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition

bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numSubpartitions, partitionId, partitionManager, partitionType

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
BufferWritingResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, BufferCompressor bufferCompressor, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

Method Summary

Modifier and Type	Method and Description
void	broadcastEvent(AbstractEvent event, boolean isPriorityEvent) Writes the given AbstractEvent to all channels.
void	broadcastRecord(ByteBuffer record) Writes the given serialized record to all subpartitions.
void	close() Closes the partition writer which releases the allocated resource, for example the buffer pool.
ResultSubpartitionView	createSubpartitionView(int subpartitionIndex, BufferAvailabilityListener availabilityListener) Returns a reader for the subpartition with the given index.
void	emitRecord(ByteBuffer record, int targetSubpartition) Writes the given serialized record to the target subpartition.
void	finish() Finishes the result partition.
protected void	flushAllSubpartitions(boolean finishProducers)
protected void	flushSubpartition(int targetSubpartition, boolean finishProducers)
ResultSubpartition[]	getAllPartitions()
TimerGauge	getBackPressuredTimeMsPerSecond()
int	getNumberOfQueuedBuffers() Returns the total number of queued buffers of all subpartitions.
int	getNumberOfQueuedBuffers(int targetSubpartition) Returns the number of queued buffers of the given target subpartition.
protected void	releaseInternal() Releases all produced data including both those stored in memory and persisted on disk.
void	setMetricGroup(TaskIOMetricGroup metrics) Sets the metric group for the ResultPartitionWriter.
void	setup() Registers a buffer pool with this result partition.

Methods inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition

canBeCompressed, checkInProduceState, fail, getAllDataProcessedFuture, getAvailableFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, isFinished, isReleased, notifyEndOfData, onSubpartitionAllDataProcessed, release, release, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter

flush, flushAll

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Field Detail

subpartitions

protected final ResultSubpartition[] subpartitions

The subpartitions of this partition. At least one.

Constructor Detail

BufferWritingResultPartition

public BufferWritingResultPartition(String owningTaskName,
                                    int partitionIndex,
                                    ResultPartitionID partitionId,
                                    ResultPartitionType partitionType,
                                    ResultSubpartition[] subpartitions,
                                    int numTargetKeyGroups,
                                    ResultPartitionManager partitionManager,
                                    @Nullable
                                    BufferCompressor bufferCompressor,
                                    org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

Method Detail

setup

public void setup()
           throws IOException

Description copied from class: ResultPartition

Registers a buffer pool with this result partition.

There is one pool for each result partition, which is shared by all its sub partitions.

The pool is registered with the partition *after* it as been constructed in order to conform to the life-cycle of task registrations in the TaskExecutor.

Specified by:

setup in interface ResultPartitionWriter

Overrides:

setup in class ResultPartition

Throws:

IOException

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

Description copied from class: ResultPartition

Returns the total number of queued buffers of all subpartitions.

Specified by:

getNumberOfQueuedBuffers in class ResultPartition

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers(int targetSubpartition)

Description copied from class: ResultPartition

Returns the number of queued buffers of the given target subpartition.

Specified by:

getNumberOfQueuedBuffers in class ResultPartition

flushSubpartition

protected void flushSubpartition(int targetSubpartition,
                                 boolean finishProducers)

flushAllSubpartitions

protected void flushAllSubpartitions(boolean finishProducers)

emitRecord

public void emitRecord(ByteBuffer record,
                       int targetSubpartition)
                throws IOException

Description copied from interface: ResultPartitionWriter

Writes the given serialized record to the target subpartition.

Throws:

IOException

broadcastRecord

public void broadcastRecord(ByteBuffer record)
                     throws IOException

Description copied from interface: ResultPartitionWriter

Writes the given serialized record to all subpartitions. One can also achieve the same effect by emitting the same record to all subpartitions one by one, however, this method can have better performance for the underlying implementation can do some optimizations, for example coping the given serialized record only once to a shared channel which can be consumed by all subpartitions.

Throws:

IOException

broadcastEvent

public void broadcastEvent(AbstractEvent event,
                           boolean isPriorityEvent)
                    throws IOException

Description copied from interface: ResultPartitionWriter

Writes the given AbstractEvent to all channels.

Throws:

IOException

setMetricGroup

public void setMetricGroup(TaskIOMetricGroup metrics)

Description copied from interface: ResultPartitionWriter

Sets the metric group for the ResultPartitionWriter.

Specified by:

setMetricGroup in interface ResultPartitionWriter

Overrides:

setMetricGroup in class ResultPartition

createSubpartitionView

public ResultSubpartitionView createSubpartitionView(int subpartitionIndex,
                                                     BufferAvailabilityListener availabilityListener)
                                              throws IOException

Description copied from interface: ResultPartitionWriter

Returns a reader for the subpartition with the given index.

Throws:

IOException

finish

public void finish()
            throws IOException

Description copied from class: ResultPartition

Finishes the result partition.

After this operation, it is not possible to add further data to the result partition.

For BLOCKING results, this will trigger the deployment of consuming tasks.

Specified by:

finish in interface ResultPartitionWriter

Overrides:

finish in class ResultPartition

Throws:

IOException

releaseInternal

protected void releaseInternal()

Description copied from class: ResultPartition

Releases all produced data including both those stored in memory and persisted on disk.

Specified by:

releaseInternal in class ResultPartition

close

public void close()

Description copied from interface: ResultPartitionWriter

Closes the partition writer which releases the allocated resource, for example the buffer pool.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ResultPartitionWriter

Overrides:

close in class ResultPartition

getBackPressuredTimeMsPerSecond

@VisibleForTesting
public TimerGauge getBackPressuredTimeMsPerSecond()

getAllPartitions

@VisibleForTesting
public ResultSubpartition[] getAllPartitions()