org.apache.flink.runtime.io.network.partition.hybrid

类 HsMemoryDataManager

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.HsMemoryDataManager

所有已实现的接口:

HsMemoryDataManagerOperation, HsSpillingInfoProvider

public class HsMemoryDataManager
extends Object
implements HsSpillingInfoProvider, HsMemoryDataManagerOperation

This class is responsible for managing data in memory.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.runtime.io.network.partition.hybrid.HsSpillingInfoProvider

HsSpillingInfoProvider.ConsumeStatus, HsSpillingInfoProvider.ConsumeStatusWithId, HsSpillingInfoProvider.SpillStatus

构造器概要

构造器

构造器和说明
HsMemoryDataManager(int numSubpartitions, int bufferSize, BufferPool bufferPool, HsSpillingStrategy spillStrategy, HsFileDataIndex fileDataIndex, Path dataFilePath, BufferCompressor bufferCompressor, long poolSizeCheckInterval)

方法概要

限定符和类型	方法和说明
void	append(ByteBuffer record, int targetChannel, Buffer.DataType dataType) Append record to HsMemoryDataManager, It will be managed by HsSubpartitionMemoryDataManager witch it belongs to.
void	close() Close this HsMemoryDataManager, it means no data can append to memory and all buffer taken by this class will recycle.
Deque<BufferIndexAndChannel>	getBuffersInOrder(int subpartitionId, HsSpillingInfoProvider.SpillStatus spillStatus, HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId) Get all buffers with the expected status from the subpartition.
List<Integer>	getNextBufferIndexToConsume(HsConsumerId consumerId) Get all subpartition's next buffer index to consume of specific consumer.
int	getNumSubpartitions() Get the number of downstream consumers.
int	getNumTotalRequestedBuffers() Get total number of buffers requested from buffer pool.
int	getNumTotalUnSpillBuffers() Get total number of not decided to spill buffers.
int	getPoolSize() Get the current size of buffer pool.
void	markBufferReleasedFromFile(int subpartitionId, int bufferIndex) This method is called when buffer should mark as released in HsFileDataIndex.
void	onBufferConsumed(BufferIndexAndChannel consumedBuffer) This method is called when buffer is consumed.
void	onBufferFinished() This method is called when buffer is finished.
void	onConsumerReleased(int subpartitionId, HsConsumerId consumerId) This method is called when consumer is decided to released.
void	onDataAvailable(int subpartitionId, Collection<HsConsumerId> consumerIds) This method is called when subpartition data become available.
HsDataView	registerNewConsumer(int subpartitionId, HsConsumerId consumerId, HsSubpartitionConsumerInternalOperations viewOperations) Register HsSubpartitionConsumerInternalOperations to subpartitionViewOperationsMap.
BufferBuilder	requestBufferFromPool() Request buffer from buffer pool.
void	setOutputMetrics(HsOutputMetrics metrics)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

HsMemoryDataManager

public HsMemoryDataManager(int numSubpartitions,
                           int bufferSize,
                           BufferPool bufferPool,
                           HsSpillingStrategy spillStrategy,
                           HsFileDataIndex fileDataIndex,
                           Path dataFilePath,
                           BufferCompressor bufferCompressor,
                           long poolSizeCheckInterval)
                    throws IOException

抛出:

IOException

方法详细资料

append

public void append(ByteBuffer record,
                   int targetChannel,
                   Buffer.DataType dataType)
            throws IOException

Append record to HsMemoryDataManager, It will be managed by HsSubpartitionMemoryDataManager witch it belongs to.

参数:

record - to be managed by this class.

targetChannel - target subpartition of this record.

dataType - the type of this record. In other words, is it data or event.

抛出:

IOException

registerNewConsumer

public HsDataView registerNewConsumer(int subpartitionId,
                                      HsConsumerId consumerId,
                                      HsSubpartitionConsumerInternalOperations viewOperations)

Register HsSubpartitionConsumerInternalOperations to subpartitionViewOperationsMap. It is used to obtain the consumption progress of the subpartition.

close

public void close()

Close this HsMemoryDataManager, it means no data can append to memory and all buffer taken by this class will recycle.

setOutputMetrics

public void setOutputMetrics(HsOutputMetrics metrics)

getPoolSize

public int getPoolSize()

从接口复制的说明: HsSpillingInfoProvider

Get the current size of buffer pool.

指定者:

getPoolSize 在接口中 HsSpillingInfoProvider

getNumSubpartitions

public int getNumSubpartitions()

从接口复制的说明: HsSpillingInfoProvider

Get the number of downstream consumers.

指定者:

getNumSubpartitions 在接口中 HsSpillingInfoProvider

返回:

Number of subpartitions.

getNumTotalRequestedBuffers

public int getNumTotalRequestedBuffers()

从接口复制的说明: HsSpillingInfoProvider

Get total number of buffers requested from buffer pool.

指定者:

getNumTotalRequestedBuffers 在接口中 HsSpillingInfoProvider

getNumTotalUnSpillBuffers

public int getNumTotalUnSpillBuffers()

从接口复制的说明: HsSpillingInfoProvider

Get total number of not decided to spill buffers.

指定者:

getNumTotalUnSpillBuffers 在接口中 HsSpillingInfoProvider

getBuffersInOrder

public Deque<BufferIndexAndChannel> getBuffersInOrder(int subpartitionId,
                                                      HsSpillingInfoProvider.SpillStatus spillStatus,
                                                      HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId)

从接口复制的说明: HsSpillingInfoProvider

Get all buffers with the expected status from the subpartition.

指定者:

getBuffersInOrder 在接口中 HsSpillingInfoProvider

参数:

subpartitionId - target buffers belong to.

spillStatus - expected buffer spill status.

consumeStatusWithId - expected buffer consume status and consumer id.

返回:

all buffers satisfy specific status of this subpartition, This queue must be sorted according to bufferIndex from small to large, in other words, head is the buffer with the minimum bufferIndex in the current subpartition.

getNextBufferIndexToConsume

public List<Integer> getNextBufferIndexToConsume(HsConsumerId consumerId)

从接口复制的说明: HsSpillingInfoProvider

Get all subpartition's next buffer index to consume of specific consumer.

指定者:

getNextBufferIndexToConsume 在接口中 HsSpillingInfoProvider

参数:

consumerId - of the target downstream consumer.

返回:

A list containing all subpartition's next buffer index to consume of specific consumer, if the downstream subpartition view has not been registered, the corresponding return value is -1.

markBufferReleasedFromFile

public void markBufferReleasedFromFile(int subpartitionId,
                                       int bufferIndex)

从接口复制的说明: HsMemoryDataManagerOperation

This method is called when buffer should mark as released in HsFileDataIndex.

指定者:

markBufferReleasedFromFile 在接口中 HsMemoryDataManagerOperation

参数:

subpartitionId - the subpartition that target buffer belong to.

bufferIndex - index of buffer to mark as released.

requestBufferFromPool

public BufferBuilder requestBufferFromPool()
                                    throws InterruptedException

从接口复制的说明: HsMemoryDataManagerOperation

Request buffer from buffer pool.

指定者:

requestBufferFromPool 在接口中 HsMemoryDataManagerOperation

返回:

requested buffer.

抛出:

InterruptedException

onBufferConsumed

public void onBufferConsumed(BufferIndexAndChannel consumedBuffer)

从接口复制的说明: HsMemoryDataManagerOperation

This method is called when buffer is consumed.

指定者:

onBufferConsumed 在接口中 HsMemoryDataManagerOperation

参数:

consumedBuffer - target buffer to mark as consumed.

onBufferFinished

public void onBufferFinished()

从接口复制的说明: HsMemoryDataManagerOperation

This method is called when buffer is finished.

指定者:

onBufferFinished 在接口中 HsMemoryDataManagerOperation

onDataAvailable

public void onDataAvailable(int subpartitionId,
                            Collection<HsConsumerId> consumerIds)

从接口复制的说明: HsMemoryDataManagerOperation

This method is called when subpartition data become available.

指定者:

onDataAvailable 在接口中 HsMemoryDataManagerOperation

参数:

subpartitionId - the subpartition's identifier that this consumer belongs to.

consumerIds - the consumer's identifier which need notify data available.

onConsumerReleased

public void onConsumerReleased(int subpartitionId,
                               HsConsumerId consumerId)

从接口复制的说明: HsMemoryDataManagerOperation

This method is called when consumer is decided to released.

指定者:

onConsumerReleased 在接口中 HsMemoryDataManagerOperation

参数:

subpartitionId - the subpartition's identifier that this consumer belongs to.

consumerId - the consumer's identifier which decided to be released.