org.apache.flink.runtime.io.network.buffer

Class NetworkBufferPool

java.lang.Object

org.apache.flink.runtime.io.network.buffer.NetworkBufferPool

All Implemented Interfaces:

org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider, BufferPoolFactory

public class NetworkBufferPool
extends Object
implements BufferPoolFactory, org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider

The NetworkBufferPool is a fixed size pool of MemorySegment instances for the network stack.

The NetworkBufferPool creates LocalBufferPools from which the individual tasks draw the buffers for the network data transfer. When new local buffer pools are created, the NetworkBufferPool dynamically redistributes the buffers between the pools.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
NetworkBufferPool(int numberOfSegmentsToAllocate, int segmentSize)
NetworkBufferPool(int numberOfSegmentsToAllocate, int segmentSize, java.time.Duration requestSegmentsTimeout) Allocates all MemorySegment instances managed by this pool.

Method Summary

Modifier and Type	Method and Description
int	countBuffers()
BufferPool	createBufferPool(int numRequiredBuffers, int maxUsedBuffers) Tries to create a buffer pool, which is guaranteed to provide at least the number of required buffers.
BufferPool	createBufferPool(int numRequiredBuffers, int maxUsedBuffers, int numSubpartitions, int maxBuffersPerChannel) Tries to create a buffer pool with an owner, which is guaranteed to provide at least the number of required buffers.
void	destroy()
void	destroyAllBufferPools() Destroys all buffer pools that allocate their buffers from this buffer pool (created via createBufferPool(int, int)).
void	destroyBufferPool(BufferPool bufferPool) Destroy callback for updating factory book keeping.
CompletableFuture<?>	getAvailableFuture() Returns a future that is completed when there are free segments in this pool.
long	getAvailableMemory()
int	getNumberOfAvailableMemorySegments()
int	getNumberOfRegisteredBufferPools()
int	getNumberOfUsedMemorySegments()
long	getTotalMemory()
int	getTotalNumberOfMemorySegments()
long	getUsedMemory()
boolean	isDestroyed()
void	recycle(org.apache.flink.core.memory.MemorySegment segment)
void	recycleMemorySegments(Collection<org.apache.flink.core.memory.MemorySegment> segments)
org.apache.flink.core.memory.MemorySegment	requestMemorySegment()
List<org.apache.flink.core.memory.MemorySegment>	requestMemorySegments(int numberOfSegmentsToRequest)
List<org.apache.flink.core.memory.MemorySegment>	requestMemorySegmentsBlocking(int numberOfSegmentsToRequest)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Constructor Detail

NetworkBufferPool

@VisibleForTesting
public NetworkBufferPool(int numberOfSegmentsToAllocate,
                                            int segmentSize)

NetworkBufferPool

public NetworkBufferPool(int numberOfSegmentsToAllocate,
                         int segmentSize,
                         java.time.Duration requestSegmentsTimeout)

Allocates all MemorySegment instances managed by this pool.

Method Detail

requestMemorySegment

@Nullable
public org.apache.flink.core.memory.MemorySegment requestMemorySegment()

requestMemorySegmentsBlocking

public List<org.apache.flink.core.memory.MemorySegment> requestMemorySegmentsBlocking(int numberOfSegmentsToRequest)
                                                                               throws IOException

Throws:

IOException

recycle

public void recycle(org.apache.flink.core.memory.MemorySegment segment)

requestMemorySegments

public List<org.apache.flink.core.memory.MemorySegment> requestMemorySegments(int numberOfSegmentsToRequest)
                                                                       throws IOException

Specified by:

requestMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider

Throws:

IOException

recycleMemorySegments

public void recycleMemorySegments(Collection<org.apache.flink.core.memory.MemorySegment> segments)

Specified by:

recycleMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider

destroy

public void destroy()

isDestroyed

public boolean isDestroyed()

getTotalNumberOfMemorySegments

public int getTotalNumberOfMemorySegments()

getTotalMemory

public long getTotalMemory()

getNumberOfAvailableMemorySegments

public int getNumberOfAvailableMemorySegments()

getAvailableMemory

public long getAvailableMemory()

getNumberOfUsedMemorySegments

public int getNumberOfUsedMemorySegments()

getUsedMemory

public long getUsedMemory()

getNumberOfRegisteredBufferPools

public int getNumberOfRegisteredBufferPools()

countBuffers

public int countBuffers()

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Returns a future that is completed when there are free segments in this pool.

Specified by:

getAvailableFuture in interface AvailabilityProvider

Returns:

a future that is completed if the respective provider is available.

createBufferPool

public BufferPool createBufferPool(int numRequiredBuffers,
                                   int maxUsedBuffers)
                            throws IOException

Description copied from interface: BufferPoolFactory

Tries to create a buffer pool, which is guaranteed to provide at least the number of required buffers.

The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

Specified by:

createBufferPool in interface BufferPoolFactory

Parameters:

numRequiredBuffers - minimum number of network buffers in this pool

maxUsedBuffers - maximum number of network buffers this pool offers

Throws:

IOException

createBufferPool

public BufferPool createBufferPool(int numRequiredBuffers,
                                   int maxUsedBuffers,
                                   int numSubpartitions,
                                   int maxBuffersPerChannel)
                            throws IOException

Description copied from interface: BufferPoolFactory

Tries to create a buffer pool with an owner, which is guaranteed to provide at least the number of required buffers.

The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

Specified by:

createBufferPool in interface BufferPoolFactory

Parameters:

numRequiredBuffers - minimum number of network buffers in this pool

maxUsedBuffers - maximum number of network buffers this pool offers

numSubpartitions - number of subpartitions in this pool

maxBuffersPerChannel - maximum number of buffers to use for each channel

Throws:

IOException

destroyBufferPool

public void destroyBufferPool(BufferPool bufferPool)

Description copied from interface: BufferPoolFactory

Destroy callback for updating factory book keeping.

Specified by:

destroyBufferPool in interface BufferPoolFactory

destroyAllBufferPools

public void destroyAllBufferPools()

Destroys all buffer pools that allocate their buffers from this buffer pool (created via createBufferPool(int, int)).