org.apache.flink.runtime.operators.sort

Class AbstractMergeIterator<T1,T2,O>

java.lang.Object

org.apache.flink.runtime.operators.sort.AbstractMergeIterator<T1,T2,O>

All Implemented Interfaces:

JoinTaskIterator<T1,T2,O>

Direct Known Subclasses:

AbstractMergeInnerJoinIterator, AbstractMergeOuterJoinIterator

public abstract class AbstractMergeIterator<T1,T2,O>
extends Object
implements JoinTaskIterator<T1,T2,O>

Field Summary

Fields

Modifier and Type	Field and Description
protected T2	blockHeadCopy
protected T1	copy1
protected T2	copy2
protected KeyGroupedIterator<T1>	iterator1
protected KeyGroupedIterator<T2>	iterator2
protected org.apache.flink.api.common.typeutils.TypePairComparator<T1,T2>	pairComparator
protected org.apache.flink.api.common.typeutils.TypeSerializer<T1>	serializer1
protected org.apache.flink.api.common.typeutils.TypeSerializer<T2>	serializer2
protected T1	spillHeadCopy

Constructor Summary

Constructors

Constructor and Description

AbstractMergeIterator(org.apache.flink.util.MutableObjectIterator<T1> input1, org.apache.flink.util.MutableObjectIterator<T2> input2, org.apache.flink.api.common.typeutils.TypeSerializer<T1> serializer1, org.apache.flink.api.common.typeutils.TypeComparator<T1> comparator1, org.apache.flink.api.common.typeutils.TypeSerializer<T2> serializer2, org.apache.flink.api.common.typeutils.TypeComparator<T2> comparator2, org.apache.flink.api.common.typeutils.TypePairComparator<T1,T2> pairComparator, MemoryManager memoryManager, IOManager ioManager, int numMemoryPages, AbstractInvokable parentTask)

Method Summary

Modifier and Type	Method and Description
void	abort() Aborts the matching process.
abstract boolean	callWithNextKey(org.apache.flink.api.common.functions.FlatJoinFunction<T1,T2,O> joinFunction, org.apache.flink.util.Collector<O> collector) Calls the JoinFunction#join() method for all two key-value pairs that share the same key and come from different inputs.
void	close() General-purpose close method.
protected abstract <T> T	createCopy(org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer, T value, T reuse) Copies an instance of the given type, potentially reusing the object passed as the reuse parameter, which may be null.
protected abstract <T> KeyGroupedIterator<T>	createKeyGroupedIterator(org.apache.flink.util.MutableObjectIterator<T> input, org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer, org.apache.flink.api.common.typeutils.TypeComparator<T> comparator)
protected void	crossMatchingGroup(Iterator<T1> values1, Iterator<T2> values2, org.apache.flink.api.common.functions.FlatJoinFunction<T1,T2,O> joinFunction, org.apache.flink.util.Collector<O> collector)
void	open() General-purpose open method.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pairComparator

protected org.apache.flink.api.common.typeutils.TypePairComparator<T1,T2> pairComparator

iterator1

protected KeyGroupedIterator<T1> iterator1

iterator2

protected KeyGroupedIterator<T2> iterator2

serializer1

protected final org.apache.flink.api.common.typeutils.TypeSerializer<T1> serializer1

serializer2

protected final org.apache.flink.api.common.typeutils.TypeSerializer<T2> serializer2

copy1

protected T1 copy1

spillHeadCopy

protected T1 spillHeadCopy

copy2

protected T2 copy2

blockHeadCopy

protected T2 blockHeadCopy

Constructor Detail

AbstractMergeIterator

public AbstractMergeIterator(org.apache.flink.util.MutableObjectIterator<T1> input1,
                             org.apache.flink.util.MutableObjectIterator<T2> input2,
                             org.apache.flink.api.common.typeutils.TypeSerializer<T1> serializer1,
                             org.apache.flink.api.common.typeutils.TypeComparator<T1> comparator1,
                             org.apache.flink.api.common.typeutils.TypeSerializer<T2> serializer2,
                             org.apache.flink.api.common.typeutils.TypeComparator<T2> comparator2,
                             org.apache.flink.api.common.typeutils.TypePairComparator<T1,T2> pairComparator,
                             MemoryManager memoryManager,
                             IOManager ioManager,
                             int numMemoryPages,
                             AbstractInvokable parentTask)
                      throws MemoryAllocationException

Throws:

MemoryAllocationException

Method Detail

open

public void open()
          throws IOException

Description copied from interface: JoinTaskIterator

General-purpose open method. Initializes the internal strategy (for example triggers the sorting of the inputs or starts building hash tables).

Specified by:

open in interface JoinTaskIterator<T1,T2,O>

Throws:

IOException - Thrown, if an I/O error occurred while preparing the data. An example is a failing external sort.

close

public void close()

Description copied from interface: JoinTaskIterator

General-purpose close method. Works after the principle of best effort. The internal structures are released, but errors that occur on the way are not reported.

Specified by:

close in interface JoinTaskIterator<T1,T2,O>

abort

public void abort()

Description copied from interface: JoinTaskIterator

Aborts the matching process. This extra abort method is supplied, because a significant time may pass while calling the match stub with the cross product of all values that share the same key. A call to this abort method signals an interrupt to that procedure.

Specified by:

abort in interface JoinTaskIterator<T1,T2,O>

callWithNextKey

public abstract boolean callWithNextKey(org.apache.flink.api.common.functions.FlatJoinFunction<T1,T2,O> joinFunction,
                                        org.apache.flink.util.Collector<O> collector)
                                 throws Exception

Calls the JoinFunction#join() method for all two key-value pairs that share the same key and come from different inputs. The output of the join() method is forwarded.

This method first zig-zags between the two sorted inputs in order to find a common key, and then calls the join stub with the cross product of the values.

Specified by:

callWithNextKey in interface JoinTaskIterator<T1,T2,O>

Parameters:

joinFunction - The match stub containing the match function which is called with the keys.

collector - The collector to pass the match function.

Returns:

True, if a next key exists, false if no more keys exist.

Throws:

Exception - Forwards all exceptions from the user code and the I/O system.

See Also:

JoinTaskIterator.callWithNextKey(org.apache.flink.api.common.functions.FlatJoinFunction, org.apache.flink.util.Collector)

crossMatchingGroup

protected void crossMatchingGroup(Iterator<T1> values1,
                                  Iterator<T2> values2,
                                  org.apache.flink.api.common.functions.FlatJoinFunction<T1,T2,O> joinFunction,
                                  org.apache.flink.util.Collector<O> collector)
                           throws Exception

Throws:

Exception

createKeyGroupedIterator

protected abstract <T> KeyGroupedIterator<T> createKeyGroupedIterator(org.apache.flink.util.MutableObjectIterator<T> input,
                                                                      org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer,
                                                                      org.apache.flink.api.common.typeutils.TypeComparator<T> comparator)

createCopy

protected abstract <T> T createCopy(org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer,
                                    T value,
                                    T reuse)

Copies an instance of the given type, potentially reusing the object passed as the reuse parameter, which may be null.