org.apache.flink.formats.parquet

Class ParquetColumnarRowInputFormat<SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

java.lang.Object

org.apache.flink.formats.parquet.ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT>

org.apache.flink.formats.parquet.ParquetColumnarRowInputFormat<SplitT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.table.data.RowData>, org.apache.flink.connector.file.src.reader.BulkFormat<org.apache.flink.table.data.RowData,SplitT>

public class ParquetColumnarRowInputFormat<SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>
extends ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT>

A ParquetVectorizedInputFormat to provide RowData iterator. Using ColumnarRowData to provide a row view of column batch.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

ParquetVectorizedInputFormat.ParquetReaderBatch<T>

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.reader.BulkFormat

org.apache.flink.connector.file.src.reader.BulkFormat.Reader<T>, org.apache.flink.connector.file.src.reader.BulkFormat.RecordIterator<T>

Constructor Summary

Constructors

Constructor and Description
ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType projectedType, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive) Constructor to create parquet format without extra fields.
ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType projectedType, org.apache.flink.table.types.logical.RowType producedType, ColumnBatchFactory<SplitT> batchFactory, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive) Constructor to create parquet format with extra fields created by ColumnBatchFactory.

Method Summary

Modifier and Type	Method and Description
static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> ParquetColumnarRowInputFormat<SplitT>	createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType producedRowType, List<String> partitionKeys, org.apache.flink.table.filesystem.PartitionFieldExtractor<SplitT> extractor, int batchSize, boolean isUtcTimestamp, boolean isCaseSensitive) Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.
protected ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>	createReaderBatch(org.apache.flink.table.data.vector.writable.WritableColumnVector[] writableVectors, org.apache.flink.table.data.vector.VectorizedColumnBatch columnarBatch, org.apache.flink.connector.file.src.util.Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>> recycler)
org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData>	getProducedType()
protected int	numBatchesToCirculate(org.apache.flink.configuration.Configuration config)

Methods inherited from class org.apache.flink.formats.parquet.ParquetVectorizedInputFormat

createReader, isSplittable, restoreReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetColumnarRowInputFormat

public ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                     org.apache.flink.table.types.logical.RowType projectedType,
                                     int batchSize,
                                     boolean isUtcTimestamp,
                                     boolean isCaseSensitive)

Constructor to create parquet format without extra fields.

ParquetColumnarRowInputFormat

public ParquetColumnarRowInputFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                     org.apache.flink.table.types.logical.RowType projectedType,
                                     org.apache.flink.table.types.logical.RowType producedType,
                                     ColumnBatchFactory<SplitT> batchFactory,
                                     int batchSize,
                                     boolean isUtcTimestamp,
                                     boolean isCaseSensitive)

Constructor to create parquet format with extra fields created by ColumnBatchFactory.

Parameters:

projectedType - the projected row type for parquet format, excludes extra fields.

producedType - the produced row type for this input format, includes extra fields.

batchFactory - factory for creating column batch, can cram in extra fields.

Method Detail

numBatchesToCirculate

protected int numBatchesToCirculate(org.apache.flink.configuration.Configuration config)

Overrides:

numBatchesToCirculate in class ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

createReaderBatch

protected ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData> createReaderBatch(org.apache.flink.table.data.vector.writable.WritableColumnVector[] writableVectors,
                                                                                                                 org.apache.flink.table.data.vector.VectorizedColumnBatch columnarBatch,
                                                                                                                 org.apache.flink.connector.file.src.util.Pool.Recycler<ParquetVectorizedInputFormat.ParquetReaderBatch<org.apache.flink.table.data.RowData>> recycler)

Specified by:

createReaderBatch in class ParquetVectorizedInputFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

Parameters:

writableVectors - vectors to be write

columnarBatch - vectors to be read

recycler - batch recycler

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> getProducedType()

createPartitionedFormat

public static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> ParquetColumnarRowInputFormat<SplitT> createPartitionedFormat(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                                                                                                                 org.apache.flink.table.types.logical.RowType producedRowType,
                                                                                                                                                 List<String> partitionKeys,
                                                                                                                                                 org.apache.flink.table.filesystem.PartitionFieldExtractor<SplitT> extractor,
                                                                                                                                                 int batchSize,
                                                                                                                                                 boolean isUtcTimestamp,
                                                                                                                                                 boolean isCaseSensitive)

Create a partitioned ParquetColumnarRowInputFormat, the partition columns can be generated by Path.