org.apache.flink.formats.parquet

Class ParquetAvroInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,org.apache.flink.core.fs.FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<E>

org.apache.flink.formats.parquet.ParquetInputFormat<org.apache.avro.generic.GenericRecord>

org.apache.flink.formats.parquet.ParquetAvroInputFormat

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>, org.apache.flink.api.common.io.InputFormat<org.apache.avro.generic.GenericRecord,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.avro.generic.GenericRecord>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

public class ParquetAvroInputFormat
extends ParquetInputFormat<org.apache.avro.generic.GenericRecord>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.avro.generic.GenericRecord>

An implementation of ParquetInputFormat to read records from Parquet files and convert them to Avro GenericRecord. To use it the user needs to add flink-avro optional dependency to the classpath. Usage:

 final ParquetAvroInputFormat inputFormat = new ParquetAvroInputFormat(new Path(filePath), parquetSchema);
 DataSource<GenericRecord> source = env.createInput(inputFormat, new GenericRecordAvroTypeInfo(inputFormat.getAvroSchema()));

 

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.formats.parquet.ParquetInputFormat

PARQUET_SKIP_CORRUPTED_RECORD, PARQUET_SKIP_WRONG_SCHEMA_SPLITS

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
ParquetAvroInputFormat(org.apache.flink.core.fs.Path filePath, org.apache.parquet.schema.MessageType messageType)

Method Summary

Modifier and Type	Method and Description
protected org.apache.avro.generic.GenericRecord	convert(org.apache.flink.types.Row row) This ParquetInputFormat read parquet record as Row by default.
org.apache.avro.Schema	getAvroSchema()
org.apache.flink.formats.avro.typeutils.GenericRecordAvroTypeInfo	getProducedType()
void	selectFields(String[] fieldNames) Configures the fields to be read and returned by the ParquetInputFormat.

Methods inherited from class org.apache.flink.formats.parquet.ParquetInputFormat

close, configure, getCurrentState, getFieldNames, getFieldTypes, getPredicate, nextRecord, open, reachedEnd, reopen, setFilterPredicate

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, supportsMultiPaths, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetAvroInputFormat

public ParquetAvroInputFormat(org.apache.flink.core.fs.Path filePath,
                              org.apache.parquet.schema.MessageType messageType)

Method Detail

selectFields

public void selectFields(String[] fieldNames)

Description copied from class: ParquetInputFormat

Configures the fields to be read and returned by the ParquetInputFormat. Selected fields must be present in the configured schema.

Overrides:

selectFields in class ParquetInputFormat<org.apache.avro.generic.GenericRecord>

Parameters:

fieldNames - Names of all selected fields.

convert

protected org.apache.avro.generic.GenericRecord convert(org.apache.flink.types.Row row)

Description copied from class: ParquetInputFormat

This ParquetInputFormat read parquet record as Row by default. Sub classes of it can extend this method to further convert row to other types, such as POJO, Map or Tuple.

Specified by:

convert in class ParquetInputFormat<org.apache.avro.generic.GenericRecord>

Parameters:

row - row read from parquet file

Returns:

E target result type

getProducedType

public org.apache.flink.formats.avro.typeutils.GenericRecordAvroTypeInfo getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.avro.generic.GenericRecord>

getAvroSchema

public org.apache.avro.Schema getAvroSchema()