org.apache.flink.formats.parquet

Class ParquetInputFormat<E>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,org.apache.flink.core.fs.FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<E>

org.apache.flink.formats.parquet.ParquetInputFormat<E>

Type Parameters:

E - The type of record to read.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>, org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

Direct Known Subclasses:

ParquetAvroInputFormat, ParquetMapInputFormat, ParquetPojoInputFormat, ParquetRowInputFormat

public abstract class ParquetInputFormat<E>
extends org.apache.flink.api.common.io.FileInputFormat<E>
implements org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>

The base InputFormat class to read from Parquet files. For specific return types the convert(Row) method need to be implemented.

Using ParquetRecordReader to read files instead of FSDataInputStream, we override open(FileInputSplit) and close() to change the behaviors.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARQUET_SKIP_CORRUPTED_RECORD The config parameter which defines whether to skip corrupted record.
static String	PARQUET_SKIP_WRONG_SCHEMA_SPLITS The config parameter which defines whether to skip file split with wrong schema.

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ParquetInputFormat(org.apache.flink.core.fs.Path path, org.apache.parquet.schema.MessageType messageType) Read parquet files with given parquet file schema.

Method Summary

Modifier and Type	Method and Description
void	close()
void	configure(org.apache.flink.configuration.Configuration parameters)
protected abstract E	convert(org.apache.flink.types.Row row) This ParquetInputFormat read parquet record as Row by default.
org.apache.flink.api.java.tuple.Tuple2<Long,Long>	getCurrentState()
protected String[]	getFieldNames() Get field names of read result.
protected org.apache.flink.api.common.typeinfo.TypeInformation[]	getFieldTypes() Get field types of read result.
protected org.apache.parquet.filter2.predicate.FilterPredicate	getPredicate()
E	nextRecord(E e)
void	open(org.apache.flink.core.fs.FileInputSplit split)
boolean	reachedEnd()
void	reopen(org.apache.flink.core.fs.FileInputSplit split, org.apache.flink.api.java.tuple.Tuple2<Long,Long> state)
void	selectFields(String[] fieldNames) Configures the fields to be read and returned by the ParquetInputFormat.
void	setFilterPredicate(org.apache.parquet.filter2.predicate.FilterPredicate filterPredicate)

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setFilesFilter, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, supportsMultiPaths, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PARQUET_SKIP_WRONG_SCHEMA_SPLITS

public static final String PARQUET_SKIP_WRONG_SCHEMA_SPLITS

The config parameter which defines whether to skip file split with wrong schema.

See Also:

Constant Field Values

PARQUET_SKIP_CORRUPTED_RECORD

public static final String PARQUET_SKIP_CORRUPTED_RECORD

The config parameter which defines whether to skip corrupted record.

See Also:

Constant Field Values

Constructor Detail

ParquetInputFormat

protected ParquetInputFormat(org.apache.flink.core.fs.Path path,
                             org.apache.parquet.schema.MessageType messageType)

Read parquet files with given parquet file schema.

Parameters:

path - The path of the file to read.

messageType - schema of parquet file

Method Detail

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

Specified by:

configure in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>

Overrides:

configure in class org.apache.flink.api.common.io.FileInputFormat<E>

selectFields

public void selectFields(String[] fieldNames)

Configures the fields to be read and returned by the ParquetInputFormat. Selected fields must be present in the configured schema.

Parameters:

fieldNames - Names of all selected fields.

setFilterPredicate

public void setFilterPredicate(org.apache.parquet.filter2.predicate.FilterPredicate filterPredicate)

getCurrentState

public org.apache.flink.api.java.tuple.Tuple2<Long,Long> getCurrentState()

Specified by:

getCurrentState in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>

open

public void open(org.apache.flink.core.fs.FileInputSplit split)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>

Overrides:

open in class org.apache.flink.api.common.io.FileInputFormat<E>

Throws:

IOException

reopen

public void reopen(org.apache.flink.core.fs.FileInputSplit split,
                   org.apache.flink.api.java.tuple.Tuple2<Long,Long> state)
            throws IOException

Specified by:

reopen in interface org.apache.flink.api.common.io.CheckpointableInputFormat<org.apache.flink.core.fs.FileInputSplit,org.apache.flink.api.java.tuple.Tuple2<Long,Long>>

Throws:

IOException

getFieldNames

protected String[] getFieldNames()

Get field names of read result.

Returns:

field names array

getFieldTypes

protected org.apache.flink.api.common.typeinfo.TypeInformation[] getFieldTypes()

Get field types of read result.

Returns:

field types array

getPredicate

@VisibleForTesting
protected org.apache.parquet.filter2.predicate.FilterPredicate getPredicate()

close

public void close()
           throws IOException

Specified by:

close in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>

Overrides:

close in class org.apache.flink.api.common.io.FileInputFormat<E>

Throws:

IOException

reachedEnd

public boolean reachedEnd()
                   throws IOException

Specified by:

reachedEnd in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>

Throws:

IOException

nextRecord

public E nextRecord(E e)
             throws IOException

Specified by:

nextRecord in interface org.apache.flink.api.common.io.InputFormat<E,org.apache.flink.core.fs.FileInputSplit>

Throws:

IOException

convert

protected abstract E convert(org.apache.flink.types.Row row)

This ParquetInputFormat read parquet record as Row by default. Sub classes of it can extend this method to further convert row to other types, such as POJO, Map or Tuple.

Parameters:

row - row read from parquet file

Returns:

E target result type