org.apache.hudi

Class DataSourceUtils

java.lang.Object

org.apache.hudi.DataSourceUtils

public class DataSourceUtils
extends Object

Utilities used throughout the data source.

Constructor Summary

Constructors

Constructor and Description
DataSourceUtils()

Method Summary

Modifier and Type	Method and Description
static void	checkRequiredProperties(TypedProperties props, List<String> checkPropNames)
static SparkRDDWriteClient	createHoodieClient(org.apache.spark.api.java.JavaSparkContext jssc, String schemaStr, String basePath, String tblName, Map<String,String> parameters)
static HoodieWriteConfig	createHoodieConfig(String schemaStr, String basePath, String tblName, Map<String,String> parameters)
static HoodieRecord	createHoodieRecord(org.apache.avro.generic.GenericRecord gr, Comparable orderingVal, HoodieKey hKey, String payloadClass)
static HoodieRecord	createHoodieRecord(org.apache.avro.generic.GenericRecord gr, HoodieKey hKey, String payloadClass)
static HoodieRecordPayload	createPayload(String payloadClass, org.apache.avro.generic.GenericRecord record) Create a payload class via reflection, do not ordering/precombine value.
static HoodieRecordPayload	createPayload(String payloadClass, org.apache.avro.generic.GenericRecord record, Comparable orderingVal) Create a payload class via reflection, passing in an ordering/precombine value.
static Option<BulkInsertPartitioner<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>>	createUserDefinedBulkInsertPartitionerWithRows(HoodieWriteConfig config) Create a UserDefinedBulkInsertPartitionerRows class via reflection, if the class name of UserDefinedBulkInsertPartitioner is configured through the HoodieWriteConfig.
static HoodieWriteResult	doDeleteOperation(SparkRDDWriteClient client, org.apache.spark.api.java.JavaRDD<HoodieKey> hoodieKeys, String instantTime)
static HoodieWriteResult	doDeletePartitionsOperation(SparkRDDWriteClient client, List<String> partitionsToDelete, String instantTime)
static HoodieWriteResult	doWriteOperation(SparkRDDWriteClient client, org.apache.spark.api.java.JavaRDD<HoodieRecord> hoodieRecords, String instantTime, WriteOperationType operation)
static org.apache.spark.api.java.JavaRDD<HoodieRecord>	dropDuplicates(org.apache.spark.api.java.JavaSparkContext jssc, org.apache.spark.api.java.JavaRDD<HoodieRecord> incomingHoodieRecords, HoodieWriteConfig writeConfig) Drop records already present in the dataset.
static org.apache.spark.api.java.JavaRDD<HoodieRecord>	dropDuplicates(org.apache.spark.api.java.JavaSparkContext jssc, org.apache.spark.api.java.JavaRDD<HoodieRecord> incomingHoodieRecords, Map<String,String> parameters)
static Map<String,String>	getExtraMetadata(Map<String,String> properties)
static String	getTablePath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path[] userProvidedPaths)
static void	tryOverrideParquetWriteLegacyFormatProperty(Map<String,String> properties, org.apache.spark.sql.types.StructType schema) Checks whether default value (false) of "hoodie.parquet.writelegacyformat.enabled" should be overridden in case: Property has not been explicitly set by the writer Data schema contains DecimalType that would be affected by it If both of the aforementioned conditions are true, will override the default value of the config (by essentially setting the value) to make sure that the produced Parquet data files could be read by AvroParquetReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataSourceUtils

public DataSourceUtils()

Method Detail

getTablePath

public static String getTablePath(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path[] userProvidedPaths)
                           throws IOException

Throws:

IOException

createUserDefinedBulkInsertPartitionerWithRows

public static Option<BulkInsertPartitioner<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>> createUserDefinedBulkInsertPartitionerWithRows(HoodieWriteConfig config)
                                                                                                                                            throws HoodieException

Create a UserDefinedBulkInsertPartitionerRows class via reflection,
if the class name of UserDefinedBulkInsertPartitioner is configured through the HoodieWriteConfig.

Throws:

HoodieException

See Also:

HoodieWriteConfig.getUserDefinedBulkInsertPartitionerClass()

createPayload

public static HoodieRecordPayload createPayload(String payloadClass,
                                                org.apache.avro.generic.GenericRecord record,
                                                Comparable orderingVal)
                                         throws IOException

Create a payload class via reflection, passing in an ordering/precombine value.

Throws:

IOException

getExtraMetadata

public static Map<String,String> getExtraMetadata(Map<String,String> properties)

createPayload

public static HoodieRecordPayload createPayload(String payloadClass,
                                                org.apache.avro.generic.GenericRecord record)
                                         throws IOException

Create a payload class via reflection, do not ordering/precombine value.

Throws:

IOException

checkRequiredProperties

public static void checkRequiredProperties(TypedProperties props,
                                           List<String> checkPropNames)

createHoodieConfig

public static HoodieWriteConfig createHoodieConfig(String schemaStr,
                                                   String basePath,
                                                   String tblName,
                                                   Map<String,String> parameters)

createHoodieClient

public static SparkRDDWriteClient createHoodieClient(org.apache.spark.api.java.JavaSparkContext jssc,
                                                     String schemaStr,
                                                     String basePath,
                                                     String tblName,
                                                     Map<String,String> parameters)

doWriteOperation

public static HoodieWriteResult doWriteOperation(SparkRDDWriteClient client,
                                                 org.apache.spark.api.java.JavaRDD<HoodieRecord> hoodieRecords,
                                                 String instantTime,
                                                 WriteOperationType operation)
                                          throws HoodieException

Throws:

HoodieException

doDeleteOperation

public static HoodieWriteResult doDeleteOperation(SparkRDDWriteClient client,
                                                  org.apache.spark.api.java.JavaRDD<HoodieKey> hoodieKeys,
                                                  String instantTime)

doDeletePartitionsOperation

public static HoodieWriteResult doDeletePartitionsOperation(SparkRDDWriteClient client,
                                                            List<String> partitionsToDelete,
                                                            String instantTime)

createHoodieRecord

public static HoodieRecord createHoodieRecord(org.apache.avro.generic.GenericRecord gr,
                                              Comparable orderingVal,
                                              HoodieKey hKey,
                                              String payloadClass)
                                       throws IOException

Throws:

IOException

createHoodieRecord

public static HoodieRecord createHoodieRecord(org.apache.avro.generic.GenericRecord gr,
                                              HoodieKey hKey,
                                              String payloadClass)
                                       throws IOException

Throws:

IOException

dropDuplicates

public static org.apache.spark.api.java.JavaRDD<HoodieRecord> dropDuplicates(org.apache.spark.api.java.JavaSparkContext jssc,
                                                                             org.apache.spark.api.java.JavaRDD<HoodieRecord> incomingHoodieRecords,
                                                                             HoodieWriteConfig writeConfig)

Drop records already present in the dataset.

Parameters:

jssc - JavaSparkContext

incomingHoodieRecords - HoodieRecords to deduplicate

writeConfig - HoodieWriteConfig

dropDuplicates

public static org.apache.spark.api.java.JavaRDD<HoodieRecord> dropDuplicates(org.apache.spark.api.java.JavaSparkContext jssc,
                                                                             org.apache.spark.api.java.JavaRDD<HoodieRecord> incomingHoodieRecords,
                                                                             Map<String,String> parameters)

tryOverrideParquetWriteLegacyFormatProperty

public static void tryOverrideParquetWriteLegacyFormatProperty(Map<String,String> properties,
                                                               org.apache.spark.sql.types.StructType schema)

Checks whether default value (false) of "hoodie.parquet.writelegacyformat.enabled" should be overridden in case:

• Property has not been explicitly set by the writer
• Data schema contains DecimalType that would be affected by it

If both of the aforementioned conditions are true, will override the default value of the config (by essentially setting the value) to make sure that the produced Parquet data files could be read by AvroParquetReader

Parameters:

properties - properties specified by the writer

schema - schema of the dataset being written