t

org.apache.spark.sql.connector.write

RequiresDistributionAndOrdering

trait RequiresDistributionAndOrdering extends Write

A write that requires a specific distribution and ordering of data.

Annotations
@Experimental()
Since

3.2.0

Linear Supertypes
Write, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RequiresDistributionAndOrdering
  2. Write
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def requiredDistribution(): Distribution

    Returns the distribution required by this write.

    Returns the distribution required by this write.

    Spark will distribute incoming records across partitions to satisfy the required distribution before passing the records to the data source table on write.

    Batch and micro-batch writes can request a particular data distribution. If a distribution is requested in the micro-batch context, incoming records in each micro batch will satisfy the required distribution (but not across micro batches). The continuous execution mode continuously processes streaming data and does not support distribution requirements.

    Implementations may return UnspecifiedDistribution if they don't require any specific distribution of data on write.

    returns

    the required distribution

  2. abstract def requiredOrdering(): Array[SortOrder]

    Returns the ordering required by this write.

    Returns the ordering required by this write.

    Spark will order incoming records within partitions to satisfy the required ordering before passing those records to the data source table on write.

    Batch and micro-batch writes can request a particular data ordering. If an ordering is requested in the micro-batch context, incoming records in each micro batch will satisfy the required ordering (but not across micro batches). The continuous execution mode continuously processes streaming data and does not support ordering requirements.

    Implementations may return an empty array if they don't require any specific ordering of data on write.

    returns

    the required ordering

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def description(): String

    Returns the description associated with this write.

    Returns the description associated with this write.

    Definition Classes
    Write
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. def requiredNumPartitions(): Int

    Returns the number of partitions required by this write.

    Returns the number of partitions required by this write.

    Implementations may override this to require a specific number of input partitions.

    Note that Spark doesn't support the number of partitions on UnspecifiedDistribution, the query will fail if the number of partitions are provided but the distribution is unspecified.

    returns

    the required number of partitions, any value less than 1 mean no requirement.

  17. def supportedCustomMetrics(): Array[CustomMetric]

    Returns an array of supported custom metrics with name and description.

    Returns an array of supported custom metrics with name and description. By default it returns empty array.

    Definition Classes
    Write
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toBatch(): BatchWrite

    Returns a BatchWrite to write data to batch source.

    Returns a BatchWrite to write data to batch source. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this write returns TableCapability#BATCH_WRITE support in its Table#capabilities().

    Definition Classes
    Write
  20. def toStreaming(): StreamingWrite

    Returns a StreamingWrite to write data to streaming source.

    Returns a StreamingWrite to write data to streaming source. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this write returns TableCapability#STREAMING_WRITE support in its Table#capabilities().

    Definition Classes
    Write
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Write

Inherited from AnyRef

Inherited from Any

Ungrouped