Packages

class RatePerMicroBatchProvider extends SimpleTableProvider with DataSourceRegister

A source that generates increment long values with timestamps. Each generated row has two columns: a timestamp column for the generated time and an auto increment long column starting with 0L.

This source supports the following options:

  • rowsPerMicroBatch (e.g. 100): How many rows should be generated per micro-batch.
  • numPartitions (e.g. 10, default: Spark's default parallelism): The partition number for the generated rows.
  • startTimestamp (e.g. 1000, default: 0): starting value of generated time
  • advanceMillisPerMicroBatch (e.g. 1000, default: 1000): the amount of time being advanced in generated time on each micro-batch.

Unlike rate data source, this data source provides a consistent set of input rows per micro-batch regardless of query execution (configuration of trigger, query being lagging, etc.), say, batch 0 will produce 0~999 and batch 1 will produce 1000~1999, and so on. Same applies to the generated time.

As the name represents, this data source only supports micro-batch read.

Linear Supertypes
DataSourceRegister, SimpleTableProvider, TableProvider, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RatePerMicroBatchProvider
  2. DataSourceRegister
  3. SimpleTableProvider
  4. TableProvider
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new RatePerMicroBatchProvider()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getTable(options: CaseInsensitiveStringMap): Table
    Definition Classes
    RatePerMicroBatchProvider → SimpleTableProvider
  11. def getTable(schema: StructType, partitioning: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    SimpleTableProvider → TableProvider
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def inferPartitioning(arg0: CaseInsensitiveStringMap): Array[Transform]
    Definition Classes
    TableProvider
  14. def inferSchema(options: CaseInsensitiveStringMap): StructType
    Definition Classes
    SimpleTableProvider → TableProvider
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. def shortName(): String

    The string that represents the format that this data source provider uses.

    The string that represents the format that this data source provider uses. This is overridden by children to provide a nice alias for the data source. For example:

    override def shortName(): String = "parquet"
    Definition Classes
    RatePerMicroBatchProvider → DataSourceRegister
    Since

    1.5.0

  20. def supportsExternalMetadata(): Boolean
    Definition Classes
    TableProvider
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from DataSourceRegister

Inherited from SimpleTableProvider

Inherited from TableProvider

Inherited from AnyRef

Inherited from Any

Ungrouped