c

ai.chronon.api

StagingQuery

class StagingQuery extends TBase[StagingQuery, _Fields] with Serializable with Cloneable with Comparable[StagingQuery]

Staging Query encapsulates arbitrary spark computation. One key feature is that the computation follows a "fill-what's-missing" pattern. Basically instead of explicitly specifying dates you specify two macros. {{ start_date }} and {{end_date}}. Chronon will pass in earliest-missing-partition for start_date and execution-date / today for end_date. So the query will compute multiple partitions at once.

Annotations
@SuppressWarnings() @Generated()
Linear Supertypes
Cloneable, TBase[StagingQuery, _Fields], Serializable, TSerializable, Comparable[StagingQuery], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StagingQuery
  2. Cloneable
  3. TBase
  4. Serializable
  5. TSerializable
  6. Comparable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StagingQuery(other: StagingQuery)

    Performs a deep copy on other.

  2. new StagingQuery()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addToSetups(elem: String): Unit
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clear(): Unit
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Override()
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  8. def compareTo(other: StagingQuery): Int
    Definition Classes
    StagingQuery → Comparable
    Annotations
    @Override()
  9. def deepCopy(): StagingQuery
    Definition Classes
    StagingQuery → TBase
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(that: StagingQuery): Boolean
  12. def equals(that: AnyRef): Boolean
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  13. def fieldForId(fieldId: Int): _Fields
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Nullable()
  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  15. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def getFieldValue(field: _Fields): AnyRef
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Nullable()
  17. def getMetaData(): MetaData

    Contains name, team, output_namespace, execution parameters etc.

    Contains name, team, output_namespace, execution parameters etc. Things that don't change the semantics of the computation itself.

    Annotations
    @Nullable()
  18. def getQuery(): String

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    • {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.
    • {{ end_date }} is the end partition of the computing range.
    • {{ latest_date }} is the end partition independent of the computing range (meant for cumulative sources).
    • {{ max_date(table=namespace.my_table) }} is the max partition available for a given table.
    Annotations
    @Nullable()
  19. def getSetups(): List[String]

    Spark SQL setup statements.

    Spark SQL setup statements. Used typically to register UDFs.

    Annotations
    @Nullable()
  20. def getSetupsIterator(): Iterator[String]
    Annotations
    @Nullable()
  21. def getSetupsSize(): Int
  22. def getStartPartition(): String

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

    Annotations
    @Nullable()
  23. def hashCode(): Int
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  24. def isCreateView(): Boolean

    If true, creates a view in the warehouse (for intermediate tables).

    If true, creates a view in the warehouse (for intermediate tables). If false, creates a table in the warehouse (for final tables).

  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. def isSet(field: _Fields): Boolean

    Returns true if field corresponding to fieldID is set (has been assigned a value) and false otherwise

    Returns true if field corresponding to fieldID is set (has been assigned a value) and false otherwise

    Definition Classes
    StagingQuery → TBase
  27. def isSetCreateView(): Boolean

    Returns true if field createView is set (has been assigned a value) and false otherwise

  28. def isSetMetaData(): Boolean

    Returns true if field metaData is set (has been assigned a value) and false otherwise

  29. def isSetQuery(): Boolean

    Returns true if field query is set (has been assigned a value) and false otherwise

  30. def isSetSetups(): Boolean

    Returns true if field setups is set (has been assigned a value) and false otherwise

  31. def isSetStartPartition(): Boolean

    Returns true if field startPartition is set (has been assigned a value) and false otherwise

  32. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  33. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  35. def read(iprot: TProtocol): Unit
    Definition Classes
    StagingQuery → TSerializable
  36. def setCreateView(createView: Boolean): StagingQuery

    If true, creates a view in the warehouse (for intermediate tables).

    If true, creates a view in the warehouse (for intermediate tables). If false, creates a table in the warehouse (for final tables).

  37. def setCreateViewIsSet(value: Boolean): Unit
  38. def setFieldValue(field: _Fields, value: AnyRef): Unit
    Definition Classes
    StagingQuery → TBase
  39. def setMetaData(metaData: MetaData): StagingQuery

    Contains name, team, output_namespace, execution parameters etc.

    Contains name, team, output_namespace, execution parameters etc. Things that don't change the semantics of the computation itself.

  40. def setMetaDataIsSet(value: Boolean): Unit
  41. def setQuery(query: String): StagingQuery

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    • {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.
    • {{ end_date }} is the end partition of the computing range.
    • {{ latest_date }} is the end partition independent of the computing range (meant for cumulative sources).
    • {{ max_date(table=namespace.my_table) }} is the max partition available for a given table.
  42. def setQueryIsSet(value: Boolean): Unit
  43. def setSetups(setups: List[String]): StagingQuery

    Spark SQL setup statements.

    Spark SQL setup statements. Used typically to register UDFs.

  44. def setSetupsIsSet(value: Boolean): Unit
  45. def setStartPartition(startPartition: String): StagingQuery

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

  46. def setStartPartitionIsSet(value: Boolean): Unit
  47. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  48. def toString(): String
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  49. def unsetCreateView(): Unit
  50. def unsetMetaData(): Unit
  51. def unsetQuery(): Unit
  52. def unsetSetups(): Unit
  53. def unsetStartPartition(): Unit
  54. def validate(): Unit
  55. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  56. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  57. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  58. def write(oprot: TProtocol): Unit
    Definition Classes
    StagingQuery → TSerializable

Inherited from Cloneable

Inherited from TBase[StagingQuery, _Fields]

Inherited from Serializable

Inherited from TSerializable

Inherited from Comparable[StagingQuery]

Inherited from AnyRef

Inherited from Any

Ungrouped