Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def canCast(from: DataType, to: DataType): Boolean

   Whether casting from as to is valid.

   Whether casting from as to is valid.

   Definition Classes

   TypeCoercion → TypeCoercionBase

6.  def castIfNotSameType(expr: Expression, dt: DataType): Expression

   Attributes

   protected

   Definition Classes

   TypeCoercionBase

7.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

10.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11.  def findCommonTypeDifferentOnlyInNullFlags(types: Seq[DataType]): Option[DataType]
12.  def findCommonTypeDifferentOnlyInNullFlags(t1: DataType, t2: DataType): Option[DataType]

    The method finds a common type for data types that differ only in nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType.

    The method finds a common type for data types that differ only in nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType. If the input types are different besides nullable flags, None is returned.

13.  def findCommonTypeForBinaryComparison(dt1: DataType, dt2: DataType, conf: SQLConf): Option[DataType]

    This function determines the target type of a comparison operator when one operand is a String and the other is not.

    This function determines the target type of a comparison operator when one operand is a String and the other is not. It also handles when one op is a Date and the other is a Timestamp by making the target type to be String.

14.  val findTightestCommonType: (DataType, DataType) ⇒ Option[DataType]

    Find the tightest common type of two types that might be used in a binary expression.

    Find the tightest common type of two types that might be used in a binary expression. This handles all numeric types except fixed-precision decimals interacting with each other or with primitive types, because in that case the precision and scale of the result depends on the operation. Those rules are implemented in DecimalPrecision.

    Definition Classes

    TypeCoercion → TypeCoercionBase

15.  def findTypeForComplex(t1: DataType, t2: DataType, findTypeFunc: (DataType, DataType) ⇒ Option[DataType]): Option[DataType]

    Attributes

    protected

    Definition Classes

    TypeCoercionBase

16.  def findWiderCommonType(types: Seq[DataType]): Option[DataType]

    Looking for a widened data type of a given sequence of data types with some acceptable loss of precision.

    Looking for a widened data type of a given sequence of data types with some acceptable loss of precision. E.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double.

    Definition Classes

    TypeCoercion → TypeCoercionBase

17.  def findWiderTypeForDecimal(dt1: DataType, dt2: DataType): Option[DataType]

    Finds a wider type when one or both types are decimals.

    Finds a wider type when one or both types are decimals. If the wider decimal type exceeds system limitation, this rule will truncate the decimal type. If a decimal and other fractional types are compared, returns a double type.

    Attributes

    protected

    Definition Classes

    TypeCoercionBase

18.  def findWiderTypeForTwo(t1: DataType, t2: DataType): Option[DataType]

    Looking for a widened data type of two given data types with some acceptable loss of precision.

    Looking for a widened data type of two given data types with some acceptable loss of precision. E.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double.

    Definition Classes

    TypeCoercion → TypeCoercionBase

19.  def findWiderTypeWithoutStringPromotion(types: Seq[DataType]): Option[DataType]

    Definition Classes

    TypeCoercionBase

20.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

21.  def hasStringType(dt: DataType): Boolean

    Whether the data type contains StringType.

    Whether the data type contains StringType.

    Annotations

    @tailrec()

22.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

23.  def haveSameType(types: Seq[DataType]): Boolean

    Check whether the given types are equal ignoring nullable, containsNull and valueContainsNull.

    Check whether the given types are equal ignoring nullable, containsNull and valueContainsNull.

    Definition Classes

    TypeCoercionBase

24.  def implicitCast(e: Expression, expectedType: AbstractDataType): Option[Expression]

    Given an expected data type, try to cast the expression and return the cast expression.

    Given an expected data type, try to cast the expression and return the cast expression.

    If the expression already fits the input type, we simply return the expression itself. If the expression has an incompatible type that cannot be implicitly cast, return None.

    Definition Classes

    TypeCoercion → TypeCoercionBase

25.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

26.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

27.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

28.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

29.  val numericPrecedence: IndexedSeq[NumericType with Product with Serializable]
30.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

31.  def toString(): String

    Definition Classes

    AnyRef → Any

32.  def typeCoercionRules: List[Rule[LogicalPlan]]

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    Definition Classes

    TypeCoercion → TypeCoercionBase

33.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

34.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

35.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()

36.  object BooleanEquality extends Rule[LogicalPlan] with TypeCoercionRule

    Changes numeric values to booleans so that expressions like true = 1 can be evaluated.

37.  object DateTimeOperations extends Rule[LogicalPlan] with TypeCoercionRule
38.  object PromoteStrings extends Rule[LogicalPlan] with TypeCoercionRule

    Promotes strings that appear in arithmetic expressions.

39.  object CaseWhenCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of a CASE WHEN statement to a common type.

    Coerces the type of different branches of a CASE WHEN statement to a common type.

    Definition Classes

    TypeCoercionBase

40.  object ConcatCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the types of Concat children to expected ones.

    Coerces the types of Concat children to expected ones.

    If spark.sql.function.concatBinaryAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

    Definition Classes

    TypeCoercionBase

41.  object Division extends Rule[LogicalPlan] with TypeCoercionRule

    Hive only performs integral division with the DIV operator.

    Hive only performs integral division with the DIV operator. The arguments to / are always converted to fractional types.

    Definition Classes

    TypeCoercionBase

42.  object EltCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the types of Elt children to expected ones.

    Coerces the types of Elt children to expected ones.

    If spark.sql.function.eltOutputAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

    Definition Classes

    TypeCoercionBase

43.  object FunctionArgumentConversion extends Rule[LogicalPlan] with TypeCoercionRule

    This ensure that the types for various functions are as expected.

    This ensure that the types for various functions are as expected.

    Definition Classes

    TypeCoercionBase

44.  object IfCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of If statement to a common type.

    Coerces the type of different branches of If statement to a common type.

    Definition Classes

    TypeCoercionBase

45.  object ImplicitTypeCasts extends Rule[LogicalPlan] with TypeCoercionRule

    Casts types according to the expected input types for Expressions.

    Casts types according to the expected input types for Expressions.

    Definition Classes

    TypeCoercionBase

46.  object InConversion extends Rule[LogicalPlan] with TypeCoercionRule

    Handles type coercion for both IN expression with subquery and IN expressions without subquery.

    Handles type coercion for both IN expression with subquery and IN expressions without subquery. 1. In the first case, find the common type by comparing the left hand side (LHS) expression types against corresponding right hand side (RHS) expression derived from the subquery expression's plan output. Inject appropriate casts in the LHS and RHS side of IN expression.

    2. In the second case, convert the value and in list expressions to the common operator type by looking at all the argument types and finding the closest one that all the arguments can be cast to. When no common operator type is found the original expression will be returned and an Analysis Exception will be raised at the type checking phase.

    Definition Classes

    TypeCoercionBase

47.  object IntegralDivision extends Rule[LogicalPlan] with TypeCoercionRule

    The DIV operator always returns long-type value.

    The DIV operator always returns long-type value. This rule cast the integral inputs to long type, to avoid overflow during calculation.

    Definition Classes

    TypeCoercionBase

48.  object MapZipWithCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces key types of two different MapType arguments of the MapZipWith expression to a common type.

    Coerces key types of two different MapType arguments of the MapZipWith expression to a common type.

    Definition Classes

    TypeCoercionBase

49.  object StackCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces NullTypes in the Stack expression to the column types of the corresponding positions.

    Coerces NullTypes in the Stack expression to the column types of the corresponding positions.

    Definition Classes

    TypeCoercionBase

50.  object StringLiteralCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround.

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround. TODO(SPARK-28589): implement ANSI type type coercion and handle string literals.

    Definition Classes

    TypeCoercionBase

51.  object WidenSetOperationTypes extends Rule[LogicalPlan]

    Widens the data types of the children of Union/Except/Intersect.

    Widens the data types of the children of Union/Except/Intersect. 1. When ANSI mode is off: Loosely based on rules from "Hadoop: The Definitive Guide" 2nd edition, by Tom White

    The implicit conversion rules can be summarized as follows:

    • Any integral numeric type can be implicitly converted to a wider type.
    • All the integral numeric types, FLOAT, and (perhaps surprisingly) STRING can be implicitly converted to DOUBLE.
    • TINYINT, SMALLINT, and INT can all be converted to FLOAT.
    • BOOLEAN types cannot be converted to any other type.
    • Any integral numeric type can be implicitly converted to decimal type.
    • two different decimal types will be converted into a wider decimal type for both of them.
    • decimal type will be converted into double if there float or double together with it.

    All types when UNION-ed with strings will be promoted to strings. Other string conversions are handled by PromoteStrings.

    Widening types might result in loss of precision in the following cases:

    • IntegerType to FloatType
    • LongType to FloatType
    • LongType to DoubleType
    • DecimalType to Double

    2. When ANSI mode is on: The implicit conversion is determined by the closest common data type from the precedent lists from left and right child. See the comments of Object AnsiTypeCoercion for details.

    Definition Classes

    TypeCoercionBase

52.  object WindowFrameCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Cast WindowFrame boundaries to the type they operate upon.

    Cast WindowFrame boundaries to the type they operate upon.

    Definition Classes

    TypeCoercionBase