Packages

o

org.apache.spark.sql.catalyst.expressions

FileSourceGeneratedMetadataAttribute

object FileSourceGeneratedMetadataAttribute

The internal representation of the FileSourceGeneratedMetadataAttribute. It sets metadata_col and file_source_generated_metadata_col to true in AttributeReference's metadata. In contrast to FileSourceConstantMetadataAttribute it represents metadata columns that are not constant per file and are generated as part of the scan.

- apply() will create a file source generated metadata attribute reference - unapply() will check if an attribute reference is the file source generated metadata attribute reference

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileSourceGeneratedMetadataAttribute
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY: String
  5. def apply(name: String, dataType: DataType, nullable: Boolean = false): AttributeReference

    We keep generated metadata attributes nullability configurable here: 1.

    We keep generated metadata attributes nullability configurable here: 1. Before passing to readers, we create generated metadata attributes as nullable; Because, for row_index, the readers do not consider the column required. row_index can be generated with null in the process by readers. 2. When applying the projection, we change the nullability back to not-nullable; For row_index, it is generated with nulls which are then replaced, so it will not be null in the returned output. See FileSourceStrategy for more information

  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def isGeneratedMetadataColumn(structField: StructField): Boolean

    True if structField represents a file source generated metadata column.

  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. def unapply(attr: AttributeReference): Option[AttributeReference]
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped