Type Members

1.  case class AddColumns(table: LogicalPlan, columnsToAdd: Seq[QualifiedColType]) extends LogicalPlan with AlterTableCommand with Product with Serializable

   The logical plan of the ALTER TABLE ...

   The logical plan of the ALTER TABLE ... ADD COLUMNS command.

2.  case class AddPartitions(table: LogicalPlan, parts: Seq[PartitionSpec], ifNotExists: Boolean) extends LogicalPlan with V2PartitionCommand with Product with Serializable

   The logical plan of the ALTER TABLE ADD PARTITION command.

   The logical plan of the ALTER TABLE ADD PARTITION command.

   The syntax of this command is:

   ALTER TABLE table ADD [IF NOT EXISTS]
               PARTITION spec1 [LOCATION 'loc1'][, PARTITION spec2 [LOCATION 'loc2'], ...];

3.  case class Aggregate(groupingExpressions: Seq[Expression], aggregateExpressions: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

   This is a Group by operator with the aggregate functions and projections.

   This is a Group by operator with the aggregate functions and projections.

   groupingExpressions

   expressions for grouping keys

   aggregateExpressions

   expressions for a project list, which could contain AggregateExpressions. Note: Currently, aggregateExpressions is the project list of this Group by operator. Before separating projection from grouping and aggregate, we should avoid expression-level optimization on aggregateExpressions, which could reference an expression in groupingExpressions. For example, see the rule org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps

4.  case class AlterColumn(table: LogicalPlan, column: FieldName, dataType: Option[DataType], nullable: Option[Boolean], comment: Option[String], position: Option[FieldPosition], setDefaultExpression: Option[String]) extends LogicalPlan with AlterTableCommand with Product with Serializable

   The logical plan of the ALTER TABLE ...

   The logical plan of the ALTER TABLE ... ALTER COLUMN command.

5.  trait AlterTableCommand extends LogicalPlan with UnaryCommand

   The base trait for commands that need to alter a v2 table with TableChanges.

6.  case class AlterViewAs(child: LogicalPlan, originalText: String, query: LogicalPlan) extends LogicalPlan with BinaryCommand with Product with Serializable

   The logical plan of the ALTER VIEW ...

   The logical plan of the ALTER VIEW ... AS command.

7.  trait AnalysisHelper extends QueryPlan[LogicalPlan]

   AnalysisHelper defines some infrastructure for the query analyzer.

   AnalysisHelper defines some infrastructure for the query analyzer. In particular, in query analysis we don't want to repeatedly re-analyze sub-plans that have previously been analyzed.

   This trait defines a flag analyzed that can be set to true once analysis is done on the tree. This also provides a set of resolve methods that do not recurse down to sub-plans that have the analyzed flag set to true.

   The analyzer rules should use the various resolve methods, in lieu of the various transform methods defined in org.apache.spark.sql.catalyst.trees.TreeNode and QueryPlan.

   To prevent accidental use of the transform methods, this trait also overrides the transform methods to throw exceptions in test mode, if they are used in the analyzer.

8.  trait AnalysisOnlyCommand extends LogicalPlan with Command

   A logical node that can be used for a command that requires its children to be only analyzed, but not optimized.

   A logical node that can be used for a command that requires its children to be only analyzed, but not optimized. An example would be "create view": we don't need to optimize the view subtree because we will just store the entire view text as is in the catalog.

   The way we do this is by setting the children to empty once the subtree is analyzed. This will prevent the optimizer (or the analyzer from that point on) from traversing into the children.

   There's a corresponding rule org.apache.spark.sql.catalyst.analysis.Analyzer.HandleSpecialCommand that marks these commands analyzed.

9.  case class AnalyzeColumn(child: LogicalPlan, columnNames: Option[Seq[String]], allColumns: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

   The logical plan of the ANALYZE TABLE FOR COLUMNS command.

10.  case class AnalyzeTable(child: LogicalPlan, partitionSpec: Map[String, Option[String]], noScan: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the ANALYZE TABLE command.

11.  case class AnalyzeTables(namespace: LogicalPlan, noScan: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the ANALYZE TABLES command.

12.  case class AppendColumns(func: (Any) ⇒ Any, argumentClass: Class[_], argumentSchema: StructType, deserializer: Expression, serializer: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    A relation produced by applying func to each element of the child, concatenating the resulting columns at the end of the input row.

    A relation produced by applying func to each element of the child, concatenating the resulting columns at the end of the input row.

    deserializer

    used to extract the input to func from an input row.

    serializer

    use to serialize the output of func.

13.  case class AppendColumnsWithObject(func: (Any) ⇒ Any, childSerializer: Seq[NamedExpression], newColumnsSerializer: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with ObjectConsumer with Product with Serializable

    An optimized version of AppendColumns, that can be executed on deserialized object directly.

14.  case class AppendData(table: NamedRelation, query: LogicalPlan, writeOptions: Map[String, String], isByName: Boolean, write: Option[Write] = None, analyzedQuery: Option[LogicalPlan] = None) extends LogicalPlan with V2WriteCommand with Product with Serializable

    Append data to an existing table.

15.  case class ArrowEvalPython(udfs: Seq[PythonUDF], resultAttrs: Seq[Attribute], child: LogicalPlan, evalType: Int) extends LogicalPlan with BaseEvalPython with Product with Serializable

    A logical plan that evaluates a PythonUDF with Apache Arrow.

16.  case class ArrowEvalPythonUDTF(udtf: PythonUDTF, requiredChildOutput: Seq[Attribute], resultAttrs: Seq[Attribute], child: LogicalPlan, evalType: Int) extends LogicalPlan with BaseEvalPythonUDTF with Product with Serializable

    A logical plan that evaluates a PythonUDTF using Apache Arrow.

    A logical plan that evaluates a PythonUDTF using Apache Arrow.

    udtf

    the user-defined Python function

    requiredChildOutput

    the required output of the child plan. It's used for omitting data generation that will be discarded next by a projection.

    resultAttrs

    the output schema of the Python UDTF.

    child

    the child plan

17.  case class AsOfJoin(left: LogicalPlan, right: LogicalPlan, asOfCondition: Expression, condition: Option[Expression], joinType: JoinType, orderExpression: Expression, toleranceAssertion: Option[Expression]) extends LogicalPlan with BinaryNode with Product with Serializable

    A logical plan for as-of join.

18.  case class Assignment(key: Expression, value: Expression) extends Expression with Unevaluable with BinaryLike[Expression] with Product with Serializable
19.  case class AttachDistributedSequence(sequenceAttr: Attribute, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    A logical plan that adds a new long column with the name name that increases one by one.

    A logical plan that adds a new long column with the name name that increases one by one. This is for 'distributed-sequence' default index in pandas API on Spark.

20.  trait BaseEvalPython extends LogicalPlan with UnaryNode
21.  trait BaseEvalPythonUDTF extends LogicalPlan with UnaryNode
22.  case class BatchEvalPython(udfs: Seq[PythonUDF], resultAttrs: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with BaseEvalPython with Product with Serializable

    A logical plan that evaluates a PythonUDF

23.  case class BatchEvalPythonUDTF(udtf: PythonUDTF, requiredChildOutput: Seq[Attribute], resultAttrs: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with BaseEvalPythonUDTF with Product with Serializable

    A logical plan that evaluates a PythonUDTF.

    A logical plan that evaluates a PythonUDTF.

    udtf

    the user-defined Python function

    requiredChildOutput

    the required output of the child plan. It's used for omitting data generation that will be discarded next by a projection.

    resultAttrs

    the output schema of the Python UDTF.

    child

    the child plan

24.  trait BinaryCommand extends LogicalPlan with Command with BinaryLike[LogicalPlan]
25.  trait BinaryNode extends LogicalPlan with BinaryLike[LogicalPlan]

    A logical plan node with a left and right child.

26.  case class CTERelationDef(child: LogicalPlan, id: Long = CTERelationDef.newId, originalPlanWithPredicates: Option[(LogicalPlan, Seq[Expression])] = None, underSubquery: Boolean = false) extends LogicalPlan with UnaryNode with Product with Serializable

    A wrapper for CTE definition plan with a unique ID.

    A wrapper for CTE definition plan with a unique ID.

    child

    The CTE definition query plan.

    id

    The unique ID for this CTE definition.

    originalPlanWithPredicates

    The original query plan before predicate pushdown and the predicates that have been pushed down into child. This is a temporary field used by optimization rules for CTE predicate pushdown to help ensure rule idempotency.

    underSubquery

    If true, it means we don't need to add a shuffle for this CTE relation as subquery reuse will be applied to reuse CTE relation output.

27.  case class CTERelationRef(cteId: Long, _resolved: Boolean, output: Seq[Attribute], statsOpt: Option[Statistics] = None) extends LogicalPlan with LeafNode with MultiInstanceRelation with Product with Serializable

    Represents the relation of a CTE reference.

    Represents the relation of a CTE reference.

    cteId

    The ID of the corresponding CTE definition.

    _resolved

    Whether this reference is resolved.

    output

    The output attributes of this CTE reference, which can be different from the output of its corresponding CTE definition after attribute de-duplication.

    statsOpt

    The optional statistics inferred from the corresponding CTE definition.

28.  case class CacheTable(table: LogicalPlan, multipartIdentifier: Seq[String], isLazy: Boolean, options: Map[String, String], isAnalyzed: Boolean = false) extends LogicalPlan with AnalysisOnlyCommand with Product with Serializable

    The logical plan of the CACHE TABLE command.

29.  case class CacheTableAsSelect(tempViewName: String, plan: LogicalPlan, originalText: String, isLazy: Boolean, options: Map[String, String], isAnalyzed: Boolean = false, referredTempFunctions: Seq[String] = Seq.empty) extends LogicalPlan with AnalysisOnlyCommand with Product with Serializable

    The logical plan of the CACHE TABLE ...

    The logical plan of the CACHE TABLE ... AS SELECT command.

30.  case class CoGroup(func: (Any, Iterator[Any], Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, leftDeserializer: Expression, rightDeserializer: Expression, leftGroup: Seq[Attribute], rightGroup: Seq[Attribute], leftAttr: Seq[Attribute], rightAttr: Seq[Attribute], leftOrder: Seq[SortOrder], rightOrder: Seq[SortOrder], outputObjAttr: Attribute, left: LogicalPlan, right: LogicalPlan) extends LogicalPlan with BinaryNode with ObjectProducer with Product with Serializable

    A relation produced by applying func to each grouping key and associated values from left and right children.

31.  case class CollectMetrics(name: String, metrics: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Collect arbitrary (named) metrics from a dataset.

    Collect arbitrary (named) metrics from a dataset. As soon as the query reaches a completion point (batch query completes or streaming query epoch completes) an event is emitted on the driver which can be observed by attaching a listener to the spark session. The metrics are named so we can collect metrics at multiple places in a single dataset.

    This node behaves like a global aggregate. All the metrics collected must be aggregate functions or be literals.

32.  case class ColumnStat(distinctCount: Option[BigInt] = None, min: Option[Any] = None, max: Option[Any] = None, nullCount: Option[BigInt] = None, avgLen: Option[Long] = None, maxLen: Option[Long] = None, histogram: Option[Histogram] = None, version: Int = CatalogColumnStat.VERSION) extends Product with Serializable

    Statistics collected for a column.

    Statistics collected for a column.

    1. The JVM data type stored in min/max is the internal data type for the corresponding Catalyst data type. For example, the internal type of DateType is Int, and that the internal type of TimestampType is Long. 2. There is no guarantee that the statistics collected are accurate. Approximation algorithms (sketches) might have been used, and the data collected can also be stale.

    distinctCount

    number of distinct values

    min

    minimum value

    max

    maximum value

    nullCount

    number of nulls

    avgLen

    average length of the values. For fixed-length types, this should be a constant.

    maxLen

    maximum length of the values. For fixed-length types, this should be a constant.

    histogram

    histogram of the values

    version

    version of statistics saved to or retrieved from the catalog

33.  trait Command extends LogicalPlan

    A logical node that represents a non-query command to be executed by the system.

    A logical node that represents a non-query command to be executed by the system. For example, commands can be used by parsers to represent DDL operations. Commands, unlike queries, are eagerly executed.

34.  case class CommentOnNamespace(child: LogicalPlan, comment: String) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan that defines or changes the comment of an NAMESPACE for v2 catalogs.

    The logical plan that defines or changes the comment of an NAMESPACE for v2 catalogs.

    COMMENT ON (DATABASE|SCHEMA|NAMESPACE) namespaceIdentifier IS ('text' | NULL)

    where the text is the new comment written as a string literal; or NULL to drop the comment.

35.  case class CommentOnTable(table: LogicalPlan, comment: String) extends LogicalPlan with AlterTableCommand with Product with Serializable

    The logical plan that defines or changes the comment of an TABLE for v2 catalogs.

    The logical plan that defines or changes the comment of an TABLE for v2 catalogs.

    COMMENT ON TABLE tableIdentifier IS ('text' | NULL)

    where the text is the new comment written as a string literal; or NULL to drop the comment.

36.  trait ConstraintHelper extends AnyRef
37.  case class CreateFunction(child: LogicalPlan, className: String, resources: Seq[FunctionResource], ifExists: Boolean, replace: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the CREATE FUNCTION command.

38.  case class CreateIndex(table: LogicalPlan, indexName: String, indexType: String, ignoreIfExists: Boolean, columns: Seq[(FieldName, Map[String, String])], properties: Map[String, String]) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the CREATE INDEX command.

39.  case class CreateNamespace(name: LogicalPlan, ifNotExists: Boolean, properties: Map[String, String]) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the CREATE NAMESPACE command.

40.  case class CreateTable(name: LogicalPlan, tableSchema: StructType, partitioning: Seq[Transform], tableSpec: TableSpecBase, ignoreIfExists: Boolean) extends LogicalPlan with UnaryCommand with V2CreateTablePlan with Product with Serializable

    Create a new table with a v2 catalog.

41.  case class CreateTableAsSelect(name: LogicalPlan, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpecBase, writeOptions: Map[String, String], ignoreIfExists: Boolean, isAnalyzed: Boolean = false) extends LogicalPlan with V2CreateTableAsSelectPlan with Product with Serializable

    Create a new table from a select query with a v2 catalog.

42.  case class CreateView(child: LogicalPlan, userSpecifiedColumns: Seq[(String, Option[String])], comment: Option[String], properties: Map[String, String], originalText: Option[String], query: LogicalPlan, allowExisting: Boolean, replace: Boolean) extends LogicalPlan with BinaryCommand with Product with Serializable

    The logical plan of the CREATE VIEW ...

    The logical plan of the CREATE VIEW ... command.

43.  case class Deduplicate(keys: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    A logical plan for dropDuplicates.

44.  case class DeduplicateWithinWatermark(keys: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
45.  case class DeleteAction(condition: Option[Expression]) extends MergeAction with Product with Serializable
46.  case class DeleteFromTable(table: LogicalPlan, condition: Expression) extends LogicalPlan with UnaryCommand with SupportsSubquery with Product with Serializable

    The logical plan of the DELETE FROM command.

47.  case class DeleteFromTableWithFilters(table: LogicalPlan, condition: Seq[Predicate]) extends LogicalPlan with LeafCommand with Product with Serializable

    The logical plan of the DELETE FROM command that can be executed using data source filters.

    The logical plan of the DELETE FROM command that can be executed using data source filters.

    As opposed to DeleteFromTable, this node represents a DELETE operation where the condition was converted into filters and the data source reported that it can handle all of them.

48.  case class DescribeColumn(relation: LogicalPlan, column: Expression, isExtended: Boolean, output: Seq[Attribute] = DescribeColumn.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DESCRIBE relation_name col_name command.

49.  case class DescribeFunction(child: LogicalPlan, isExtended: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DESCRIBE FUNCTION command.

50.  case class DescribeNamespace(namespace: LogicalPlan, extended: Boolean, output: Seq[Attribute] = DescribeNamespace.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DESCRIBE NAMESPACE command.

51.  case class DescribeRelation(relation: LogicalPlan, partitionSpec: TablePartitionSpec, isExtended: Boolean, output: Seq[Attribute] = DescribeRelation.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DESCRIBE relation_name command.

52.  case class DeserializeToObject(deserializer: Expression, outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with UnaryNode with ObjectProducer with Product with Serializable

    Takes the input row from child and turns it into object using the given deserializer expression.

53.  case class Distinct(child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Returns a new logical plan that dedups input rows.

54.  case class DomainJoin(domainAttrs: Seq[Attribute], child: LogicalPlan, joinType: JoinType = Inner, condition: Option[Expression] = None) extends LogicalPlan with UnaryNode with Product with Serializable

    A placeholder for domain join that can be added when decorrelating subqueries.

    A placeholder for domain join that can be added when decorrelating subqueries. It should be rewritten during the optimization phase.

55.  case class DropColumns(table: LogicalPlan, columnsToDrop: Seq[FieldName], ifExists: Boolean) extends LogicalPlan with AlterTableCommand with Product with Serializable

    The logical plan of the ALTER TABLE ...

    The logical plan of the ALTER TABLE ... DROP COLUMNS command.

56.  case class DropFunction(child: LogicalPlan, ifExists: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DROP FUNCTION command.

57.  case class DropIndex(table: LogicalPlan, indexName: String, ignoreIfNotExists: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DROP INDEX command.

58.  case class DropNamespace(namespace: LogicalPlan, ifExists: Boolean, cascade: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DROP NAMESPACE command.

59.  case class DropPartitions(table: LogicalPlan, parts: Seq[PartitionSpec], ifExists: Boolean, purge: Boolean) extends LogicalPlan with V2PartitionCommand with Product with Serializable

    The logical plan of the ALTER TABLE DROP PARTITION command.

    The logical plan of the ALTER TABLE DROP PARTITION command. This may remove the data and metadata for this partition.

    If the PURGE option is set, the table catalog must remove partition data by skipping the trash even when the catalog has configured one. The option is applicable only for managed tables.

    The syntax of this command is:

    ALTER TABLE table DROP [IF EXISTS] PARTITION spec1[, PARTITION spec2, ...] [PURGE];

60.  case class DropTable(child: LogicalPlan, ifExists: Boolean, purge: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DROP TABLE command.

    The logical plan of the DROP TABLE command.

    If the PURGE option is set, the table catalog must remove table data by skipping the trash even when the catalog has configured one. The option is applicable only for managed tables.

    The syntax of this command is:

    DROP TABLE [IF EXISTS] table [PURGE];

61.  case class DropView(child: LogicalPlan, ifExists: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the DROP VIEW command.

62.  case class EventTimeWatermark(eventTime: Attribute, delay: CalendarInterval, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Used to mark a user specified column as holding the event time for a row.

63.  case class Except(left: LogicalPlan, right: LogicalPlan, isAll: Boolean) extends SetOperation with Product with Serializable
64.  case class Expand(projections: Seq[Seq[Expression]], output: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Apply a number of projections to every input row, hence we will get multiple output rows for an input row.

    Apply a number of projections to every input row, hence we will get multiple output rows for an input row.

    projections

    to apply

    output

    of all projections.

    child

    operator.

65.  trait ExposesMetadataColumns extends LogicalPlan

    A logical plan node that can generate metadata columns

66.  case class Filter(condition: Expression, child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with PredicateHelper with Product with Serializable
67.  case class FlatMapCoGroupsInPandas(leftGroupingLen: Int, rightGroupingLen: Int, functionExpr: Expression, output: Seq[Attribute], left: LogicalPlan, right: LogicalPlan) extends LogicalPlan with BinaryNode with Product with Serializable

    Flatmap cogroups using a udf: pandas.Dataframe, pandas.Dataframe -> pandas.Dataframe This is used by DataFrame.groupby().cogroup().apply().

68.  case class FlatMapGroupsInPandas(groupingAttributes: Seq[Attribute], functionExpr: Expression, output: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    FlatMap groups using a udf: pandas.Dataframe -> pandas.DataFrame.

    FlatMap groups using a udf: pandas.Dataframe -> pandas.DataFrame. This is used by DataFrame.groupby().apply().

69.  case class FlatMapGroupsInPandasWithState(functionExpr: Expression, groupingAttributes: Seq[Attribute], outputAttrs: Seq[Attribute], stateType: StructType, outputMode: OutputMode, timeout: GroupStateTimeout, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Similar with FlatMapGroupsWithState.

    Similar with FlatMapGroupsWithState. Applies func to each unique group in child, based on the evaluation of groupingAttributes, while using state data. functionExpr is invoked with an pandas DataFrame representation and the grouping key (tuple).

    functionExpr

    function called on each group

    groupingAttributes

    used to group the data

    outputAttrs

    used to define the output rows

    stateType

    used to serialize/deserialize state before calling functionExpr

    outputMode

    the output mode of func

    timeout

    used to timeout groups that have not received data in a while

    child

    logical plan of the underlying data

70.  case class FlatMapGroupsInR(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, outputSchema: StructType, keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with UnaryNode with ObjectProducer with Product with Serializable
71.  case class FlatMapGroupsInRWithArrow(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, output: Seq[Attribute], keyDeserializer: Expression, groupingAttributes: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Similar with FlatMapGroupsInR but serializes and deserializes input/output in Arrow format.

    Similar with FlatMapGroupsInR but serializes and deserializes input/output in Arrow format. This is also somewhat similar with FlatMapGroupsInPandas.

72.  case class FlatMapGroupsWithState(func: (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, stateEncoder: ExpressionEncoder[Any], outputMode: OutputMode, isMapGroupsWithState: Boolean = false, timeout: GroupStateTimeout, hasInitialState: Boolean = false, initialStateGroupAttrs: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], initialStateDeserializer: Expression, initialState: LogicalPlan, child: LogicalPlan) extends LogicalPlan with BinaryNode with ObjectProducer with Product with Serializable

    Applies func to each unique group in child, based on the evaluation of groupingAttributes, while using state data.

    Applies func to each unique group in child, based on the evaluation of groupingAttributes, while using state data. Func is invoked with an object representation of the grouping key an iterator containing the object representation of all the rows with that key.

    func

    function called on each group

    keyDeserializer

    used to extract the key object for each group.

    valueDeserializer

    used to extract the items in the iterator from an input row.

    groupingAttributes

    used to group the data

    dataAttributes

    used to read the data

    outputObjAttr

    used to define the output object

    stateEncoder

    used to serialize/deserialize state before calling func

    outputMode

    the output mode of func

    isMapGroupsWithState

    whether it is created by the mapGroupsWithState method

    timeout

    used to timeout groups that have not received data in a while

    hasInitialState

    Indicates whether initial state needs to be applied or not.

    initialStateGroupAttrs

    grouping attributes for the initial state

    initialStateDataAttrs

    used to read the initial state

    initialStateDeserializer

    used to extract the initial state objects.

    initialState

    user defined initial state that is applied in the first batch.

    child

    logical plan of the underlying data

73.  case class FormatClasses(input: String, output: String) extends Product with Serializable
74.  trait FunctionBuilderBase[T] extends AnyRef

    This is a base trait that is used for implementing builder classes that can be used to construct expressions or logical plans depending on if it is a table-valued or scalar-valued function.

    This is a base trait that is used for implementing builder classes that can be used to construct expressions or logical plans depending on if it is a table-valued or scalar-valued function.

    Two classes of builders currently exist for this trait: GeneratorBuilder and ExpressionBuilder. If a new class of functions are to be added, a new trait should also be created which extends this trait.

    T

    The type that is expected to be returned by the FunctionBuilderBase.build function

75.  case class FunctionSignature(parameters: Seq[InputParameter]) extends Product with Serializable

    Represents a method signature and the list of arguments it receives as input.

    Represents a method signature and the list of arguments it receives as input. Currently, overloads are not supported and only one FunctionSignature is allowed per function expression.

    parameters

    The list of arguments which the function takes

76.  case class Generate(generator: Generator, unrequiredChildIndex: Seq[Int], outer: Boolean, qualifier: Option[String], generatorOutput: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows.

    Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows. This operation is similar to a flatMap in functional programming with one important additional feature, which allows the input rows to be joined with their output.

    generator

    the generator expression

    unrequiredChildIndex

    this parameter starts as Nil and gets filled by the Optimizer. It's used as an optimization for omitting data generation that will be discarded next by a projection. A common use case is when we explode(array(..)) and are interested only in the exploded data and not in the original array. before this optimization the array got duplicated for each of its elements, causing O(n^^2) memory consumption. (see [SPARK-21657])

    outer

    when true, each input row will be output at least once, even if the output of the given generator is empty.

    qualifier

    Qualifier for the attributes of generator(UDTF)

    generatorOutput

    The output schema of the Generator.

    child

    Children logical plan node

77.  case class GlobalLimit(limitExpr: Expression, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

    A global (coordinated) limit.

    A global (coordinated) limit. This operator can emit at most limitExpr number in total.

    See Limit for more information.

    Note that, we can not make it inherit OrderPreservingUnaryNode due to the different strategy of physical plan. The output ordering of child will be broken if a shuffle exchange comes in between the child and global limit, due to the fact that shuffle reader fetches blocks in random order.

78.  trait HasPartitionExpressions extends SQLConfHelper
79.  trait HintErrorHandler extends AnyRef

    The callback for implementing customized strategies of handling hint errors.

80.  case class HintInfo(strategy: Option[JoinStrategyHint] = None) extends Product with Serializable

    The hint attributes to be applied on a specific node.

    The hint attributes to be applied on a specific node.

    strategy

    The preferred join strategy.

81.  case class Histogram(height: Double, bins: Array[HistogramBin]) extends Product with Serializable

    This class is an implementation of equi-height histogram.

    This class is an implementation of equi-height histogram. Equi-height histogram represents the distribution of a column's values by a sequence of bins. Each bin has a value range and contains approximately the same number of rows.

    height

    number of rows in each bin

    bins

    equi-height histogram bins

82.  case class HistogramBin(lo: Double, hi: Double, ndv: Long) extends Product with Serializable

    A bin in an equi-height histogram.

    A bin in an equi-height histogram. We use double type for lower/higher bound for simplicity.

    lo

    lower bound of the value range in this bin

    hi

    higher bound of the value range in this bin

    ndv

    approximate number of distinct values in this bin

83.  trait IgnoreCachedData extends LogicalPlan

    A LogicalPlan operator that does not use the cached results stored in CacheManager

84.  case class InputParameter(name: String, default: Option[Expression] = None) extends Product with Serializable

    Represents a parameter of a function expression.

    Represents a parameter of a function expression. Function expressions should use this class to construct the argument lists returned in Builder

    name

    The name of the string.

    default

    The default value of the argument. If the default is none, then that means the argument is required. If no argument is provided, an exception is thrown.

85.  case class InsertAction(condition: Option[Expression], assignments: Seq[Assignment]) extends MergeAction with Product with Serializable
86.  case class InsertIntoDir(isLocal: Boolean, storage: CatalogStorageFormat, provider: Option[String], child: LogicalPlan, overwrite: Boolean = true) extends LogicalPlan with UnaryNode with Product with Serializable

    Insert query result into a directory.

    Insert query result into a directory.

    isLocal

    Indicates whether the specified directory is local directory

    storage

    Info about output file, row and what serialization format

    provider

    Specifies what data source to use; only used for data source file.

    child

    The query to be executed

    overwrite

    If true, the existing directory will be overwritten Note that this plan is unresolved and has to be replaced by the concrete implementations during analysis.

87.  case class InsertIntoStatement(table: LogicalPlan, partitionSpec: Map[String, Option[String]], userSpecifiedCols: Seq[String], query: LogicalPlan, overwrite: Boolean, ifPartitionNotExists: Boolean, byName: Boolean = false) extends ParsedStatement with UnaryParsedStatement with Product with Serializable

    An INSERT INTO statement, as parsed from SQL.

    An INSERT INTO statement, as parsed from SQL.

    table

    the logical plan representing the table.

    partitionSpec

    a map from the partition key to the partition value (optional). If the value is missing, dynamic partition insert will be performed. As an example, INSERT INTO tbl PARTITION (a=1, b=2) AS would have Map('a' -> Some('1'), 'b' -> Some('2')), and INSERT INTO tbl PARTITION (a=1, b) AS ... would have Map('a' -> Some('1'), 'b' -> None).

    userSpecifiedCols

    the user specified list of columns that belong to the table.

    query

    the logical plan representing data to write to.

    overwrite

    overwrite existing table or partitions.

    ifPartitionNotExists

    If true, only write if the partition does not exist. Only valid for static partitions.

    byName

    If true, reorder the data columns to match the column names of the target table.

88.  case class InsertStarAction(condition: Option[Expression]) extends MergeAction with Product with Serializable
89.  case class Intersect(left: LogicalPlan, right: LogicalPlan, isAll: Boolean) extends SetOperation with Product with Serializable
90.  case class Join(left: LogicalPlan, right: LogicalPlan, joinType: JoinType, condition: Option[Expression], hint: JoinHint) extends LogicalPlan with BinaryNode with PredicateHelper with Product with Serializable
91.  case class JoinHint(leftHint: Option[HintInfo], rightHint: Option[HintInfo]) extends Product with Serializable

    Hint that is associated with a Join node, with HintInfo on its left child and on its right child respectively.

92.  sealed abstract class JoinStrategyHint extends AnyRef
93.  trait KeepAnalyzedQuery extends LogicalPlan with Command
94.  case class LateralJoin(left: LogicalPlan, right: LateralSubquery, joinType: JoinType, condition: Option[Expression]) extends LogicalPlan with UnaryNode with Product with Serializable

    A logical plan for lateral join.

95.  trait LeafCommand extends LogicalPlan with Command with LeafLike[LogicalPlan]
96.  trait LeafNode extends LogicalPlan with LeafLike[LogicalPlan]

    A logical plan node with no children.

97.  trait LeafParsedStatement extends ParsedStatement with LeafLike[LogicalPlan]
98.  case class LoadData(child: LogicalPlan, path: String, isLocal: Boolean, isOverwrite: Boolean, partition: Option[TablePartitionSpec]) extends LogicalPlan with UnaryCommand with Product with Serializable

    The logical plan of the LOAD DATA INTO TABLE command.

99.  case class LocalLimit(limitExpr: Expression, child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable

    A partition-local (non-coordinated) limit.

    A partition-local (non-coordinated) limit. This operator can emit at most limitExpr number of tuples on each physical partition.

    See Limit for more information.

100.  case class LocalRelation(output: Seq[Attribute], data: Seq[InternalRow] = Nil, isStreaming: Boolean = false) extends LogicalPlan with LeafNode with MultiInstanceRelation with Product with Serializable

     Logical plan node for scanning data from a local collection.

     Logical plan node for scanning data from a local collection.

     data

     The local collection holding the data. It doesn't need to be sent to executors and then doesn't need to be serializable.

101.  abstract class LogicalPlan extends QueryPlan[LogicalPlan] with AnalysisHelper with LogicalPlanStats with LogicalPlanDistinctKeys with QueryPlanConstraints with Logging
102.  trait LogicalPlanDistinctKeys extends AnyRef

     A trait to add distinct attributes to LogicalPlan.

     A trait to add distinct attributes to LogicalPlan. For example:

     SELECT a, b, SUM(c) FROM Tab1 GROUP BY a, b
     // returns a, b

103.  trait LogicalPlanVisitor[T] extends AnyRef

     A visitor pattern for traversing a LogicalPlan tree and computing some properties.

104.  case class MapElements(func: AnyRef, argumentClass: Class[_], argumentSchema: StructType, outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with ObjectConsumer with ObjectProducer with Product with Serializable

     A relation produced by applying func to each element of the child.

105.  case class MapGroups(func: (Any, Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], dataOrder: Seq[SortOrder], outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with UnaryNode with ObjectProducer with Product with Serializable

     Applies func to each unique group in child, based on the evaluation of groupingAttributes.

     Applies func to each unique group in child, based on the evaluation of groupingAttributes. Func is invoked with an object representation of the grouping key an iterator containing the object representation of all the rows with that key. Given an additional dataOrder, data in the iterator will be sorted accordingly. That sorting does not add computational complexity.

     keyDeserializer

     used to extract the key object for each group.

     valueDeserializer

     used to extract the items in the iterator from an input row.

106.  case class MapInPandas(functionExpr: Expression, output: Seq[Attribute], child: LogicalPlan, isBarrier: Boolean) extends LogicalPlan with UnaryNode with Product with Serializable

     Map partitions using a udf: iter(pandas.Dataframe) -> iter(pandas.DataFrame).

     Map partitions using a udf: iter(pandas.Dataframe) -> iter(pandas.DataFrame). This is used by DataFrame.mapInPandas()

107.  case class MapPartitions(func: (Iterator[Any]) ⇒ Iterator[Any], outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with ObjectConsumer with ObjectProducer with Product with Serializable

     A relation produced by applying func to each partition of the child.

108.  case class MapPartitionsInR(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, outputSchema: StructType, outputObjAttr: Attribute, child: LogicalPlan) extends LogicalPlan with ObjectConsumer with ObjectProducer with Product with Serializable

     A relation produced by applying a serialized R function func to each partition of the child.

109.  case class MapPartitionsInRWithArrow(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, output: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     Similar with MapPartitionsInR but serializes and deserializes input/output in Arrow format.

     Similar with MapPartitionsInR but serializes and deserializes input/output in Arrow format.

     This is somewhat similar with org.apache.spark.sql.execution.python.ArrowEvalPython

110.  sealed abstract class MergeAction extends Expression with Unevaluable
111.  case class MergeIntoTable(targetTable: LogicalPlan, sourceTable: LogicalPlan, mergeCondition: Expression, matchedActions: Seq[MergeAction], notMatchedActions: Seq[MergeAction], notMatchedBySourceActions: Seq[MergeAction]) extends LogicalPlan with BinaryCommand with SupportsSubquery with Product with Serializable

     The logical plan of the MERGE INTO command.

112.  case class MergeRows(isSourceRowPresent: Expression, isTargetRowPresent: Expression, matchedInstructions: Seq[Instruction], notMatchedInstructions: Seq[Instruction], notMatchedBySourceInstructions: Seq[Instruction], checkCardinality: Boolean, output: Seq[Attribute], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
113.  case class NoopCommand(commandName: String, multipartIdentifier: Seq[String]) extends LogicalPlan with LeafCommand with Product with Serializable

     The logical plan for no-op command handling non-existing table.

114.  trait ObjectConsumer extends LogicalPlan with UnaryNode with ReferenceAllColumns[LogicalPlan]

     A trait for logical operators that consumes domain objects as input.

     A trait for logical operators that consumes domain objects as input. The output of its child must be a single-field row containing the input object.

115.  trait ObjectProducer extends LogicalPlan

     A trait for logical operators that produces domain objects as output.

     A trait for logical operators that produces domain objects as output. The output of this operator is a single-field safe row containing the produced object.

116.  case class Offset(offsetExpr: Expression, child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable

     A logical offset, which may removing a specified number of rows from the beginning of the output of child logical plan.

117.  case class OneRowRelation() extends LogicalPlan with LeafNode with Product with Serializable

     A relation with one row.

     A relation with one row. This is used in "SELECT ..." without a from clause.

118.  case class OptionList(options: Seq[(String, Expression)]) extends Expression with Unevaluable with Product with Serializable

     This contains the expressions in an OPTIONS list.

     This contains the expressions in an OPTIONS list. We store it alongside anywhere the above UnresolvedTableSpec lives. We use a separate object so that tree traversals in analyzer rules can descend into the child expressions naturally without extra treatment.

119.  trait OrderPreservingUnaryNode extends LogicalPlan with UnaryNode with AliasAwareQueryOutputOrdering[LogicalPlan]
120.  case class OverwriteByExpression(table: NamedRelation, deleteExpr: Expression, query: LogicalPlan, writeOptions: Map[String, String], isByName: Boolean, write: Option[Write] = None, analyzedQuery: Option[LogicalPlan] = None) extends LogicalPlan with V2WriteCommand with Product with Serializable

     Overwrite data matching a filter in an existing table.

121.  case class OverwritePartitionsDynamic(table: NamedRelation, query: LogicalPlan, writeOptions: Map[String, String], isByName: Boolean, write: Option[Write] = None) extends LogicalPlan with V2WriteCommand with Product with Serializable

     Dynamically overwrite partitions in an existing table.

122.  abstract class ParsedStatement extends LogicalPlan

     A logical plan node that contains exactly what was parsed from SQL.

     A logical plan node that contains exactly what was parsed from SQL.

     This is used to hold information parsed from SQL when there are multiple implementations of a query or command. For example, CREATE TABLE may be implemented by different nodes for v1 and v2. Instead of parsing directly to a v1 CreateTable that keeps metadata in CatalogTable, and then converting that v1 metadata to the v2 equivalent, the sql CreateTableStatement plan is produced by the parser and converted once into both implementations.

     Parsed logical plans are not resolved because they must be converted to concrete logical plans.

     Parsed logical plans are located in Catalyst so that as much SQL parsing logic as possible is be kept in a org.apache.spark.sql.catalyst.parser.AbstractSqlParser.

123.  case class Pivot(groupByExprsOpt: Option[Seq[NamedExpression]], pivotColumn: Expression, pivotValues: Seq[Expression], aggregates: Seq[Expression], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     A constructor for creating a pivot, which will later be converted to a Project or an Aggregate during the query analysis.

     A constructor for creating a pivot, which will later be converted to a Project or an Aggregate during the query analysis.

     groupByExprsOpt

     A sequence of group by expressions. This field should be None if coming from SQL, in which group by expressions are not explicitly specified.

     pivotColumn

     The pivot column.

     pivotValues

     A sequence of values for the pivot column.

     aggregates

     The aggregation expressions, each with or without an alias.

     child

     Child operator

124.  case class Project(projectList: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable
125.  case class PythonMapInArrow(functionExpr: Expression, output: Seq[Attribute], child: LogicalPlan, isBarrier: Boolean) extends LogicalPlan with UnaryNode with Product with Serializable

     Map partitions using a udf: iter(pyarrow.RecordBatch) -> iter(pyarrow.RecordBatch).

     Map partitions using a udf: iter(pyarrow.RecordBatch) -> iter(pyarrow.RecordBatch). This is used by DataFrame.mapInArrow() in PySpark

126.  case class QualifiedColType(path: Option[FieldName], colName: String, dataType: DataType, nullable: Boolean, comment: Option[String], position: Option[FieldPosition], default: Option[String]) extends Product with Serializable

     Column data as parsed by ALTER TABLE ...

     Column data as parsed by ALTER TABLE ... (ADD|REPLACE) COLUMNS.

127.  trait QueryPlanConstraints extends ConstraintHelper
128.  case class Range(start: Long, end: Long, step: Long, numSlices: Option[Int], output: Seq[Attribute] = Range.getOutputAttrs, isStreaming: Boolean = false) extends LogicalPlan with LeafNode with MultiInstanceRelation with Product with Serializable

     Annotations

     @ExpressionDescription()

129.  case class RebalancePartitions(partitionExpressions: Seq[Expression], child: LogicalPlan, optNumPartitions: Option[Int] = None, optAdvisoryPartitionSize: Option[Long] = None) extends LogicalPlan with UnaryNode with HasPartitionExpressions with Product with Serializable

     This operator is used to rebalance the output partitions of the given child, so that every partition is of a reasonable size (not too small and not too big).

     This operator is used to rebalance the output partitions of the given child, so that every partition is of a reasonable size (not too small and not too big). It also try its best to partition the child output by partitionExpressions. If there are skews, Spark will split the skewed partitions, to make these partitions not too big. This operator is useful when you need to write the result of child to a table, to avoid too small/big files.

     Note that, this operator only makes sense when AQE is enabled.

130.  case class RecoverPartitions(child: LogicalPlan) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... RECOVER PARTITIONS command.

131.  case class RefreshFunction(child: LogicalPlan) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the REFRESH FUNCTION command.

132.  case class RefreshTable(child: LogicalPlan) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the REFRESH TABLE command.

133.  case class RenameColumn(table: LogicalPlan, column: FieldName, newName: String) extends LogicalPlan with AlterTableCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... RENAME COLUMN command.

134.  case class RenamePartitions(table: LogicalPlan, from: PartitionSpec, to: PartitionSpec) extends LogicalPlan with V2PartitionCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... RENAME TO PARTITION command.

135.  case class RenameTable(child: LogicalPlan, newName: Seq[String], isView: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER [TABLE|VIEW] ...

     The logical plan of the ALTER [TABLE|VIEW] ... RENAME TO command.

136.  case class RepairTable(child: LogicalPlan, enableAddPartitions: Boolean, enableDropPartitions: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the MSCK REPAIR TABLE command.

137.  case class Repartition(numPartitions: Int, shuffle: Boolean, child: LogicalPlan) extends RepartitionOperation with Product with Serializable

     Returns a new RDD that has exactly numPartitions partitions.

     Returns a new RDD that has exactly numPartitions partitions. Differs from RepartitionByExpression as this method is called directly by DataFrame's, because the user asked for coalesce or repartition. RepartitionByExpression is used when the consumer of the output requires some specific ordering or distribution of the data.

138.  case class RepartitionByExpression(partitionExpressions: Seq[Expression], child: LogicalPlan, optNumPartitions: Option[Int], optAdvisoryPartitionSize: Option[Long] = None) extends RepartitionOperation with HasPartitionExpressions with Product with Serializable

     This method repartitions data using Expressions into optNumPartitions, and receives information about the number of partitions during execution.

     This method repartitions data using Expressions into optNumPartitions, and receives information about the number of partitions during execution. Used when a specific ordering or distribution is expected by the consumer of the query result. Use Repartition for RDD-like coalesce and repartition. If no optNumPartitions is given, by default it partitions data into numShufflePartitions defined in SQLConf, and could be coalesced by AQE.

139.  abstract class RepartitionOperation extends LogicalPlan with UnaryNode

     A base interface for RepartitionByExpression and Repartition

140.  case class ReplaceColumns(table: LogicalPlan, columnsToAdd: Seq[QualifiedColType]) extends LogicalPlan with AlterTableCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... REPLACE COLUMNS command.

141.  case class ReplaceData(table: NamedRelation, condition: Expression, query: LogicalPlan, originalTable: NamedRelation, groupFilterCondition: Option[Expression] = None, write: Option[Write] = None) extends LogicalPlan with RowLevelWrite with Product with Serializable

     Replace groups of data in an existing table during a row-level operation.

     Replace groups of data in an existing table during a row-level operation.

     This node is constructed in rules that rewrite DELETE, UPDATE, MERGE operations for data sources that can replace groups of data (e.g. files, partitions).

     table

     a plan that references a row-level operation table

     condition

     a condition that defines matching groups

     query

     a query with records that should replace the records that were read

     originalTable

     a plan for the original table for which the row-level command was triggered

     groupFilterCondition

     a condition that can be used to filter groups at runtime

     write

     a logical write, if already constructed

142.  case class ReplaceTable(name: LogicalPlan, tableSchema: StructType, partitioning: Seq[Transform], tableSpec: TableSpecBase, orCreate: Boolean) extends LogicalPlan with UnaryCommand with V2CreateTablePlan with Product with Serializable

     Replace a table with a v2 catalog.

     Replace a table with a v2 catalog.

     If the table does not exist, and orCreate is true, then it will be created. If the table does not exist, and orCreate is false, then an exception will be thrown.

     The persisted table will have no contents as a result of this operation.

143.  case class ReplaceTableAsSelect(name: LogicalPlan, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpecBase, writeOptions: Map[String, String], orCreate: Boolean, isAnalyzed: Boolean = false) extends LogicalPlan with V2CreateTableAsSelectPlan with Product with Serializable

     Replaces a table from a select query with a v2 catalog.

     Replaces a table from a select query with a v2 catalog.

     If the table does not exist, and orCreate is true, then it will be created. If the table does not exist, and orCreate is false, then an exception will be thrown.

144.  case class ResolvedHint(child: LogicalPlan, hints: HintInfo = HintInfo()) extends LogicalPlan with UnaryNode with Product with Serializable

     A resolved hint node.

     A resolved hint node. The analyzer should convert all UnresolvedHint into ResolvedHint. This node will be eliminated before optimization starts.

145.  case class ReturnAnswer(child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     When planning take() or collect() operations, this special node is inserted at the top of the logical plan before invoking the query planner.

     When planning take() or collect() operations, this special node is inserted at the top of the logical plan before invoking the query planner.

     Rules can pattern-match on this node in order to apply transformations that only take effect at the top of the logical query plan.

146.  trait RowLevelWrite extends LogicalPlan with V2WriteCommand with SupportsSubquery
147.  case class Sample(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     Sample the dataset.

     Sample the dataset.

     lowerBound

     Lower-bound of the sampling probability (usually 0.0)

     upperBound

     Upper-bound of the sampling probability. The expected fraction sampled will be ub - lb.

     withReplacement

     Whether to sample with replacement.

     seed

     the random seed

     child

     the LogicalPlan

148.  case class ScriptInputOutputSchema(inputRowFormat: Seq[(String, String)], outputRowFormat: Seq[(String, String)], inputSerdeClass: Option[String], outputSerdeClass: Option[String], inputSerdeProps: Seq[(String, String)], outputSerdeProps: Seq[(String, String)], recordReaderClass: Option[String], recordWriterClass: Option[String], schemaLess: Boolean) extends Product with Serializable

     Input and output properties when passing data to a script.

     Input and output properties when passing data to a script. For example, in Hive this would specify which SerDes to use.

149.  case class ScriptTransformation(script: String, output: Seq[Attribute], child: LogicalPlan, ioschema: ScriptInputOutputSchema) extends LogicalPlan with UnaryNode with ReferenceAllColumns[LogicalPlan] with Product with Serializable

     Transforms the input by forking and running the specified script.

     Transforms the input by forking and running the specified script.

     script

     the command that should be executed.

     output

     the attributes that are produced by the script.

     ioschema

     the input and output schema applied in the execution of the script.

150.  case class SerdeInfo(storedAs: Option[String] = None, formatClasses: Option[FormatClasses] = None, serde: Option[String] = None, serdeProperties: Map[String, String] = Map.empty) extends Product with Serializable

     Type to keep track of Hive serde info

151.  case class SerializeFromObject(serializer: Seq[NamedExpression], child: LogicalPlan) extends LogicalPlan with ObjectConsumer with Product with Serializable

     Takes the input object from child and turns it into unsafe row using the given serializer expression.

152.  case class SetCatalogAndNamespace(child: LogicalPlan) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the USE command.

153.  case class SetNamespaceLocation(namespace: LogicalPlan, location: String) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER (DATABASE|SCHEMA|NAMESPACE) ...

     The logical plan of the ALTER (DATABASE|SCHEMA|NAMESPACE) ... SET LOCATION command.

154.  case class SetNamespaceProperties(namespace: LogicalPlan, properties: Map[String, String]) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER (DATABASE|SCHEMA|NAMESPACE) ...

     The logical plan of the ALTER (DATABASE|SCHEMA|NAMESPACE) ... SET (DBPROPERTIES|PROPERTIES) command.

155.  abstract class SetOperation extends LogicalPlan with BinaryNode
156.  case class SetTableLocation(table: LogicalPlan, partitionSpec: Option[TablePartitionSpec], location: String) extends LogicalPlan with AlterTableCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... SET LOCATION command.

157.  case class SetTableProperties(table: LogicalPlan, properties: Map[String, String]) extends LogicalPlan with AlterTableCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... SET TBLPROPERTIES command.

158.  case class SetTableSerDeProperties(child: LogicalPlan, serdeClassName: Option[String], serdeProperties: Option[Map[String, String]], partitionSpec: Option[TablePartitionSpec]) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... SET [SERDE|SERDEPROPERTIES] command.

159.  case class SetViewProperties(child: LogicalPlan, properties: Map[String, String]) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER VIEW ...

     The logical plan of the ALTER VIEW ... SET TBLPROPERTIES command.

160.  case class ShowColumns(child: LogicalPlan, namespace: Option[Seq[String]], output: Seq[Attribute] = ShowColumns.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW COLUMN command.

161.  case class ShowCreateTable(child: LogicalPlan, asSerde: Boolean = false, output: Seq[Attribute] = ShowCreateTable.getoutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW CREATE TABLE command.

162.  case class ShowFunctions(namespace: LogicalPlan, userScope: Boolean, systemScope: Boolean, pattern: Option[String], output: Seq[Attribute] = ShowFunctions.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW FUNCTIONS command.

163.  case class ShowNamespaces(namespace: LogicalPlan, pattern: Option[String], output: Seq[Attribute] = ShowNamespaces.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW NAMESPACES command.

164.  case class ShowPartitions(table: LogicalPlan, pattern: Option[PartitionSpec], output: Seq[Attribute] = ShowPartitions.getOutputAttrs) extends LogicalPlan with V2PartitionCommand with Product with Serializable

     The logical plan of the SHOW PARTITIONS command.

165.  case class ShowTableExtended(namespace: LogicalPlan, pattern: String, partitionSpec: Option[PartitionSpec], output: Seq[Attribute] = ShowTableExtended.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW TABLE EXTENDED command.

166.  case class ShowTableProperties(table: LogicalPlan, propertyKey: Option[String], output: Seq[Attribute] = ShowTableProperties.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW TBLPROPERTIES command.

167.  case class ShowTables(namespace: LogicalPlan, pattern: Option[String], output: Seq[Attribute] = ShowTables.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW TABLES command.

168.  case class ShowViews(namespace: LogicalPlan, pattern: Option[String], output: Seq[Attribute] = ShowViews.getOutputAttrs) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the SHOW VIEWS command.

     The logical plan of the SHOW VIEWS command.

     Notes: v2 catalogs do not support views API yet, the command will fallback to v1 ShowViewsCommand during ResolveSessionCatalog.

169.  case class Sort(order: Seq[SortOrder], global: Boolean, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
     order

     The ordering expressions

     global

     True means global sorting apply for entire data set, False means sorting only apply within the partition.

     child

     Child logical plan

170.  case class Statistics(sizeInBytes: BigInt, rowCount: Option[BigInt] = None, attributeStats: AttributeMap[ColumnStat] = AttributeMap(Nil), isRuntime: Boolean = false) extends Product with Serializable

     Estimates of various statistics.

     Estimates of various statistics. The default estimation logic simply lazily multiplies the corresponding statistic produced by the children. To override this behavior, override statistics and assign it an overridden version of Statistics.

     NOTE: concrete and/or overridden versions of statistics fields should pay attention to the performance of the implementations. The reason is that estimations might get triggered in performance-critical processes, such as query plan planning.

     Note that we are using a BigInt here since it is easy to overflow a 64-bit integer in cardinality estimation (e.g. cartesian joins).

     sizeInBytes

     Physical size in bytes. For leaf operators this defaults to 1, otherwise it defaults to the product of children's sizeInBytes.

     rowCount

     Estimated number of rows.

     attributeStats

     Statistics for Attributes.

     isRuntime

     Whether the statistics is inferred from query stage runtime statistics during adaptive query execution.

171.  case class Subquery(child: LogicalPlan, correlated: Boolean) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable

     This node is inserted at the top of a subquery when it is optimized.

     This node is inserted at the top of a subquery when it is optimized. This makes sure we can recognize a subquery as such, and it allows us to write subquery aware transformations.

     correlated

     flag that indicates the subquery is correlated, and will be rewritten into a join during analysis.

172.  case class SubqueryAlias(identifier: AliasIdentifier, child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable

     Aliased subquery.

     Aliased subquery.

     identifier

     the alias identifier for this subquery.

     child

     the logical plan of this subquery.

173.  trait SupportsSubquery extends LogicalPlan

     A trait to represent the commands that support subqueries.

     A trait to represent the commands that support subqueries. This is used to allow such commands in the subquery-related checks.

174.  case class TableSpec(properties: Map[String, String], provider: Option[String], options: Map[String, String], location: Option[String], comment: Option[String], serde: Option[SerdeInfo], external: Boolean) extends TableSpecBase with Product with Serializable
175.  trait TableSpecBase extends AnyRef
176.  case class Tail(limitExpr: Expression, child: LogicalPlan) extends LogicalPlan with OrderPreservingUnaryNode with Product with Serializable

     This is similar with Limit except:

     This is similar with Limit except:

     - It does not have plans for global/local separately because currently there is only single implementation which initially mimics both global/local tails. See org.apache.spark.sql.execution.CollectTailExec and org.apache.spark.sql.execution.CollectLimitExec

     - Currently, this plan can only be a root node.

177.  case class TruncatePartition(table: LogicalPlan, partitionSpec: PartitionSpec) extends LogicalPlan with V2PartitionCommand with Product with Serializable

     The logical plan of the TRUNCATE TABLE ...

     The logical plan of the TRUNCATE TABLE ... PARTITION command.

178.  case class TruncateTable(table: LogicalPlan) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the TRUNCATE TABLE command.

179.  case class TypedFilter(func: AnyRef, argumentClass: Class[_], argumentSchema: StructType, deserializer: Expression, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     A relation produced by applying func to each element of the child and filter them by the resulting boolean value.

     A relation produced by applying func to each element of the child and filter them by the resulting boolean value.

     This is logically equal to a normal Filter operator whose condition expression is decoding the input row to object and apply the given function with decoded object. However we need the encapsulation of TypedFilter to make the concept more clear and make it easier to write optimizer rules.

180.  trait UnaryCommand extends LogicalPlan with Command with UnaryLike[LogicalPlan]
181.  trait UnaryNode extends LogicalPlan with UnaryLike[LogicalPlan]

     A logical plan node with single child.

182.  trait UnaryParsedStatement extends ParsedStatement with UnaryLike[LogicalPlan]
183.  case class UncacheTable(table: LogicalPlan, ifExists: Boolean, isAnalyzed: Boolean = false) extends LogicalPlan with AnalysisOnlyCommand with Product with Serializable

     The logical plan of the UNCACHE TABLE command.

184.  case class Union(children: Seq[LogicalPlan], byName: Boolean = false, allowMissingCol: Boolean = false) extends LogicalPlan with Product with Serializable

     Logical plan for unioning multiple plans, without a distinct.

     Logical plan for unioning multiple plans, without a distinct. This is UNION ALL in SQL.

     byName

     Whether resolves columns in the children by column names.

     allowMissingCol

     Allows missing columns in children query plans. If it is true, this function allows different set of column names between two Datasets. This can be set to true only if byName is true.

185.  case class Unpivot(ids: Option[Seq[NamedExpression]], values: Option[Seq[Seq[NamedExpression]]], aliases: Option[Seq[Option[String]]], variableColumnName: String, valueColumnNames: Seq[String], child: LogicalPlan) extends LogicalPlan with UnresolvedUnaryNode with Product with Serializable

     A constructor for creating an Unpivot, which will later be converted to an Expand during the query analysis.

     A constructor for creating an Unpivot, which will later be converted to an Expand during the query analysis.

     Either ids or values array must be set. The ids array can be empty, the values array must not be empty if not None.

     A None ids array will be replaced during analysis with all resolved outputs of child except the values. This expansion allows to easily select all non-value columns as id columns.

     A None values array will be replaced during analysis with all resolved outputs of child except the ids. This expansion allows to easily unpivot all non-id columns.

     ids

     Id columns

     values

     Value columns to unpivot

     aliases

     Optional aliases for values

     variableColumnName

     Name of the variable column

     valueColumnNames

     Names of the value columns

     child

     Child operator

     See also

     org.apache.spark.sql.catalyst.analysis.Analyzer.ResolveUnpivot Multiple columns can be unpivoted in one row by providing multiple value column names and the same number of unpivot value expressions:

     // one-dimensional value columns
     Unpivot(
       Some(Seq("id")),
       Some(Seq(
         Seq("val1"),
         Seq("val2")
       )),
       None,
       "var",
       Seq("val")
     )
     
     // two-dimensional value columns
     Unpivot(
       Some(Seq("id")),
       Some(Seq(
         Seq("val1.1", "val1.2"),
         Seq("val2.1", "val2.2")
       )),
       None,
       "var",
       Seq("val1", "val2")
     )

     The variable column will contain the name of the unpivot value while the value columns contain the unpivot values. Multi-dimensional unpivot values can be given aliases: }}} // two-dimensional value columns with aliases Unpivot( Some(Seq("id")), Some(Seq( Seq("val1.1", "val1.2"), Seq("val2.1", "val2.2") )), Some(Seq( Some("val1"), Some("val2") )), "var", Seq("val1", "val2") ) }}} All "value" columns must share a least common data type. Unless they are the same data type, all "value" columns are cast to the nearest common data type. For instance, types IntegerType and LongType are cast to LongType, while IntegerType and StringType do not have a common data type and unpivot fails with an AnalysisException.

     org.apache.spark.sql.catalyst.analysis.TypeCoercionBase.UnpivotCoercion

186.  case class UnresolvedHint(name: String, parameters: Seq[Any], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     A general hint for the child that is not yet resolved.

     A general hint for the child that is not yet resolved. This node is generated by the parser and should be removed This node will be eliminated post analysis.

     name

     the name of the hint

     parameters

     the parameters of the hint

     child

     the LogicalPlan on which this hint applies

187.  case class UnresolvedTableSpec(properties: Map[String, String], provider: Option[String], optionExpression: OptionList, location: Option[String], comment: Option[String], serde: Option[SerdeInfo], external: Boolean) extends UnaryExpression with Unevaluable with TableSpecBase with Product with Serializable
188.  case class UnresolvedWith(child: LogicalPlan, cteRelations: Seq[(String, SubqueryAlias)]) extends LogicalPlan with UnaryNode with Product with Serializable

     A container for holding named common table expressions (CTEs) and a query plan.

     A container for holding named common table expressions (CTEs) and a query plan. This operator will be removed during analysis and the relations will be substituted into child.

     child

     The final query of this CTE.

     cteRelations

     A sequence of pair (alias, the CTE definition) that this CTE defined Each CTE can see the base tables and the previously defined CTEs only.

189.  case class UnsetTableProperties(table: LogicalPlan, propertyKeys: Seq[String], ifExists: Boolean) extends LogicalPlan with AlterTableCommand with Product with Serializable

     The logical plan of the ALTER TABLE ...

     The logical plan of the ALTER TABLE ... UNSET TBLPROPERTIES command.

190.  case class UnsetViewProperties(child: LogicalPlan, propertyKeys: Seq[String], ifExists: Boolean) extends LogicalPlan with UnaryCommand with Product with Serializable

     The logical plan of the ALTER VIEW ...

     The logical plan of the ALTER VIEW ... UNSET TBLPROPERTIES command.

191.  case class UpdateAction(condition: Option[Expression], assignments: Seq[Assignment]) extends MergeAction with Product with Serializable
192.  case class UpdateStarAction(condition: Option[Expression]) extends MergeAction with Product with Serializable
193.  case class UpdateTable(table: LogicalPlan, assignments: Seq[Assignment], condition: Option[Expression]) extends LogicalPlan with UnaryCommand with SupportsSubquery with Product with Serializable

     The logical plan of the UPDATE TABLE command.

194.  trait V2CreateTableAsSelectPlan extends LogicalPlan with V2CreateTablePlan with AnalysisOnlyCommand
195.  trait V2CreateTablePlan extends LogicalPlan

     A trait used for logical plan nodes that create or replace V2 table definitions.

196.  trait V2PartitionCommand extends LogicalPlan with UnaryCommand
197.  trait V2WriteCommand extends LogicalPlan with UnaryCommand with KeepAnalyzedQuery

     Base trait for DataSourceV2 write commands

198.  case class View(desc: CatalogTable, isTempView: Boolean, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable

     A container for holding the view description(CatalogTable) and info whether the view is temporary or not.

     A container for holding the view description(CatalogTable) and info whether the view is temporary or not. If it's a SQL (temp) view, the child should be a logical plan parsed from the CatalogTable.viewText. Otherwise, the view is a temporary one created from a dataframe and the view description should contain a VIEW_CREATED_FROM_DATAFRAME property; in this case, the child must be already resolved.

     This operator will be removed at the end of analysis stage.

     desc

     A view description(CatalogTable) that provides necessary information to resolve the view.

     isTempView

     A flag to indicate whether the view is temporary or not.

     child

     The logical plan of a view operator. If the view description is available, it should be a logical plan parsed from the CatalogTable.viewText.

199.  case class Window(windowExpressions: Seq[NamedExpression], partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
200.  case class WindowGroupLimit(partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], rankLikeFunction: Expression, limit: Int, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
201.  case class WithCTE(plan: LogicalPlan, cteDefs: Seq[CTERelationDef]) extends LogicalPlan with Product with Serializable

     The resolved version of UnresolvedWith with CTE referrences linked to CTE definitions through unique IDs instead of relation aliases.

     The resolved version of UnresolvedWith with CTE referrences linked to CTE definitions through unique IDs instead of relation aliases.

     plan

     The query plan.

     cteDefs

     The CTE definitions.

202.  case class WithWindowDefinition(windowDefinitions: Map[String, WindowSpecDefinition], child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
203.  case class WriteDelta(table: NamedRelation, condition: Expression, query: LogicalPlan, originalTable: NamedRelation, projections: WriteDeltaProjections, write: Option[DeltaWrite] = None) extends LogicalPlan with RowLevelWrite with Product with Serializable

     Writes a delta of rows to an existing table during a row-level operation.

     Writes a delta of rows to an existing table during a row-level operation.

     This node references a query that translates a logical DELETE, UPDATE, MERGE operation into a set of row-level changes to be encoded in the table. Each row in the query represents either a delete, update or insert and stores the operation type in a special column.

     This node is constructed in rules that rewrite DELETE, UPDATE, MERGE operations for data sources that can handle deltas of rows.

     table

     a plan that references a row-level operation table

     condition

     a condition that defines matching records

     query

     a query with a delta of records that should written

     originalTable

     a plan for the original table for which the row-level command was triggered

     projections

     projections for row ID, row, metadata attributes

     write

     a logical write, if already constructed