class CatalogImpl extends Catalog
Internal implementation of the user-facing Catalog.
- Alphabetic
- By Inheritance
- CatalogImpl
- Catalog
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new CatalogImpl(sparkSession: SparkSession)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
cacheTable(tableName: String, storageLevel: StorageLevel): Unit
Caches the specified table or view with the given storage level.
Caches the specified table or view with the given storage level.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- storageLevel
storage level to cache table.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.3.0
-
def
cacheTable(tableName: String): Unit
Caches the specified table or view in-memory.
Caches the specified table or view in-memory.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
def
clearCache(): Unit
Removes all cached tables or views from the in-memory cache.
Removes all cached tables or views from the in-memory cache.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame
(Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options.
(Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
3.1.0
-
def
createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
(Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options.
(Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.2.0
-
def
createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame
(Scala-specific) Creates a table based on the dataset in a data source and a set of options.
(Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
3.1.0
-
def
createTable(tableName: String, source: String, options: Map[String, String]): DataFrame
(Scala-specific) Creates a table based on the dataset in a data source and a set of options.
(Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.2.0
-
def
createTable(tableName: String, path: String, source: String): DataFrame
Creates a table from the given path and returns the corresponding DataFrame.
Creates a table from the given path and returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.2.0
-
def
createTable(tableName: String, path: String): DataFrame
Creates a table from the given path and returns the corresponding DataFrame.
Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.2.0
-
def
createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame
Create a table based on the dataset in a data source, a schema and a set of options.
Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Since
3.1.0
-
def
createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
Create a table based on the dataset in a data source, a schema and a set of options.
Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Since
2.2.0
-
def
createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame
Creates a table based on the dataset in a data source and a set of options.
Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Since
3.1.0
-
def
createTable(tableName: String, source: String, options: Map[String, String]): DataFrame
Creates a table based on the dataset in a data source and a set of options.
Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Since
2.2.0
-
def
currentDatabase: String
Returns the current default database in this session.
Returns the current default database in this session.
- Definition Classes
- CatalogImpl → Catalog
-
def
databaseExists(dbName: String): Boolean
Checks if the database with the specified name exists.
Checks if the database with the specified name exists.
- Definition Classes
- CatalogImpl → Catalog
-
def
dropGlobalTempView(viewName: String): Boolean
Drops the global temporary view with the given view name in the catalog.
Drops the global temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.
- viewName
the identifier of the global temporary view to be dropped.
- returns
true if the view is dropped successfully, false otherwise.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.1.0
-
def
dropTempView(viewName: String): Boolean
Drops the local temporary view with the given view name in the catalog.
Drops the local temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.
- viewName
the identifier of the temporary view to be dropped.
- returns
true if the view is dropped successfully, false otherwise.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
functionExists(dbName: String, functionName: String): Boolean
Checks if the function with the specified name exists in the specified database.
Checks if the function with the specified name exists in the specified database.
- dbName
is a name that designates a database.
- functionName
is an unqualified name that designates a function.
- Definition Classes
- CatalogImpl → Catalog
-
def
functionExists(functionName: String): Boolean
Checks if the function with the specified name exists.
Checks if the function with the specified name exists. This can either be a temporary function or a function.
- functionName
is either a qualified or unqualified name that designates a function. If no database identifier is provided, it refers to a function in the current database.
- Definition Classes
- CatalogImpl → Catalog
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getDatabase(dbName: String): Database
Gets the database with the specified name.
Gets the database with the specified name. This throws an
AnalysisExceptionwhen noDatabasecan be found.- Definition Classes
- CatalogImpl → Catalog
-
def
getFunction(dbName: String, functionName: String): Function
Gets the function with the specified name.
Gets the function with the specified name. This returns
Nonewhen noFunctioncan be found.- dbName
is a name that designates a database.
- functionName
is an unqualified name that designates a function in the specified database
- Definition Classes
- CatalogImpl → Catalog
-
def
getFunction(functionName: String): Function
Gets the function with the specified name.
Gets the function with the specified name. This function can be a temporary function or a function. This throws an
AnalysisExceptionwhen noFunctioncan be found.- functionName
is either a qualified or unqualified name that designates a function. If no database identifier is provided, it refers to a temporary function or a function in the current database.
- Definition Classes
- CatalogImpl → Catalog
-
def
getTable(dbName: String, tableName: String): Table
Gets the table or view with the specified name in the specified database.
Gets the table or view with the specified name in the specified database. This throws an
AnalysisExceptionwhen noTablecan be found.- Definition Classes
- CatalogImpl → Catalog
-
def
getTable(tableName: String): Table
Gets the table or view with the specified name.
Gets the table or view with the specified name. This table can be a temporary view or a table/view. This throws an
AnalysisExceptionwhen noTablecan be found.- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
isCached(qName: Dataset[_]): Boolean
Returns true if the Dataset is currently cached in-memory.
-
def
isCached(tableName: String): Boolean
Returns true if the table or view is currently cached in-memory.
Returns true if the table or view is currently cached in-memory.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
listColumns(dbName: String, tableName: String): Dataset[catalog.Column]
Returns a list of columns for the given table/view or temporary view in the specified database.
Returns a list of columns for the given table/view or temporary view in the specified database.
- dbName
is a name that designates a database.
- tableName
is an unqualified name that designates a table/view.
- Definition Classes
- CatalogImpl → Catalog
- Annotations
- @throws( "database or table does not exist" )
-
def
listColumns(tableName: String): Dataset[catalog.Column]
Returns a list of columns for the given table/view or temporary view.
Returns a list of columns for the given table/view or temporary view.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Annotations
- @throws( "table does not exist" )
-
def
listDatabases(): Dataset[Database]
Returns a list of databases available across all sessions.
Returns a list of databases available across all sessions.
- Definition Classes
- CatalogImpl → Catalog
-
def
listFunctions(dbName: String): Dataset[Function]
Returns a list of functions registered in the specified database.
Returns a list of functions registered in the specified database. This includes all temporary functions
- Definition Classes
- CatalogImpl → Catalog
- Annotations
- @throws( "database does not exist" )
-
def
listFunctions(): Dataset[Function]
Returns a list of functions registered in the current database.
Returns a list of functions registered in the current database. This includes all temporary functions
- Definition Classes
- CatalogImpl → Catalog
-
def
listTables(dbName: String): Dataset[Table]
Returns a list of tables in the specified database.
Returns a list of tables in the specified database. This includes all temporary tables.
- Definition Classes
- CatalogImpl → Catalog
- Annotations
- @throws( "database does not exist" )
-
def
listTables(): Dataset[Table]
Returns a list of tables in the current database.
Returns a list of tables in the current database. This includes all temporary tables.
- Definition Classes
- CatalogImpl → Catalog
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
recoverPartitions(tableName: String): Unit
Recovers all the partitions in the directory of a table and update the catalog.
Recovers all the partitions in the directory of a table and update the catalog. Only works with a partitioned table, and not a temporary view.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.1.1
-
def
refreshByPath(resourcePath: String): Unit
Refreshes the cache entry and the associated metadata for all Dataset (if any), that contain the given data source path.
Refreshes the cache entry and the associated metadata for all Dataset (if any), that contain the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
def
refreshTable(tableName: String): Unit
The method fully refreshes a table or view with the given name including:
The method fully refreshes a table or view with the given name including:
- The relation cache in the session catalog. The method removes table entry from the cache.
2. The file indexes of all relations used by the given view.
3. Table/View schema in the Hive Metastore if the SQL config
spark.sql.hive.caseSensitiveInferenceModeis set toINFER_AND_SAVE. 4. Cached data of the given table or view, and all its dependents that refer to it. Existing cached data will be cleared and the cache will be lazily filled when the next time the table/view or the dependents are accessed.
The method does not do:
- schema inference for file source tables
- statistics update
The method is supposed to be used in all cases when need to refresh table/view data and meta-data.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
- The relation cache in the session catalog. The method removes table entry from the cache.
2. The file indexes of all relations used by the given view.
3. Table/View schema in the Hive Metastore if the SQL config
-
def
setCurrentDatabase(dbName: String): Unit
Sets the current default database in this session.
Sets the current default database in this session.
- Definition Classes
- CatalogImpl → Catalog
- Annotations
- @throws( "database does not exist" )
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
tableExists(dbName: String, tableName: String): Boolean
Checks if the table or view with the specified name exists in the specified database.
Checks if the table or view with the specified name exists in the specified database.
- dbName
is a name that designates a database.
- tableName
is an unqualified name that designates a table.
- Definition Classes
- CatalogImpl → Catalog
-
def
tableExists(tableName: String): Boolean
Checks if the table or view with the specified name exists.
Checks if the table or view with the specified name exists. This can either be a temporary view or a table/view.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
uncacheTable(tableName: String): Unit
Removes the specified table or view from the in-memory cache.
Removes the specified table or view from the in-memory cache.
- tableName
is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.
- Definition Classes
- CatalogImpl → Catalog
- Since
2.0.0
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
Deprecated Value Members
-
def
createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
(Scala-specific) Create a table from the given path based on a data source, a schema and a set of options.
(Scala-specific) Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0
-
def
createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
Create a table from the given path based on a data source, a schema and a set of options.
Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0
-
def
createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame
(Scala-specific) Creates a table from the given path based on a data source and a set of options.
(Scala-specific) Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0
-
def
createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame
Creates a table from the given path based on a data source and a set of options.
Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0
-
def
createExternalTable(tableName: String, path: String, source: String): DataFrame
Creates a table from the given path based on a data source and returns the corresponding DataFrame.
Creates a table from the given path based on a data source and returns the corresponding DataFrame.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0
-
def
createExternalTable(tableName: String, path: String): DataFrame
Creates a table from the given path and returns the corresponding DataFrame.
Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.
- tableName
is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.
- Definition Classes
- Catalog
- Annotations
- @deprecated
- Deprecated
(Since version 2.2.0) use createTable instead.
- Since
2.0.0