@Generated public interface PipelinesService
Spark Declarative Pipelines is a framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data, and Spark Declarative Pipelines manages task orchestration, cluster management, monitoring, data quality, and error handling.
Instead of defining your data pipelines using a series of separate Apache Spark tasks, Spark Declarative Pipelines manages how your data is transformed based on a target schema you define for each processing step. You can also enforce data quality with Spark Declarative Pipelines expectations. Expectations allow you to define expected data quality and specify how to handle records that fail those expectations.
This is the high-level interface, that contains generated methods.
Evolving: this interface is under development. Method signatures may change.
| Modifier and Type | Method and Description |
|---|---|
ClonePipelineResponse |
clone(ClonePipelineRequest clonePipelineRequest)
Creates a new pipeline using Unity Catalog from a pipeline using Hive Metastore.
|
CreatePipelineResponse |
create(CreatePipeline createPipeline)
Creates a new data processing pipeline based on the requested configuration.
|
void |
delete(DeletePipelineRequest deletePipelineRequest)
Deletes a pipeline.
|
GetPipelineResponse |
get(GetPipelineRequest getPipelineRequest)
Get a pipeline.
|
GetPipelinePermissionLevelsResponse |
getPermissionLevels(GetPipelinePermissionLevelsRequest getPipelinePermissionLevelsRequest)
Gets the permission levels that a user can have on an object.
|
PipelinePermissions |
getPermissions(GetPipelinePermissionsRequest getPipelinePermissionsRequest)
Gets the permissions of a pipeline.
|
GetUpdateResponse |
getUpdate(GetUpdateRequest getUpdateRequest)
Gets an update from an active pipeline.
|
ListPipelineEventsResponse |
listPipelineEvents(ListPipelineEventsRequest listPipelineEventsRequest)
Retrieves events for a pipeline.
|
ListPipelinesResponse |
listPipelines(ListPipelinesRequest listPipelinesRequest)
Lists pipelines defined in the Spark Declarative Pipelines system.
|
ListUpdatesResponse |
listUpdates(ListUpdatesRequest listUpdatesRequest)
List updates for an active pipeline.
|
PipelinePermissions |
setPermissions(PipelinePermissionsRequest pipelinePermissionsRequest)
Sets permissions on an object, replacing existing permissions if they exist.
|
StartUpdateResponse |
startUpdate(StartUpdate startUpdate)
Starts a new update for the pipeline.
|
void |
stop(StopRequest stopRequest)
Stops the pipeline by canceling the active update.
|
void |
update(EditPipeline editPipeline)
Updates a pipeline with the supplied configuration.
|
PipelinePermissions |
updatePermissions(PipelinePermissionsRequest pipelinePermissionsRequest)
Updates the permissions on a pipeline.
|
ClonePipelineResponse clone(ClonePipelineRequest clonePipelineRequest)
CreatePipelineResponse create(CreatePipeline createPipeline)
void delete(DeletePipelineRequest deletePipelineRequest)
GetPipelineResponse get(GetPipelineRequest getPipelineRequest)
GetPipelinePermissionLevelsResponse getPermissionLevels(GetPipelinePermissionLevelsRequest getPipelinePermissionLevelsRequest)
PipelinePermissions getPermissions(GetPipelinePermissionsRequest getPipelinePermissionsRequest)
GetUpdateResponse getUpdate(GetUpdateRequest getUpdateRequest)
ListPipelineEventsResponse listPipelineEvents(ListPipelineEventsRequest listPipelineEventsRequest)
ListPipelinesResponse listPipelines(ListPipelinesRequest listPipelinesRequest)
ListUpdatesResponse listUpdates(ListUpdatesRequest listUpdatesRequest)
PipelinePermissions setPermissions(PipelinePermissionsRequest pipelinePermissionsRequest)
StartUpdateResponse startUpdate(StartUpdate startUpdate)
void stop(StopRequest stopRequest)
void update(EditPipeline editPipeline)
PipelinePermissions updatePermissions(PipelinePermissionsRequest pipelinePermissionsRequest)
Copyright © 2026. All rights reserved.