com.databricks.sdk.service.pipelines

Class PipelinesAPI

java.lang.Object

com.databricks.sdk.service.pipelines.PipelinesAPI

@Generated
public class PipelinesAPI
extends Object

The Lakeflow Spark Declarative Pipelines API allows you to create, edit, delete, start, and view details about pipelines.

Spark Declarative Pipelines is a framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data, and Spark Declarative Pipelines manages task orchestration, cluster management, monitoring, data quality, and error handling.

Instead of defining your data pipelines using a series of separate Apache Spark tasks, Spark Declarative Pipelines manages how your data is transformed based on a target schema you define for each processing step. You can also enforce data quality with Spark Declarative Pipelines expectations. Expectations allow you to define expected data quality and specify how to handle records that fail those expectations.

Constructor Summary

Constructors

Constructor and Description
PipelinesAPI(ApiClient apiClient) Regular-use constructor
PipelinesAPI(PipelinesService mock) Constructor for mocks

Method Summary

Modifier and Type	Method and Description
ClonePipelineResponse	clone(ClonePipelineRequest request) Creates a new pipeline using Unity Catalog from a pipeline using Hive Metastore.
CreatePipelineResponse	create(CreatePipeline request) Creates a new data processing pipeline based on the requested configuration.
void	delete(DeletePipelineRequest request) Deletes a pipeline.
void	delete(String pipelineId)
GetPipelineResponse	get(GetPipelineRequest request) Get a pipeline.
GetPipelineResponse	get(String pipelineId)
GetPipelinePermissionLevelsResponse	getPermissionLevels(GetPipelinePermissionLevelsRequest request) Gets the permission levels that a user can have on an object.
GetPipelinePermissionLevelsResponse	getPermissionLevels(String pipelineId)
PipelinePermissions	getPermissions(GetPipelinePermissionsRequest request) Gets the permissions of a pipeline.
PipelinePermissions	getPermissions(String pipelineId)
GetUpdateResponse	getUpdate(GetUpdateRequest request) Gets an update from an active pipeline.
GetUpdateResponse	getUpdate(String pipelineId, String updateId)
PipelinesService	impl()
Iterable<PipelineEvent>	listPipelineEvents(ListPipelineEventsRequest request) Retrieves events for a pipeline.
Iterable<PipelineEvent>	listPipelineEvents(String pipelineId)
Iterable<PipelineStateInfo>	listPipelines(ListPipelinesRequest request) Lists pipelines defined in the Spark Declarative Pipelines system.
ListUpdatesResponse	listUpdates(ListUpdatesRequest request) List updates for an active pipeline.
ListUpdatesResponse	listUpdates(String pipelineId)
PipelinePermissions	setPermissions(PipelinePermissionsRequest request) Sets permissions on an object, replacing existing permissions if they exist.
StartUpdateResponse	startUpdate(StartUpdate request) Starts a new update for the pipeline.
Wait<GetPipelineResponse,Void>	stop(StopRequest request) Stops the pipeline by canceling the active update.
void	update(EditPipeline request) Updates a pipeline with the supplied configuration.
PipelinePermissions	updatePermissions(PipelinePermissionsRequest request) Updates the permissions on a pipeline.
GetPipelineResponse	waitGetPipelineIdle(String pipelineId)
GetPipelineResponse	waitGetPipelineIdle(String pipelineId, Duration timeout, Consumer<GetPipelineResponse> callback)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PipelinesAPI

public PipelinesAPI(ApiClient apiClient)

Regular-use constructor

PipelinesAPI

public PipelinesAPI(PipelinesService mock)

Constructor for mocks

Method Detail

waitGetPipelineIdle

public GetPipelineResponse waitGetPipelineIdle(String pipelineId)
                                        throws TimeoutException

Throws:

TimeoutException

waitGetPipelineIdle

public GetPipelineResponse waitGetPipelineIdle(String pipelineId,
                                               Duration timeout,
                                               Consumer<GetPipelineResponse> callback)
                                        throws TimeoutException

Throws:

TimeoutException

clone

public ClonePipelineResponse clone(ClonePipelineRequest request)

Creates a new pipeline using Unity Catalog from a pipeline using Hive Metastore. This method returns the ID of the newly created clone. Additionally, this method starts an update for the newly created pipeline.

create

public CreatePipelineResponse create(CreatePipeline request)

Creates a new data processing pipeline based on the requested configuration. If successful, this method returns the ID of the new pipeline.

delete

public void delete(String pipelineId)

delete

public void delete(DeletePipelineRequest request)

Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline deletion will cascade to all pipeline tables. Please reach out to Databricks support for assistance to undo this action.

get

public GetPipelineResponse get(String pipelineId)

get

public GetPipelineResponse get(GetPipelineRequest request)

Get a pipeline.

getPermissionLevels

public GetPipelinePermissionLevelsResponse getPermissionLevels(String pipelineId)

getPermissionLevels

public GetPipelinePermissionLevelsResponse getPermissionLevels(GetPipelinePermissionLevelsRequest request)

Gets the permission levels that a user can have on an object.

getPermissions

public PipelinePermissions getPermissions(String pipelineId)

getPermissions

public PipelinePermissions getPermissions(GetPipelinePermissionsRequest request)

Gets the permissions of a pipeline. Pipelines can inherit permissions from their root object.

getUpdate

public GetUpdateResponse getUpdate(String pipelineId,
                                   String updateId)

getUpdate

public GetUpdateResponse getUpdate(GetUpdateRequest request)

Gets an update from an active pipeline.

listPipelineEvents

public Iterable<PipelineEvent> listPipelineEvents(String pipelineId)

listPipelineEvents

public Iterable<PipelineEvent> listPipelineEvents(ListPipelineEventsRequest request)

Retrieves events for a pipeline.

listPipelines

public Iterable<PipelineStateInfo> listPipelines(ListPipelinesRequest request)

Lists pipelines defined in the Spark Declarative Pipelines system.

listUpdates

public ListUpdatesResponse listUpdates(String pipelineId)

listUpdates

public ListUpdatesResponse listUpdates(ListUpdatesRequest request)

List updates for an active pipeline.

setPermissions

public PipelinePermissions setPermissions(PipelinePermissionsRequest request)

Sets permissions on an object, replacing existing permissions if they exist. Deletes all direct permissions if none are specified. Objects can inherit permissions from their root object.

startUpdate

public StartUpdateResponse startUpdate(StartUpdate request)

Starts a new update for the pipeline. If there is already an active update for the pipeline, the request will fail and the active update will remain running.

stop

public Wait<GetPipelineResponse,Void> stop(StopRequest request)

Stops the pipeline by canceling the active update. If there is no active update for the pipeline, this request is a no-op.

update

public void update(EditPipeline request)

Updates a pipeline with the supplied configuration.

updatePermissions

public PipelinePermissions updatePermissions(PipelinePermissionsRequest request)

Updates the permissions on a pipeline. Pipelines can inherit permissions from their root object.

impl

public PipelinesService impl()