org.apache.druid.metadata

Class IndexerSQLMetadataStorageCoordinator

java.lang.Object

org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator

All Implemented Interfaces:

IndexerMetadataStorageCoordinator

public class IndexerSQLMetadataStorageCoordinator
extends Object
implements IndexerMetadataStorageCoordinator

Constructor Summary

Constructors

Constructor and Description
IndexerSQLMetadataStorageCoordinator(com.fasterxml.jackson.databind.ObjectMapper jsonMapper, MetadataStorageTablesConfig dbTables, SQLMetadataConnector connector)

Method Summary

Modifier and Type	Method and Description
SegmentIdWithShardSpec	allocatePendingSegment(String dataSource, String sequenceName, String previousSegmentId, org.joda.time.Interval interval, ShardSpecFactory shardSpecFactory, String maxVersion, boolean skipSegmentLineageCheck) Allocate a new pending segment in the pending segments table.
Set<DataSegment>	announceHistoricalSegments(Set<DataSegment> segments) Attempts to insert a set of segments to the database.
SegmentPublishResult	announceHistoricalSegments(Set<DataSegment> segments, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata) Attempts to insert a set of segments to the metadata storage.
SegmentPublishResult	commitMetadataOnly(String dataSource, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata) Similar to IndexerMetadataStorageCoordinator.announceHistoricalSegments(Set), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made.
boolean	deleteDataSourceMetadata(String dataSource) Removes entry for 'dataSource' from the dataSource metadata table.
int	deletePendingSegments(String dataSource, org.joda.time.Interval deleteInterval) Delete pending segments created in the given interval for the given dataSource from the pending segments table.
void	deleteSegments(Set<DataSegment> segments)
DataSourceMetadata	getDataSourceMetadata(String dataSource) Read dataSource metadata.
List<DataSegment>	getUnusedSegmentsForInterval(String dataSource, org.joda.time.Interval interval) Get all published segments which include ONLY data within the given interval and are not marked as used.
Collection<Pair<DataSegment,String>>	getUsedSegmentAndCreatedDateForInterval(String dataSource, org.joda.time.Interval interval) Get all published segments which are marked as used and the created_date of these segments in a given datasource and interval.
Collection<DataSegment>	getUsedSegmentsForIntervals(String dataSource, List<org.joda.time.Interval> intervals, Segments visibility) Get all published segments which may include any data in the interval and are marked as used.
boolean	insertDataSourceMetadata(String dataSource, DataSourceMetadata metadata) Insert dataSourceMetadata entry for 'dataSource'.
boolean	resetDataSourceMetadata(String dataSource, DataSourceMetadata dataSourceMetadata) Resets dataSourceMetadata entry for 'dataSource' to the one supplied.
void	start()
protected org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.DataSourceMetadataUpdateResult	updateDataSourceMetadataWithHandle(org.skife.jdbi.v2.Handle handle, String dataSource, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata) Compare-and-swap dataSource metadata in a transaction.
void	updateSegmentMetadata(Set<DataSegment> segments)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator

getUsedSegmentsForInterval

Constructor Detail

IndexerSQLMetadataStorageCoordinator

@Inject
public IndexerSQLMetadataStorageCoordinator(com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
                                                    MetadataStorageTablesConfig dbTables,
                                                    SQLMetadataConnector connector)

Method Detail

start

public void start()

getUsedSegmentsForIntervals

public Collection<DataSegment> getUsedSegmentsForIntervals(String dataSource,
                                                           List<org.joda.time.Interval> intervals,
                                                           Segments visibility)

Description copied from interface: IndexerMetadataStorageCoordinator

Get all published segments which may include any data in the interval and are marked as used. The order of segments within the returned collection is unspecified, but each segment is guaranteed to appear in the collection only once.

Specified by:

getUsedSegmentsForIntervals in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - The datasource to query

intervals - The intervals for which all applicable and used datasources are requested.

visibility - Whether only visible or visible as well as overshadowed segments should be returned. The visibility is considered within the specified intervals: that is, a segment which is visible outside of the specified intervals, but overshadowed on the specified intervals will not be returned if Segments.ONLY_VISIBLE is passed. See more precise description in the doc for Segments.

Returns:

The DataSegments which include data in the requested intervals. These segments may contain data outside the requested interval.

announceHistoricalSegments

public Set<DataSegment> announceHistoricalSegments(Set<DataSegment> segments)
                                            throws IOException

Attempts to insert a set of segments to the database. Returns the set of segments actually added (segments with identifiers already in the database will not be added).

Specified by:

announceHistoricalSegments in interface IndexerMetadataStorageCoordinator

Parameters:

segments - set of segments to add

Returns:

set of segments actually added

Throws:

IOException

announceHistoricalSegments

public SegmentPublishResult announceHistoricalSegments(Set<DataSegment> segments,
                                                       @Nullable
                                                       DataSourceMetadata startMetadata,
                                                       @Nullable
                                                       DataSourceMetadata endMetadata)
                                                throws IOException

Description copied from interface: IndexerMetadataStorageCoordinator

Attempts to insert a set of segments to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

If startMetadata and endMetadata are set, this insertion will be atomic with a compare-and-swap on dataSource commit metadata.

Specified by:

announceHistoricalSegments in interface IndexerMetadataStorageCoordinator

Parameters:

segments - set of segments to add, must all be from the same dataSource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata). If null, this insert will not involve a metadata transaction

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata). If null, this insert will not involve a metadata transaction

Returns:

segment publish result indicating transaction success or failure, and set of segments actually published. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

Throws:

IOException

commitMetadataOnly

public SegmentPublishResult commitMetadataOnly(String dataSource,
                                               DataSourceMetadata startMetadata,
                                               DataSourceMetadata endMetadata)

Description copied from interface: IndexerMetadataStorageCoordinator

Similar to IndexerMetadataStorageCoordinator.announceHistoricalSegments(Set), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made. The metadata should undergo the same validation checks as performed by announceHistoricalSegments.

Specified by:

commitMetadataOnly in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - the datasource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata).

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata).

Returns:

segment publish result indicating transaction success or failure. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

allocatePendingSegment

public SegmentIdWithShardSpec allocatePendingSegment(String dataSource,
                                                     String sequenceName,
                                                     @Nullable
                                                     String previousSegmentId,
                                                     org.joda.time.Interval interval,
                                                     ShardSpecFactory shardSpecFactory,
                                                     String maxVersion,
                                                     boolean skipSegmentLineageCheck)

Description copied from interface: IndexerMetadataStorageCoordinator

Allocate a new pending segment in the pending segments table. This segment identifier will never be given out again, unless another call is made with the same dataSource, sequenceName, and previousSegmentId.

The sequenceName and previousSegmentId parameters are meant to make it easy for two independent ingestion tasks to produce the same series of segments.

Note that a segment sequence may include segments with a variety of different intervals and versions.

Specified by:

allocatePendingSegment in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - dataSource for which to allocate a segment

sequenceName - name of the group of ingestion tasks producing a segment series

previousSegmentId - previous segment in the series; may be null or empty, meaning this is the first segment

interval - interval for which to allocate a segment

shardSpecFactory - shardSpecFactory containing all necessary information to create a shardSpec for the new segmentId

maxVersion - use this version if we have no better version to use. The returned segment identifier may have a version lower than this one, but will not have one higher.

skipSegmentLineageCheck - if true, perform lineage validation using previousSegmentId for this sequence. Should be set to false if replica tasks would index events in same order

Returns:

the pending segment identifier, or null if it was impossible to allocate a new segment

deletePendingSegments

public int deletePendingSegments(String dataSource,
                                 org.joda.time.Interval deleteInterval)

Description copied from interface: IndexerMetadataStorageCoordinator

Delete pending segments created in the given interval for the given dataSource from the pending segments table. The created_date field of the pending segments table is checked to find segments to be deleted.

Specified by:

deletePendingSegments in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - dataSource

deleteInterval - interval to check the created_date of pendingSegments

Returns:

number of deleted pending segments

getDataSourceMetadata

public DataSourceMetadata getDataSourceMetadata(String dataSource)

Read dataSource metadata. Returns null if there is no metadata.

Specified by:

getDataSourceMetadata in interface IndexerMetadataStorageCoordinator

updateDataSourceMetadataWithHandle

protected org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.DataSourceMetadataUpdateResult updateDataSourceMetadataWithHandle(org.skife.jdbi.v2.Handle handle,
                                                                                                                                           String dataSource,
                                                                                                                                           DataSourceMetadata startMetadata,
                                                                                                                                           DataSourceMetadata endMetadata)
                                                                                                                                    throws IOException

Compare-and-swap dataSource metadata in a transaction. This will only modify dataSource metadata if it equals oldCommitMetadata when this function is called (based on T.equals). This method is idempotent in that if the metadata already equals newCommitMetadata, it will return true.

Parameters:

handle - database handle

dataSource - druid dataSource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata)

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata)

Returns:

SUCCESS if dataSource metadata was updated from matching startMetadata to matching endMetadata, FAILURE or TRY_AGAIN if it definitely was not updated. This guarantee is meant to help announceHistoricalSegments(Set, DataSourceMetadata, DataSourceMetadata) achieve its own guarantee.

Throws:

RuntimeException - if state is unknown after this call

IOException

deleteDataSourceMetadata

public boolean deleteDataSourceMetadata(String dataSource)

Description copied from interface: IndexerMetadataStorageCoordinator

Removes entry for 'dataSource' from the dataSource metadata table.

Specified by:

deleteDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - identifier

Returns:

true if the entry was deleted, false otherwise

resetDataSourceMetadata

public boolean resetDataSourceMetadata(String dataSource,
                                       DataSourceMetadata dataSourceMetadata)
                                throws IOException

Description copied from interface: IndexerMetadataStorageCoordinator

Resets dataSourceMetadata entry for 'dataSource' to the one supplied.

Specified by:

resetDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - identifier

dataSourceMetadata - value to set

Returns:

true if the entry was reset, false otherwise

Throws:

IOException

updateSegmentMetadata

public void updateSegmentMetadata(Set<DataSegment> segments)

Specified by:

updateSegmentMetadata in interface IndexerMetadataStorageCoordinator

deleteSegments

public void deleteSegments(Set<DataSegment> segments)

Specified by:

deleteSegments in interface IndexerMetadataStorageCoordinator

getUnusedSegmentsForInterval

public List<DataSegment> getUnusedSegmentsForInterval(String dataSource,
                                                      org.joda.time.Interval interval)

Description copied from interface: IndexerMetadataStorageCoordinator

Get all published segments which include ONLY data within the given interval and are not marked as used.

Specified by:

getUnusedSegmentsForInterval in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - The datasource the segments belong to

interval - Filter the data segments to ones that include data in this interval exclusively. Start is inclusive, end is exclusive

Returns:

DataSegments which include ONLY data within the requested interval and are not marked as used. Data segments NOT returned here may include data in the interval

getUsedSegmentAndCreatedDateForInterval

public Collection<Pair<DataSegment,String>> getUsedSegmentAndCreatedDateForInterval(String dataSource,
                                                                                    org.joda.time.Interval interval)

Description copied from interface: IndexerMetadataStorageCoordinator

Get all published segments which are marked as used and the created_date of these segments in a given datasource and interval.

Specified by:

getUsedSegmentAndCreatedDateForInterval in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - The datasource to query

interval - The interval for which all applicable and used datasources are requested. Start is inclusive, end is exclusive

Returns:

The DataSegments and the related created_date of segments which include data in the requested interval

insertDataSourceMetadata

public boolean insertDataSourceMetadata(String dataSource,
                                        DataSourceMetadata metadata)

Description copied from interface: IndexerMetadataStorageCoordinator

Insert dataSourceMetadata entry for 'dataSource'.

Specified by:

insertDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - identifier

metadata - value to set

Returns:

true if the entry was inserted, false otherwise