org.apache.flink.runtime.state.filesystem

类 AbstractFsCheckpointStorageAccess

java.lang.Object

org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorageAccess

所有已实现的接口:

CheckpointStorageAccess, CheckpointStorageCoordinatorView, CheckpointStorageWorkerView

直接已知子类:

FsCheckpointStorageAccess, MemoryBackendCheckpointStorageAccess

public abstract class AbstractFsCheckpointStorageAccess
extends Object
implements CheckpointStorageAccess

An implementation of durable checkpoint storage to file systems.

Checkpoint Layout

The checkpoint storage is configured with a base directory and persists the checkpoint data of specific checkpoints in specific subdirectories. For example, if the base directory was set to hdfs://namenode:port/flink-checkpoints/, the state backend will create a subdirectory with the job's ID that will contain the actual checkpoints: (hdfs://namenode:port/flink-checkpoints/1b080b6e710aabbef8993ab18c6de98b)

Each checkpoint individually will store all its files in a subdirectory that includes the checkpoint number, such as hdfs://namenode:port/flink-checkpoints/1b080b6e710aabbef8993ab18c6de98b/chk-17/.

Savepoint Layout

A savepoint that is set to be stored in path hdfs://namenode:port/flink-savepoints/, will create a subdirectory savepoint-jobId(0, 6)-randomDigits in which it stores all savepoint data. The random digits are added as "entropy" to avoid directory collisions.

Metadata File

A completed checkpoint writes its metadata into a file ''.

字段概要

字段

限定符和类型	字段和说明
static String	CHECKPOINT_DIR_PREFIX The prefix of the directory containing the data exclusive to a checkpoint.
static String	CHECKPOINT_SHARED_STATE_DIR The name of the directory for shared checkpoint state.
static String	CHECKPOINT_TASK_OWNED_STATE_DIR The name of the directory for state not owned/released by the master, but by the TaskManagers.
static String	METADATA_FILE_NAME The name of the metadata files in checkpoints / savepoints.

构造器概要

构造器

限定符	构造器和说明
protected	AbstractFsCheckpointStorageAccess(org.apache.flink.api.common.JobID jobId, org.apache.flink.core.fs.Path defaultSavepointDirectory) Creates a new checkpoint storage.

方法概要

限定符和类型	方法和说明
protected static org.apache.flink.core.fs.Path	createCheckpointDirectory(org.apache.flink.core.fs.Path baseDirectory, long checkpointId) Creates the directory path for the data exclusive to a specific checkpoint.
protected abstract CheckpointStorageLocation	createSavepointLocation(org.apache.flink.core.fs.FileSystem fs, org.apache.flink.core.fs.Path location)
static org.apache.flink.core.fs.Path	decodePathFromReference(CheckpointStorageLocationReference reference) Decodes the given reference into a path.
static CheckpointStorageLocationReference	encodePathAsReference(org.apache.flink.core.fs.Path path) Encodes the given path as a reference in bytes.
protected static org.apache.flink.core.fs.Path	getCheckpointDirectoryForJob(org.apache.flink.core.fs.Path baseCheckpointPath, org.apache.flink.api.common.JobID jobId) Builds directory into which a specific job checkpoints, meaning the directory inside which it creates the checkpoint-specific subdirectories.
org.apache.flink.core.fs.Path	getDefaultSavepointDirectory() Gets the default directory for savepoints.
boolean	hasDefaultSavepointLocation() Checks whether the storage has a default savepoint location configured.
CheckpointStorageLocation	initializeLocationForSavepoint(long checkpointId, String externalLocationPointer) Creates a file system based storage location for a savepoint.
CompletedCheckpointStorageLocation	resolveCheckpoint(String checkpointPointer) Resolves the given pointer to a checkpoint/savepoint into a checkpoint location.
static FsCompletedCheckpointStorageLocation	resolveCheckpointPointer(String checkpointPointer) Takes the given string (representing a pointer to a checkpoint) and resolves it to a file status for the checkpoint's metadata file.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.state.CheckpointStorageCoordinatorView

initializeBaseLocationsForCheckpoint, initializeLocationForCheckpoint, supportsHighlyAvailableStorage

从接口继承的方法 org.apache.flink.runtime.state.CheckpointStorageWorkerView

createTaskOwnedCheckpointStateToolset, createTaskOwnedStateStream, resolveCheckpointStorageLocation

字段详细资料

CHECKPOINT_DIR_PREFIX

public static final String CHECKPOINT_DIR_PREFIX

The prefix of the directory containing the data exclusive to a checkpoint.

另请参阅:

常量字段值

CHECKPOINT_SHARED_STATE_DIR

public static final String CHECKPOINT_SHARED_STATE_DIR

The name of the directory for shared checkpoint state.

另请参阅:

常量字段值

CHECKPOINT_TASK_OWNED_STATE_DIR

public static final String CHECKPOINT_TASK_OWNED_STATE_DIR

The name of the directory for state not owned/released by the master, but by the TaskManagers.

另请参阅:

常量字段值

METADATA_FILE_NAME

public static final String METADATA_FILE_NAME

The name of the metadata files in checkpoints / savepoints.

另请参阅:

常量字段值

构造器详细资料

AbstractFsCheckpointStorageAccess

protected AbstractFsCheckpointStorageAccess(org.apache.flink.api.common.JobID jobId,
                                            @Nullable
                                            org.apache.flink.core.fs.Path defaultSavepointDirectory)

Creates a new checkpoint storage.

参数:

jobId - The ID of the job that writes the checkpoints.

defaultSavepointDirectory - The default location for savepoints, or null, if none is set.

方法详细资料

getDefaultSavepointDirectory

@Nullable
public org.apache.flink.core.fs.Path getDefaultSavepointDirectory()

Gets the default directory for savepoints. Returns null, if no default savepoint directory is configured.

hasDefaultSavepointLocation

public boolean hasDefaultSavepointLocation()

从接口复制的说明: CheckpointStorageCoordinatorView

Checks whether the storage has a default savepoint location configured.

指定者:

hasDefaultSavepointLocation 在接口中 CheckpointStorageCoordinatorView

resolveCheckpoint

public CompletedCheckpointStorageLocation resolveCheckpoint(String checkpointPointer)
                                                     throws IOException

从接口复制的说明: CheckpointStorageCoordinatorView

Resolves the given pointer to a checkpoint/savepoint into a checkpoint location. The location supports reading the checkpoint metadata, or disposing the checkpoint storage location.

If the state backend cannot understand the format of the pointer (for example because it was created by a different state backend) this method should throw an IOException.

指定者:

resolveCheckpoint 在接口中 CheckpointStorageCoordinatorView

参数:

checkpointPointer - The external checkpoint pointer to resolve.

返回:

The checkpoint location handle.

抛出:

IOException - Thrown, if the state backend does not understand the pointer, or if the pointer could not be resolved due to an I/O error.

initializeLocationForSavepoint

public CheckpointStorageLocation initializeLocationForSavepoint(long checkpointId,
                                                                @Nullable
                                                                String externalLocationPointer)
                                                         throws IOException

Creates a file system based storage location for a savepoint.

This methods implements the logic that decides which location to use (given optional parameters for a configured location and a location passed for this specific savepoint) and how to name and initialize the savepoint directory.

指定者:

initializeLocationForSavepoint 在接口中 CheckpointStorageCoordinatorView

参数:

externalLocationPointer - The target location pointer for the savepoint. Must be a valid URI. Null, if not supplied.

checkpointId - The checkpoint ID of the savepoint.

返回:

The checkpoint storage location for the savepoint.

抛出:

IOException - Thrown if the target directory could not be created.

createSavepointLocation

protected abstract CheckpointStorageLocation createSavepointLocation(org.apache.flink.core.fs.FileSystem fs,
                                                                     org.apache.flink.core.fs.Path location)
                                                              throws IOException

抛出:

IOException

getCheckpointDirectoryForJob

protected static org.apache.flink.core.fs.Path getCheckpointDirectoryForJob(org.apache.flink.core.fs.Path baseCheckpointPath,
                                                                            org.apache.flink.api.common.JobID jobId)

Builds directory into which a specific job checkpoints, meaning the directory inside which it creates the checkpoint-specific subdirectories.

This method only succeeds if a base checkpoint directory has been set; otherwise the method fails with an exception.

参数:

jobId - The ID of the job

返回:

The job's checkpoint directory, re

抛出:

UnsupportedOperationException - Thrown, if no base checkpoint directory has been set.

createCheckpointDirectory

protected static org.apache.flink.core.fs.Path createCheckpointDirectory(org.apache.flink.core.fs.Path baseDirectory,
                                                                         long checkpointId)

Creates the directory path for the data exclusive to a specific checkpoint.

参数:

baseDirectory - The base directory into which the job checkpoints.

checkpointId - The ID (logical timestamp) of the checkpoint.

resolveCheckpointPointer

@Internal
public static FsCompletedCheckpointStorageLocation resolveCheckpointPointer(String checkpointPointer)
                                                                               throws IOException

Takes the given string (representing a pointer to a checkpoint) and resolves it to a file status for the checkpoint's metadata file.

参数:

checkpointPointer - The pointer to resolve.

返回:

A state handle to checkpoint/savepoint's metadata.

抛出:

IOException - Thrown, if the pointer cannot be resolved, the file system not accessed, or the pointer points to a location that does not seem to be a checkpoint/savepoint.

encodePathAsReference

public static CheckpointStorageLocationReference encodePathAsReference(org.apache.flink.core.fs.Path path)

Encodes the given path as a reference in bytes. The path is encoded as a UTF-8 string and prepended as a magic number.

参数:

path - The path to encode.

返回:

The location reference.

decodePathFromReference

public static org.apache.flink.core.fs.Path decodePathFromReference(CheckpointStorageLocationReference reference)

Decodes the given reference into a path. This method validates that the reference bytes start with the correct magic number (as written by encodePathAsReference(Path)) and converts the remaining bytes back to a proper path.

参数:

reference - The bytes representing the reference.

返回:

The path decoded from the reference.

抛出:

IllegalArgumentException - Thrown, if the bytes do not represent a proper reference.