org.apache.flink.runtime.checkpoint.filemerging

Interface FileMergingSnapshotManager

All Superinterfaces:

AutoCloseable, Closeable

All Known Implementing Classes:

FileMergingSnapshotManagerBase, WithinCheckpointFileMergingSnapshotManager

public interface FileMergingSnapshotManager
extends Closeable

FileMergingSnapshotManager provides an interface to manage files and meta information for checkpoint files with merging checkpoint files enabled. It manages the files for ONE single task in TM, including all subtasks of this single task that running in this TM. There is one FileMergingSnapshotManager for each task per task manager.

TODO (FLINK-32073): create output stream.

TODO (FLINK-32075): leverage checkpoint notification to delete logical files.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	FileMergingSnapshotManager.SubtaskKey A key identifies a subtask.

Method Summary

Modifier and Type	Method and Description
org.apache.flink.core.fs.Path	getManagedDir(FileMergingSnapshotManager.SubtaskKey subtaskKey, CheckpointedStateScope scope) Get the managed directory of the file-merging snapshot manager, created in initFileSystem(org.apache.flink.core.fs.FileSystem, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path) or registerSubtaskForSharedStates(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey).
void	initFileSystem(org.apache.flink.core.fs.FileSystem fileSystem, org.apache.flink.core.fs.Path checkpointBaseDir, org.apache.flink.core.fs.Path sharedStateDir, org.apache.flink.core.fs.Path taskOwnedStateDir) Initialize the file system, recording the checkpoint path the manager should work with.
void	registerSubtaskForSharedStates(FileMergingSnapshotManager.SubtaskKey subtaskKey) Register a subtask and create the managed directory for shared states.

Methods inherited from interface java.io.Closeable

close

Method Detail

initFileSystem

void initFileSystem(org.apache.flink.core.fs.FileSystem fileSystem,
                    org.apache.flink.core.fs.Path checkpointBaseDir,
                    org.apache.flink.core.fs.Path sharedStateDir,
                    org.apache.flink.core.fs.Path taskOwnedStateDir)
             throws IllegalArgumentException

Initialize the file system, recording the checkpoint path the manager should work with.

 The layout of checkpoint directory:
 /user-defined-checkpoint-dir
     /{job-id} (checkpointBaseDir)
         |
         + --shared/
             |
             + --subtask-1/
                 + -- merged shared state files
             + --subtask-2/
                 + -- merged shared state files
         + --taskowned/
             + -- merged private state files
         + --chk-1/
         + --chk-2/
         + --chk-3/
 

The reason why initializing directories in this method instead of the constructor is that the FileMergingSnapshotManager itself belongs to the TaskStateManager, which is initialized when receiving a task, while the base directories for checkpoint are created by FsCheckpointStorageAccess when the state backend initializing per subtask. After the checkpoint directories are initialized, the managed subdirectories are initialized here.

Note: This method may be called several times, the implementation should ensure idempotency, and throw IllegalArgumentException when any of the path in params change across function calls.

Parameters:

fileSystem - The filesystem to write to.

checkpointBaseDir - The base directory for checkpoints.

sharedStateDir - The directory for shared checkpoint data.

taskOwnedStateDir - The name of the directory for state not owned/released by the master, but by the TaskManagers.

Throws:

IllegalArgumentException - thrown if these three paths are not deterministic across calls.

registerSubtaskForSharedStates

void registerSubtaskForSharedStates(FileMergingSnapshotManager.SubtaskKey subtaskKey)

Register a subtask and create the managed directory for shared states.

Parameters:

subtaskKey - the subtask key identifying a subtask.

See Also:

for layout information.

getManagedDir

org.apache.flink.core.fs.Path getManagedDir(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                            CheckpointedStateScope scope)

Get the managed directory of the file-merging snapshot manager, created in initFileSystem(org.apache.flink.core.fs.FileSystem, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path) or registerSubtaskForSharedStates(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey).

Parameters:

subtaskKey - the subtask key identifying the subtask.

scope - the checkpoint scope.

Returns:

the managed directory for one subtask in specified checkpoint scope.