org.apache.flink.runtime.fs.s3

Class S3FileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.s3.S3FileSystem

public final class S3FileSystem
extends FileSystem

This class provides a FileSystem implementation which is backed by Amazon's Simple Storage Service (S3). The implementation uses the REST API of Amazon S3 to facilitate the communication and read/write the data.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

FileSystem.FSKey, FileSystem.WriteMode

Field Summary

Fields

Modifier and Type	Field and Description
static String	S3_ACCESS_KEY_KEY The configuration key to access the S3 access key.
static String	S3_HOST_KEY The configuration key to access the S3 host.
static String	S3_PORT_KEY The configuration key to access the S3 port.
static String	S3_RRS_KEY The configuration key to access the S3 Reduced Redundancy Storage setting.
static String	S3_SCHEME The scheme which is used by this file system.
static String	S3_SECRET_KEY_KEY The configuration key to access the S3 secret key.

Constructor Summary

Constructors

Constructor and Description
S3FileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
FSDataOutputStream	create(Path f, boolean overwrite)
FSDataOutputStream	create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
boolean	delete(Path f, boolean recursive)
BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len)
FileStatus	getFileStatus(Path f)
URI	getUri()
Path	getWorkingDirectory()
void	initialize(URI name)
boolean	isDistributedFS()
FileStatus[]	listStatus(Path f)
boolean	mkdirs(Path f)
FSDataInputStream	open(Path f)
FSDataInputStream	open(Path f, int bufferSize)
boolean	rename(Path src, Path dst)

Methods inherited from class org.apache.flink.core.fs.FileSystem

exists, get, getDefaultBlockSize, getLocalFileSystem, getNumberOfBlocks, initOutPathDistFS, initOutPathLocalFS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3_HOST_KEY

public static final String S3_HOST_KEY

The configuration key to access the S3 host.

See Also:

Constant Field Values

S3_PORT_KEY

public static final String S3_PORT_KEY

The configuration key to access the S3 port.

See Also:

Constant Field Values

S3_RRS_KEY

public static final String S3_RRS_KEY

The configuration key to access the S3 Reduced Redundancy Storage setting.

See Also:

Constant Field Values

S3_ACCESS_KEY_KEY

public static final String S3_ACCESS_KEY_KEY

The configuration key to access the S3 access key.

See Also:

Constant Field Values

S3_SECRET_KEY_KEY

public static final String S3_SECRET_KEY_KEY

The configuration key to access the S3 secret key.

See Also:

Constant Field Values

S3_SCHEME

public static final String S3_SCHEME

The scheme which is used by this file system.

See Also:

Constant Field Values

Constructor Detail

S3FileSystem

public S3FileSystem()

Method Detail

getWorkingDirectory

public Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class FileSystem

getUri

public URI getUri()

Specified by:

getUri in class FileSystem

initialize

public void initialize(URI name)
                throws IOException

Specified by:

initialize in class FileSystem

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Specified by:

getFileStatus in class FileSystem

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus file,
                                    long start,
                                    long len)
                                      throws IOException

Specified by:

getFileBlockLocations in class FileSystem

Throws:

IOException

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

open

public FSDataInputStream open(Path f)
                       throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Specified by:

listStatus in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException

Specified by:

delete in class FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(Path f)
               throws IOException

Specified by:

mkdirs in class FileSystem

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                        boolean overwrite,
                        int bufferSize,
                        short replication,
                        long blockSize)
                          throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                        boolean overwrite)
                          throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

rename

public boolean rename(Path src,
             Path dst)
               throws IOException

Specified by:

rename in class FileSystem

Throws:

IOException

isDistributedFS

public boolean isDistributedFS()

Specified by:

isDistributedFS in class FileSystem