io.pravega.storage.extendeds3

Class ExtendedS3Storage

java.lang.Object

io.pravega.storage.extendeds3.ExtendedS3Storage

All Implemented Interfaces:

io.pravega.segmentstore.storage.SyncStorage, java.lang.AutoCloseable

public class ExtendedS3Storage
extends java.lang.Object
implements io.pravega.segmentstore.storage.SyncStorage

Storage adapter for extended S3 based storage. Each segment is represented as a single Object on the underlying storage. This implementation works under the assumption that data is only appended and never modified. Each block of data has an initial offset assigned to it. The data and the initial offset is stored in DurableLog. In case of retries, Pravega always writes the same data to the same offset. As a result the only flow when a write call is made to the same offset twice is when ownership of the segment changes from one host to another and both the hosts are writing to it. As PutObject calls with the same start-offset to an Extended S3 object are idempotent (any attempt to re-write data with the same file offset does not cause any form of inconsistency), fencing is not required. ZkSegmentContainerMonitor watches the shared zk entry that contains the segment container ownership information and starts or stops appropriate segment containers locally. Any access to the segment from the new host causes the ownership change. Here is the expected behavior in case of ownership change: both the hosts will keep writing the same data at the same offset till the time the earlier owner gets a notification that it is not the current owner. Once the earlier owner received this notification, it stops writing to the segment. The concat operation is implemented as multi part copy. This ensures that the objects are copied server side. Multi part copy calls are idempotent too. Copying the same object at the same offset multiple times from different hosts does not cause any form of inconsistency.

Constructor Summary

Constructors

Constructor and Description
ExtendedS3Storage(com.emc.object.s3.S3Client client, ExtendedS3StorageConfig config)

Method Summary

Modifier and Type	Method and Description
void	close()
void	concat(io.pravega.segmentstore.storage.SegmentHandle targetHandle, long offset, java.lang.String sourceSegment)
io.pravega.segmentstore.storage.SegmentHandle	create(java.lang.String streamSegmentName)
void	delete(io.pravega.segmentstore.storage.SegmentHandle handle)
boolean	exists(java.lang.String streamSegmentName)
io.pravega.segmentstore.contracts.SegmentProperties	getStreamSegmentInfo(java.lang.String streamSegmentName)
void	initialize(long containerEpoch) Initialize is a no op here as we do not need a locking mechanism in case of file system write.
io.pravega.segmentstore.storage.SegmentHandle	openRead(java.lang.String streamSegmentName)
io.pravega.segmentstore.storage.SegmentHandle	openWrite(java.lang.String streamSegmentName)
int	read(io.pravega.segmentstore.storage.SegmentHandle handle, long offset, byte[] buffer, int bufferOffset, int length)
void	seal(io.pravega.segmentstore.storage.SegmentHandle handle)
boolean	supportsTruncation()
void	truncate(io.pravega.segmentstore.storage.SegmentHandle handle, long offset)
void	unseal(io.pravega.segmentstore.storage.SegmentHandle handle)
void	write(io.pravega.segmentstore.storage.SegmentHandle handle, long offset, java.io.InputStream data, int length)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.pravega.segmentstore.storage.SyncStorage

create

Constructor Detail

ExtendedS3Storage

public ExtendedS3Storage(com.emc.object.s3.S3Client client,
                         ExtendedS3StorageConfig config)

Method Detail

initialize

public void initialize(long containerEpoch)

Initialize is a no op here as we do not need a locking mechanism in case of file system write.

Specified by:

initialize in interface io.pravega.segmentstore.storage.SyncStorage

Parameters:

containerEpoch - The Container Epoch to initialize with (ignored here).

openRead

public io.pravega.segmentstore.storage.SegmentHandle openRead(java.lang.String streamSegmentName)
                                                       throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

openRead in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

read

public int read(io.pravega.segmentstore.storage.SegmentHandle handle,
                long offset,
                byte[] buffer,
                int bufferOffset,
                int length)
         throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

read in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

getStreamSegmentInfo

public io.pravega.segmentstore.contracts.SegmentProperties getStreamSegmentInfo(java.lang.String streamSegmentName)
                                                                         throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

getStreamSegmentInfo in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

exists

public boolean exists(java.lang.String streamSegmentName)

Specified by:

exists in interface io.pravega.segmentstore.storage.SyncStorage

openWrite

public io.pravega.segmentstore.storage.SegmentHandle openWrite(java.lang.String streamSegmentName)
                                                        throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

openWrite in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

create

public io.pravega.segmentstore.storage.SegmentHandle create(java.lang.String streamSegmentName)
                                                     throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

create in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

write

public void write(io.pravega.segmentstore.storage.SegmentHandle handle,
                  long offset,
                  java.io.InputStream data,
                  int length)
           throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

write in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

seal

public void seal(io.pravega.segmentstore.storage.SegmentHandle handle)
          throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

seal in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

unseal

public void unseal(io.pravega.segmentstore.storage.SegmentHandle handle)
            throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

unseal in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

concat

public void concat(io.pravega.segmentstore.storage.SegmentHandle targetHandle,
                   long offset,
                   java.lang.String sourceSegment)
            throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

concat in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

delete

public void delete(io.pravega.segmentstore.storage.SegmentHandle handle)
            throws io.pravega.segmentstore.contracts.StreamSegmentException

Specified by:

delete in interface io.pravega.segmentstore.storage.SyncStorage

Throws:

io.pravega.segmentstore.contracts.StreamSegmentException

truncate

public void truncate(io.pravega.segmentstore.storage.SegmentHandle handle,
                     long offset)

Specified by:

truncate in interface io.pravega.segmentstore.storage.SyncStorage

supportsTruncation

public boolean supportsTruncation()

Specified by:

supportsTruncation in interface io.pravega.segmentstore.storage.SyncStorage

close

public void close()

Specified by:

close in interface io.pravega.segmentstore.storage.SyncStorage

Specified by:

close in interface java.lang.AutoCloseable