Package cz.o2.proxima.flink.core.batch

Class OffsetTrackingBatchLogReader

java.lang.Object

cz.o2.proxima.flink.core.batch.OffsetTrackingBatchLogReader

All Implemented Interfaces:

BatchLogReader

@Experimental("API can be changed.")
public class OffsetTrackingBatchLogReader
extends java.lang.Object
implements BatchLogReader

A simple wrapper for batch log reader, that is able to track the "highest" consumed offset for each partition. For offset to be consider as consumed, you must call OffsetTrackingBatchLogReader.OffsetCommitter.markOffsetAsConsumed() first.

Unfortunately we can not simply extend BatchLogObserver.OnNextContext interface, therefore you need to cast it to OffsetTrackingBatchLogReader.OffsetCommitter for marking offset as consumed.

 new BatchLogObserver() {

   @Override
   public boolean onNext(StreamElement element, OnNextContext context) {
     final OffsetTrackingBatchLogReader.OffsetCommitter committer =
         (OffsetTrackingBatchLogReader.OffsetCommitter) context;
     committer.markOffsetAsConsumed();
     return true;
   }
 }
 

Now you can simply retrieve consumed offset by casting ObserveHandle to OffsetTrackingBatchLogReader.OffsetTrackingObserveHandle.

 final OffsetTrackingBatchLogReader.OffsetTrackingObserveHandle tracker =
   (OffsetTrackingBatchLogReader.OffsetTrackingObserveHandle) handle;
 tracker.getCurrentOffsets();
 

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	OffsetTrackingBatchLogReader.OffsetCommitter
static class	OffsetTrackingBatchLogReader.OffsetTrackingBatchLogObserver	Log observer, that keeps track of the "highest" consumed offset for each partition.
static interface	OffsetTrackingBatchLogReader.OffsetTrackingObserveHandle	Batch observe handle, that has access to consumed offsets.

Nested classes/interfaces inherited from interface cz.o2.proxima.direct.core.batch.BatchLogReader

BatchLogReader.Factory<T extends BatchLogReader>

Constructor Summary

Constructors

Constructor	Description
OffsetTrackingBatchLogReader(BatchLogReader delegate)

Method Summary

Modifier and Type	Method	Description
BatchLogReader.Factory<?>	asFactory()	Convert instance of this reader to BatchLogReader.Factory suitable for serialization.
java.util.List<Partition>	getPartitions(long startStamp, long endStamp)	Retrieve list of partitions covering data from the given range.
ObserveHandle	observe(java.util.List<Partition> partitions, java.util.List<AttributeDescriptor<?>> attributes, BatchLogObserver observer)	Observe data stored in given partitions.
static OffsetTrackingBatchLogReader	of(BatchLogReader reader)	Wrap batch log reader into OffsetTrackingBatchLogReader.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cz.o2.proxima.direct.core.batch.BatchLogReader

getPartitions, getPartitions, observeOffsets

Constructor Detail

OffsetTrackingBatchLogReader

public OffsetTrackingBatchLogReader(BatchLogReader delegate)

Method Detail

of

public static OffsetTrackingBatchLogReader of(BatchLogReader reader)

Wrap batch log reader into OffsetTrackingBatchLogReader.

Parameters:

reader - Log reader to wrap.

Returns:

Wrapped reader.

getPartitions

public java.util.List<Partition> getPartitions(long startStamp,
                                               long endStamp)

Description copied from interface: BatchLogReader

Retrieve list of partitions covering data from the given range.

Specified by:

getPartitions in interface BatchLogReader

Parameters:

startStamp - starting timestamp (inclusive)

endStamp - ending timestamp (exclusive)

Returns:

list of partitions covering the time range

observe

public ObserveHandle observe(java.util.List<Partition> partitions,
                             java.util.List<AttributeDescriptor<?>> attributes,
                             BatchLogObserver observer)

Description copied from interface: BatchLogReader

Observe data stored in given partitions.

Specified by:

observe in interface BatchLogReader

Parameters:

partitions - partitions to observe

attributes - attribute descriptors to filter out

observer - the observer by which to consume the data

Returns:

an ObserveHandle to control the consumption

asFactory

public BatchLogReader.Factory<?> asFactory()

Description copied from interface: BatchLogReader

Convert instance of this reader to BatchLogReader.Factory suitable for serialization.

Specified by:

asFactory in interface BatchLogReader

Returns:

the BatchLogReader.Factory representing this reader