Package cz.o2.proxima.direct.core.commitlog

Interface CommitLogReader

All Known Implementing Classes:

CommitLogReaders.LimitedCommitLogReader, KafkaLogReader, PubSubBulkReader, WebsocketReader

@Stable
public interface CommitLogReader

Read access to commit log. The commit log is read by registering a (possible) named observer on the stream. If the observer is named then if multiple registration exist with the same name, it is automatically load balanced.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static interface	CommitLogReader.Factory<T extends CommitLogReader>	Serializable factory for CommitLogReader.

Method Summary

Modifier and Type	Method	Description
CommitLogReader.Factory<?>	asFactory()	Convert instance of this reader to CommitLogReader.Factory suitable for serialization.
default java.util.Map<Partition,Offset>	fetchOffsets(Position position, java.util.List<Partition> partitions)	Fetch offsets at given position from given partitions.
default OffsetExternalizer	getOffsetExternalizer()	Provides offset externalizer for serializing log specific offsets to external formats (bytes, JSON).
java.util.List<Partition>	getPartitions()	Retrieve list of partitions of this commit log.
java.net.URI	getUri()	Retrieve URI representing this resource.
default boolean	hasExternalizableOffsets()	Signals the user that offsets used by this reader can be externalized and reused later.
ObserveHandle	observe(java.lang.String name, Position position, CommitLogObserver observer)	Subscribe observer by name to the commit log.
default ObserveHandle	observe(java.lang.String name, CommitLogObserver observer)	Subscribe observer by name to the commit log and read the newest data.
ObserveHandle	observeBulk(java.lang.String name, Position position, boolean stopAtCurrent, CommitLogObserver observer)	Subscribe to the commit log in a bulk fashion.
default ObserveHandle	observeBulk(java.lang.String name, Position position, CommitLogObserver observer)	Subscribe to the commit log in a bulk fashion.
default ObserveHandle	observeBulk(java.lang.String name, CommitLogObserver observer)	Subscribe to the commit log in a bulk fashion from newest data.
ObserveHandle	observeBulkOffsets(java.util.Collection<Offset> offsets, boolean stopAtCurrent, CommitLogObserver observer)	Consume from given offsets in a bulk fashion.
default ObserveHandle	observeBulkOffsets(java.util.Collection<Offset> offsets, CommitLogObserver observer)	Consume from given offsets in a bulk fashion.
ObserveHandle	observeBulkPartitions(java.lang.String name, java.util.Collection<Partition> partitions, Position position, boolean stopAtCurrent, CommitLogObserver observer)	Subscribe to given partitions in a bulk fashion.
default ObserveHandle	observeBulkPartitions(java.lang.String name, java.util.Collection<Partition> partitions, Position position, CommitLogObserver observer)	Subscribe to given partitions in a bulk fashion.
default ObserveHandle	observeBulkPartitions(java.util.Collection<Partition> partitions, Position position, boolean stopAtCurrent, CommitLogObserver observer)	Subscribe to given partitions in a bulk fashion.
default ObserveHandle	observeBulkPartitions(java.util.Collection<Partition> partitions, Position position, CommitLogObserver observer)	Subscribe to given partitions in a bulk fashion.
ObserveHandle	observePartitions(java.lang.String name, java.util.Collection<Partition> partitions, Position position, boolean stopAtCurrent, CommitLogObserver observer)	Subscribe to given set of partitions.
default ObserveHandle	observePartitions(java.util.Collection<Partition> partitions, Position position, boolean stopAtCurrent, CommitLogObserver observer)	Subscribe to given set of partitions.
default ObserveHandle	observePartitions(java.util.Collection<Partition> partitions, Position position, CommitLogObserver observer)	Subscribe to given set of partitions.
default ObserveHandle	observePartitions(java.util.Collection<Partition> partitions, CommitLogObserver observer)	Subscribe to given set of partitions and read newest data.
default boolean	restoresSequentialIds()	Signals the user that this CommitLogReader is able to recover sequential IDs sent to it in StreamElements through its associated writer.

Method Detail

getUri

java.net.URI getUri()

Retrieve URI representing this resource.

Returns:

URI representing this resource

getPartitions

java.util.List<Partition> getPartitions()

Retrieve list of partitions of this commit log.

Returns:

list of partitions of this reader

observe

ObserveHandle observe(java.lang.String name,
                      Position position,
                      CommitLogObserver observer)

Subscribe observer by name to the commit log. Each observer maintains its own position in the commit log, so that the observers with different names do not interfere If multiple observers share the same name, then the ingests are load-balanced between them (in an undefined manner). This is a non blocking call.

Parameters:

name - identifier of the consumer

position - the position to seek for in the commit log

observer - the observer to subscribe to the commit log

Returns:

ObserveHandle to asynchronously cancel the observation

observe

default ObserveHandle observe(java.lang.String name,
                              CommitLogObserver observer)

Subscribe observer by name to the commit log and read the newest data. Each observer maintains its own position in the commit log, so that the observers with different names do not interfere If multiple observers share the same name, then the ingests are load-balanced between them (in an undefined manner). This is a non blocking call.

Parameters:

name - identifier of the consumer

observer - the observer to subscribe to the commit log

Returns:

ObserveHandle to asynchronously cancel the observation

observePartitions

ObserveHandle observePartitions(java.lang.String name,
                                java.util.Collection<Partition> partitions,
                                Position position,
                                boolean stopAtCurrent,
                                CommitLogObserver observer)

Subscribe to given set of partitions. If you use this call then the reader stops being automatically load balanced and the set of partitions can only be changed by call to this method again.

Parameters:

name - name of the observer

partitions - the list of partitions to subscribe to

position - the position to seek to in the partitions

stopAtCurrent - when true then stop the observer as soon as it reaches most recent record

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observePartitions

default ObserveHandle observePartitions(java.util.Collection<Partition> partitions,
                                        Position position,
                                        boolean stopAtCurrent,
                                        CommitLogObserver observer)

Subscribe to given set of partitions. If you use this call then the reader stops being automatically load balanced and the set of partitions can only be changed by call to this method again.

Parameters:

partitions - the list of partitions to subscribe to

position - the position to seek to in the partitions

stopAtCurrent - when true then stop the observer as soon as it reaches most recent record

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observePartitions

default ObserveHandle observePartitions(java.util.Collection<Partition> partitions,
                                        Position position,
                                        CommitLogObserver observer)

Subscribe to given set of partitions. If you use this call then the reader stops being automatically load balanced and the set of partitions can only be changed by call to this method again.

Parameters:

partitions - the list of partitions to subscribe to

position - the position to seek to in the partitions

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observePartitions

default ObserveHandle observePartitions(java.util.Collection<Partition> partitions,
                                        CommitLogObserver observer)

Subscribe to given set of partitions and read newest data. If you use this call then the reader stops being automatically load balanced and the set of partitions can only be changed by call to this method again.

Parameters:

partitions - the partitions to subscribe to

observer - the observer to subscribe to the given partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulk

ObserveHandle observeBulk(java.lang.String name,
                          Position position,
                          boolean stopAtCurrent,
                          CommitLogObserver observer)

Subscribe to the commit log in a bulk fashion. That implies that elements are not committed one-by-one, but in a bulks, where all elements in a bulk are committed at once. This is useful for micro-batching approach of data processing.

Parameters:

name - name of the observer

position - the position to seek to in the partitions

stopAtCurrent - when true then stop the observer as soon as it reaches most recent record

observer - the observer to subscribe

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulk

default ObserveHandle observeBulk(java.lang.String name,
                                  Position position,
                                  CommitLogObserver observer)

Subscribe to the commit log in a bulk fashion. That implies that elements are not committed one-by-one, but in a bulks, where all elements in a bulk are committed at once. This is useful for micro-batching approach of data processing.

Parameters:

name - name of the observer

position - the position to seek to in the partitions

observer - the observer to subscribe

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulk

default ObserveHandle observeBulk(java.lang.String name,
                                  CommitLogObserver observer)

Subscribe to the commit log in a bulk fashion from newest data. That implies that elements are not committed one-by-one, but in a bulks, where all elements in a bulk are committed at once. This is useful for micro-batching approach of data processing.

Parameters:

name - name of the observer

observer - the observer to subscribe

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkPartitions

default ObserveHandle observeBulkPartitions(java.lang.String name,
                                            java.util.Collection<Partition> partitions,
                                            Position position,
                                            CommitLogObserver observer)

Subscribe to given partitions in a bulk fashion.

Parameters:

name - name of the observer

partitions - the partitions to subscribe to

position - the position to seek to in the partitions

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkPartitions

ObserveHandle observeBulkPartitions(java.lang.String name,
                                    java.util.Collection<Partition> partitions,
                                    Position position,
                                    boolean stopAtCurrent,
                                    CommitLogObserver observer)

Subscribe to given partitions in a bulk fashion.

Parameters:

name - name of the observer

partitions - the partitions to subscribe to

position - the position to seek to in the partitions

stopAtCurrent - when true then stop the observer as soon as it reaches most recent record

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkPartitions

default ObserveHandle observeBulkPartitions(java.util.Collection<Partition> partitions,
                                            Position position,
                                            boolean stopAtCurrent,
                                            CommitLogObserver observer)

Subscribe to given partitions in a bulk fashion.

Parameters:

partitions - the partitions to subscribe to

position - the position to seek to in the partitions

stopAtCurrent - when true then stop the observer as soon as it reaches most recent record

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkPartitions

default ObserveHandle observeBulkPartitions(java.util.Collection<Partition> partitions,
                                            Position position,
                                            CommitLogObserver observer)

Subscribe to given partitions in a bulk fashion.

Parameters:

partitions - the partitions to subscribe to

position - the position to seek to in the partitions

observer - the observer to subscribe to the partitions

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkOffsets

default ObserveHandle observeBulkOffsets(java.util.Collection<Offset> offsets,
                                         CommitLogObserver observer)

Consume from given offsets in a bulk fashion. A typical use-case for this type of consumption is to first use observeBulkPartitions(java.lang.String, java.util.Collection<cz.o2.proxima.core.storage.Partition>, cz.o2.proxima.core.storage.commitlog.Position, cz.o2.proxima.direct.core.commitlog.CommitLogObserver), observe for some time, than interrupt the consumption, store associated offsets and resume the consumption from these offsets later

Parameters:

offsets - the @{link Offset}s to subscribe to

observer - the observer to subscribe to the offsets

Returns:

ObserveHandle to asynchronously cancel the observation

observeBulkOffsets

ObserveHandle observeBulkOffsets(java.util.Collection<Offset> offsets,
                                 boolean stopAtCurrent,
                                 CommitLogObserver observer)

Consume from given offsets in a bulk fashion. A typical use-case for this type of consumption is to first use observeBulkPartitions(java.lang.String, java.util.Collection<cz.o2.proxima.core.storage.Partition>, cz.o2.proxima.core.storage.commitlog.Position, cz.o2.proxima.direct.core.commitlog.CommitLogObserver), observe for some time, than interrupt the consumption, store associated offsets and resume the consumption from these offsets later

Parameters:

offsets - the @{link Offset}s to subscribe to

stopAtCurrent - true if the processing should stop at current offsets

observer - the observer to subscribe to the offsets

Returns:

ObserveHandle to asynchronously cancel the observation

hasExternalizableOffsets

default boolean hasExternalizableOffsets()

Signals the user that offsets used by this reader can be externalized and reused later.

Returns:

true if Offsets of this reader are externalizable

getOffsetExternalizer

default OffsetExternalizer getOffsetExternalizer()

Provides offset externalizer for serializing log specific offsets to external formats (bytes, JSON).

Returns:

OffsetExternalizer when externalizable offsets are supported by this reader, check it by hasExternalizableOffsets().

restoresSequentialIds

default boolean restoresSequentialIds()

Signals the user that this CommitLogReader is able to recover sequential IDs sent to it in StreamElements through its associated writer.

Returns:

true if StreamElement read from the commit log contain valid sequential IDs.

fetchOffsets

default java.util.Map<Partition,Offset> fetchOffsets(Position position,
                                                           java.util.List<Partition> partitions)

Fetch offsets at given position from given partitions.

Parameters:

position - position of the offsets

partitions - partitions for which to fetch the offsets

Throws:

java.lang.UnsupportedOperationException - if CommitLogReader does not have externalizable offsets

asFactory

CommitLogReader.Factory<?> asFactory()

Convert instance of this reader to CommitLogReader.Factory suitable for serialization.

Returns:

the CommitLogReader.Factory representing this reader