Package cz.o2.proxima.beam.tools.groovy

Class BeamStreamProvider

java.lang.Object

cz.o2.proxima.beam.tools.groovy.BeamStreamProvider

All Implemented Interfaces:

StreamProvider, java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

BeamStreamProvider.Default

public abstract class BeamStreamProvider
extends java.lang.Object
implements StreamProvider

A StreamProvider for groovy tools based on beam.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	BeamStreamProvider.Default
static interface	BeamStreamProvider.RunnerRegistrar

Nested classes/interfaces inherited from interface cz.o2.proxima.tools.groovy.StreamProvider

StreamProvider.TerminatePredicate

Constructor Summary

Constructors

Constructor	Description
BeamStreamProvider()

Method Summary

Modifier and Type	Method	Description
void	close()	Close and release all resources.
WindowedStream<StreamElement>	getBatchSnapshot(long fromStamp, long toStamp, StreamProvider.TerminatePredicate terminateCheck, AttributeDescriptor<?>... attrs)	Retrieve batch snapshot stream.
WindowedStream<StreamElement>	getBatchUpdates(long startStamp, long endStamp, StreamProvider.TerminatePredicate terminateCheck, AttributeDescriptor<?>... attrs)	Retrieve batch updates stream.
protected UnaryFunction<org.apache.beam.sdk.options.PipelineOptions,org.apache.beam.sdk.Pipeline>	getCreatePipelineFromOpts()	Convert PipelineOptions into Pipeline.
protected java.util.function.Supplier<org.apache.beam.sdk.options.PipelineOptions>	getPipelineOptionsFactory()	Create factory to be used for pipeline creation.
Stream<StreamElement>	getStream(Position position, boolean stopAtCurrent, boolean eventTime, StreamProvider.TerminatePredicate terminateCheck, AttributeDescriptor<?>... attrs)	Create stream from commit log(s).
<T> WindowedStream<T>	impulse(java.lang.String name, groovy.lang.Closure<T> factory)	Return new WindowedStream that contains single element generated by calling given (parameterless) Closure.
void	init(Repository repo, java.lang.String[] args)	Initialize the provider with given repository.
<T> WindowedStream<T>	periodicImpulse(java.lang.String name, groovy.lang.Closure<T> factory, long durationMs)	Create new WindowedStream that contains single element per fixed time windowed.
static <T> Stream<T>	wrap(Repository repo, org.apache.beam.sdk.values.PCollection<T> pCollection)	Create a Stream from given PCollection.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cz.o2.proxima.tools.groovy.StreamProvider

impulse, periodicImpulse

Constructor Detail

BeamStreamProvider

public BeamStreamProvider()

Method Detail

wrap

public static <T> Stream<T> wrap(Repository repo,
                                 org.apache.beam.sdk.values.PCollection<T> pCollection)

Create a Stream from given PCollection.

Parameters:

repo - Repository to use

pCollection - the PCollection to wrap

Returns:

PCollection wrapped as Stream.

init

public void init(Repository repo,
                 java.lang.String[] args)

Description copied from interface: StreamProvider

Initialize the provider with given repository.

Specified by:

init in interface StreamProvider

Parameters:

repo - the repository

args - command line arguments passed to Console

getStream

public Stream<StreamElement> getStream(Position position,
                                       boolean stopAtCurrent,
                                       boolean eventTime,
                                       StreamProvider.TerminatePredicate terminateCheck,
                                       AttributeDescriptor<?>... attrs)

Description copied from interface: StreamProvider

Create stream from commit log(s).

Specified by:

getStream in interface StreamProvider

Parameters:

position - position in commit log

stopAtCurrent - true to stop at current data

eventTime - true to process using event time

terminateCheck - Predicate that tests if the execution of any terminal operation should be interrupted

attrs - attributes to get stream for

Returns:

stream from commit log

getBatchUpdates

public WindowedStream<StreamElement> getBatchUpdates(long startStamp,
                                                     long endStamp,
                                                     StreamProvider.TerminatePredicate terminateCheck,
                                                     AttributeDescriptor<?>... attrs)

Description copied from interface: StreamProvider

Retrieve batch updates stream.

Specified by:

getBatchUpdates in interface StreamProvider

Parameters:

startStamp - starting stamp (inclusive)

endStamp - ending stamp (exclusive)

terminateCheck - Predicate that tests if the execution of any terminal operation should be interrupted

attrs - attributes to read

Returns:

globally windowed stream

getBatchSnapshot

public WindowedStream<StreamElement> getBatchSnapshot(long fromStamp,
                                                      long toStamp,
                                                      StreamProvider.TerminatePredicate terminateCheck,
                                                      AttributeDescriptor<?>... attrs)

Description copied from interface: StreamProvider

Retrieve batch snapshot stream.

Specified by:

getBatchSnapshot in interface StreamProvider

Parameters:

fromStamp - starting stamp (inclusive)

toStamp - ending stamp (exclusive)

terminateCheck - Predicate that tests if the execution of any terminal operation should be interrupted

attrs - attributes to read

Returns:

globally windowed stream

impulse

public <T> WindowedStream<T> impulse(java.lang.String name,
                                     groovy.lang.Closure<T> factory)

Description copied from interface: StreamProvider

Return new WindowedStream that contains single element generated by calling given (parameterless) Closure.

Specified by:

impulse in interface StreamProvider

Type Parameters:

T - type of the resulting stream

Parameters:

name - name of the transform

factory - the closure that generates the element

Returns:

globally windowed stream with single element

periodicImpulse

public <T> WindowedStream<T> periodicImpulse(java.lang.String name,
                                             groovy.lang.Closure<T> factory,
                                             long durationMs)

Description copied from interface: StreamProvider

Create new WindowedStream that contains single element per fixed time windowed.

Specified by:

periodicImpulse in interface StreamProvider

Type Parameters:

T - type of the resulting stream

Parameters:

name - name of the transform

factory - the closure that generates the element

durationMs - duration of the fixed time window in ms

Returns:

globally windowed stream with single element

close

public void close()

Description copied from interface: StreamProvider

Close and release all resources.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface StreamProvider

getPipelineOptionsFactory

protected java.util.function.Supplier<org.apache.beam.sdk.options.PipelineOptions> getPipelineOptionsFactory()

Create factory to be used for pipeline creation.

Returns:

the factory

getCreatePipelineFromOpts

protected UnaryFunction<org.apache.beam.sdk.options.PipelineOptions,org.apache.beam.sdk.Pipeline> getCreatePipelineFromOpts()

Convert PipelineOptions into Pipeline.

Returns:

function to use for creating pipeline from options