Package cz.o2.proxima.beam.core.direct.io

Class DirectBatchUnboundedSource

java.lang.Object

org.apache.beam.sdk.io.Source<OutputT>

org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>

cz.o2.proxima.beam.core.direct.io.DirectBatchUnboundedSource

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class DirectBatchUnboundedSource
extends org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>

Source reading from BatchLogReader in unbounded manner.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DirectBatchUnboundedSource.Checkpoint
static class	DirectBatchUnboundedSource.CheckpointCoder	Use gzip to compress the serialized checkpoint, as it might easily grow in size (batch partitions might contain many files).

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.UnboundedSource

org.apache.beam.sdk.io.UnboundedSource.CheckpointMark, org.apache.beam.sdk.io.UnboundedSource.UnboundedReader<OutputT extends java.lang.Object>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

org.apache.beam.sdk.io.Source.Reader<T extends java.lang.Object>

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	CFG_ENABLE_CHECKPOINT_PARTITION_MERGE

Method Summary

Modifier and Type	Method	Description
org.apache.beam.sdk.io.UnboundedSource.UnboundedReader<StreamElement>	createReader(org.apache.beam.sdk.options.PipelineOptions options, DirectBatchUnboundedSource.Checkpoint checkpointMark)
boolean	equals(java.lang.Object o)
org.apache.beam.sdk.coders.Coder<DirectBatchUnboundedSource.Checkpoint>	getCheckpointMarkCoder()
org.apache.beam.sdk.coders.Coder<StreamElement>	getOutputCoder()
int	hashCode()
static DirectBatchUnboundedSource	of(RepositoryFactory factory, BatchLogReader reader, java.util.List<AttributeDescriptor<?>> attrs, long startStamp, long endStamp, java.util.Map<java.lang.String,java.lang.Object> cfg)
java.util.List<? extends org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>>	split(int desiredNumSplits, org.apache.beam.sdk.options.PipelineOptions options)

Methods inherited from class org.apache.beam.sdk.io.UnboundedSource

offsetBasedDeduplicationSupported, requiresDeduping

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

CFG_ENABLE_CHECKPOINT_PARTITION_MERGE

public static final java.lang.String CFG_ENABLE_CHECKPOINT_PARTITION_MERGE

See Also:

Constant Field Values

Method Detail

of

public static DirectBatchUnboundedSource of(RepositoryFactory factory,
                                            BatchLogReader reader,
                                            java.util.List<AttributeDescriptor<?>> attrs,
                                            long startStamp,
                                            long endStamp,
                                            java.util.Map<java.lang.String,java.lang.Object> cfg)

split

public java.util.List<? extends org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>> split(int desiredNumSplits,
                                                                                                                                         org.apache.beam.sdk.options.PipelineOptions options)

Specified by:

split in class org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>

createReader

public org.apache.beam.sdk.io.UnboundedSource.UnboundedReader<StreamElement> createReader(org.apache.beam.sdk.options.PipelineOptions options,
                                                                                          DirectBatchUnboundedSource.Checkpoint checkpointMark)

Specified by:

createReader in class org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>

equals

public boolean equals(java.lang.Object o)

Overrides:

equals in class java.lang.Object

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

getCheckpointMarkCoder

public org.apache.beam.sdk.coders.Coder<DirectBatchUnboundedSource.Checkpoint> getCheckpointMarkCoder()

Specified by:

getCheckpointMarkCoder in class org.apache.beam.sdk.io.UnboundedSource<StreamElement,DirectBatchUnboundedSource.Checkpoint>

getOutputCoder

public org.apache.beam.sdk.coders.Coder<StreamElement> getOutputCoder()

Overrides:

getOutputCoder in class org.apache.beam.sdk.io.Source<StreamElement>