Package cz.o2.proxima.direct.io.kafka

Class KafkaAccessor

java.lang.Object

cz.o2.proxima.core.storage.AbstractStorage

cz.o2.proxima.core.storage.AbstractStorage.SerializableAbstractStorage

cz.o2.proxima.direct.io.kafka.KafkaAccessor

All Implemented Interfaces:

AbstractDataAccessor, DataAccessor, java.io.Serializable

public class KafkaAccessor
extends AbstractStorage.SerializableAbstractStorage
implements DataAccessor

Kafka writer and commit log using KafkaProducer.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class cz.o2.proxima.core.storage.AbstractStorage

AbstractStorage.SerializableAbstractStorage

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	ASSIGNMENT_TIMEOUT_MS	Timeout in milliseconds, that consumer should wait for group assignment before throwing an exception.
static java.lang.String	AUTO_COMMIT_INTERVAL_MS	Auto commit interval in milliseconds.
static java.lang.String	EMPTY_POLL_TIME	Minimal time poll() has to return empty records, before first moving watermark to processing time.
static java.lang.String	KAFKA_CONFIG_PREFIX
static java.lang.String	LOG_STALE_COMMIT_INTERVAL_MS	Log stale commit interval in milliseconds.
static java.lang.String	MAX_BYTES_PER_SEC	Maximal read speed in bytes per second.
static java.lang.String	MAX_POLL_RECORDS	Number of records per poll()
static java.lang.String	PARTITIONER_CLASS	Partitioner class for entity key-attribute pair.
static java.lang.String	POLL_INTERVAL_CFG	A poll interval in milliseconds.
static java.lang.String	SEQUENCE_ID_HEADER	A name for a header containing sequential ID of StreamElement (if any).
static java.lang.String	SERIALIZER_CLASS	Class performing parsing from (String, byte[]) to StreamElement.
static java.lang.String	UUID_HEADER	A name for a header containing UUID of StreamElement

Constructor Summary

Constructors

Constructor	Description
KafkaAccessor(EntityDescriptor entity, java.net.URI uri, java.util.Map<java.lang.String,java.lang.Object> cfg)

Method Summary

Modifier and Type	Method	Description
<K,V> KafkaConsumerFactory<K,V>	createConsumerFactory()	Create kafka consumer with specific rebalance listener.
java.util.Optional<CachedView>	getCachedView(Context context)	Retrieve cached view of the data.
java.util.Optional<CommitLogReader>	getCommitLogReader(Context context)	Retrieve commit log reader (if applicable).
<K,V> ElementSerializer<K,V>	getSerializer()	Retrieve ElementSerializer.
java.util.Optional<AttributeWriterBase>	getWriter(Context context)	Retrieve writer (if applicable).
boolean	isAcceptable(AttributeFamilyDescriptor familyDescriptor)	Checker for kafka topics configuration
boolean	verifyCleanupPolicy(org.apache.kafka.clients.admin.ConfigEntry cleanupPolicy)

Methods inherited from class cz.o2.proxima.core.storage.AbstractStorage

equals, hashCode

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface cz.o2.proxima.core.storage.internal.AbstractDataAccessor

getUri

Methods inherited from interface cz.o2.proxima.direct.core.DataAccessor

getBatchLogReader, getRandomAccessReader

Field Detail

KAFKA_CONFIG_PREFIX

public static final java.lang.String KAFKA_CONFIG_PREFIX

See Also:

Constant Field Values

POLL_INTERVAL_CFG

public static final java.lang.String POLL_INTERVAL_CFG

A poll interval in milliseconds.

See Also:

Constant Field Values

PARTITIONER_CLASS

public static final java.lang.String PARTITIONER_CLASS

Partitioner class for entity key-attribute pair.

See Also:

Constant Field Values

SERIALIZER_CLASS

public static final java.lang.String SERIALIZER_CLASS

Class performing parsing from (String, byte[]) to StreamElement. This kas to implement ElementSerializer interface.

See Also:

Constant Field Values

MAX_BYTES_PER_SEC

public static final java.lang.String MAX_BYTES_PER_SEC

Maximal read speed in bytes per second.

See Also:

Constant Field Values

MAX_POLL_RECORDS

public static final java.lang.String MAX_POLL_RECORDS

Number of records per poll()

See Also:

Constant Field Values

AUTO_COMMIT_INTERVAL_MS

public static final java.lang.String AUTO_COMMIT_INTERVAL_MS

Auto commit interval in milliseconds.

See Also:

Constant Field Values

LOG_STALE_COMMIT_INTERVAL_MS

public static final java.lang.String LOG_STALE_COMMIT_INTERVAL_MS

Log stale commit interval in milliseconds.

See Also:

Constant Field Values

ASSIGNMENT_TIMEOUT_MS

public static final java.lang.String ASSIGNMENT_TIMEOUT_MS

Timeout in milliseconds, that consumer should wait for group assignment before throwing an exception.

See Also:

Constant Field Values

EMPTY_POLL_TIME

public static final java.lang.String EMPTY_POLL_TIME

Minimal time poll() has to return empty records, before first moving watermark to processing time. This controls time needed to initialize kafka consumer.

See Also:

Constant Field Values

SEQUENCE_ID_HEADER

public static final java.lang.String SEQUENCE_ID_HEADER

A name for a header containing sequential ID of StreamElement (if any).

See Also:

Constant Field Values

UUID_HEADER

public static final java.lang.String UUID_HEADER

A name for a header containing UUID of StreamElement

See Also:

Constant Field Values

Constructor Detail

KafkaAccessor

public KafkaAccessor(EntityDescriptor entity,
                     java.net.URI uri,
                     java.util.Map<java.lang.String,java.lang.Object> cfg)

Method Detail

createConsumerFactory

public <K,V> KafkaConsumerFactory<K,V> createConsumerFactory()

Create kafka consumer with specific rebalance listener.

Type Parameters:

K - key type

V - value type

Returns:

KafkaConsumerFactory for creating consumers

isAcceptable

public boolean isAcceptable(AttributeFamilyDescriptor familyDescriptor)

Checker for kafka topics configuration

Specified by:

isAcceptable in interface DataAccessor

Parameters:

familyDescriptor - Attribute family descriptor.

Returns:

true if check succeeded, otherwise false

verifyCleanupPolicy

public boolean verifyCleanupPolicy(org.apache.kafka.clients.admin.ConfigEntry cleanupPolicy)

getWriter

public java.util.Optional<AttributeWriterBase> getWriter(Context context)

Description copied from interface: DataAccessor

Retrieve writer (if applicable).

Specified by:

getWriter in interface DataAccessor

Parameters:

context - the serializable context provided by repository

Returns:

optional AttributeWriterBase of this accessor

getCommitLogReader

public java.util.Optional<CommitLogReader> getCommitLogReader(Context context)

Description copied from interface: DataAccessor

Retrieve commit log reader (if applicable).

Specified by:

getCommitLogReader in interface DataAccessor

Parameters:

context - serializable context provided by repository

Returns:

optional @{link CommitLogReader} of this accessor

getCachedView

public java.util.Optional<CachedView> getCachedView(Context context)

Description copied from interface: DataAccessor

Retrieve cached view of the data.

Specified by:

getCachedView in interface DataAccessor

Parameters:

context - serializable context provided by repository

Returns:

optional CachedView of this accessor

getSerializer

public <K,V> ElementSerializer<K,V> getSerializer()

Retrieve ElementSerializer.

Type Parameters:

K - the key type

V - the value type

Returns:

the ElementSerializer.