org.wikidata.query.rdf.tool.change

Class KafkaPoller

java.lang.Object

org.wikidata.query.rdf.tool.change.KafkaPoller

All Implemented Interfaces:

Closeable, AutoCloseable, Change.Source<KafkaPoller.Batch>

public class KafkaPoller
extends Object
implements Change.Source<KafkaPoller.Batch>

This poller will connect to Kafka event source and get changes from one or more topics there. After each block of changes is processed, we store current offsets in the DB as: <http://www.wikidata.org> wikibase:kafka ( "topic:0" 1234 ) When the poller is started, the offsets are initialized from two things: a) if the storage above if found, use that for initialization b) if not, use the given timestamp to initialize This also means if the Kafka offset is ever reset, the old offsets need to be manually cleaned up. See KafkaOffsetsRepository.UPDATE_OFFSETS query for how to do it. See https://wikitech.wikimedia.org/wiki/EventBus for more docs on events in Kafka.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaPoller.Batch

Constructor Summary

Constructors

Constructor and Description
KafkaPoller(org.apache.kafka.clients.consumer.Consumer<String,ChangeEvent> consumer, WikibaseRepository.Uris uris, Instant firstStartTime, int batchSize, Collection<String> topics, KafkaOffsetsRepository kafkaOffsetsRepository, boolean ignoreStoredOffsets, com.codahale.metrics.MetricRegistry metricRegistry)

Method Summary

Modifier and Type	Method and Description
static KafkaPoller	buildKafkaPoller(String kafkaServers, String consumerId, Collection<String> clusterNames, WikibaseRepository.Uris uris, int batchSize, Instant startTime, boolean ignoreStoredOffsets, KafkaOffsetsRepository kafkaOffsetsRepository, com.codahale.metrics.MetricRegistry metricRegistry)
void	close()
com.google.common.collect.ImmutableMap<org.apache.kafka.common.TopicPartition,Long>	currentOffsets()
KafkaPoller.Batch	firstBatch() Fetch the first batch.
KafkaPoller.Batch	nextBatch(KafkaPoller.Batch lastBatch) Fetches the next batch after lastBatch.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaPoller

public KafkaPoller(org.apache.kafka.clients.consumer.Consumer<String,ChangeEvent> consumer,
                   WikibaseRepository.Uris uris,
                   Instant firstStartTime,
                   int batchSize,
                   Collection<String> topics,
                   KafkaOffsetsRepository kafkaOffsetsRepository,
                   boolean ignoreStoredOffsets,
                   com.codahale.metrics.MetricRegistry metricRegistry)

Method Detail

buildKafkaPoller

@Nonnull
public static KafkaPoller buildKafkaPoller(String kafkaServers,
                                                    String consumerId,
                                                    Collection<String> clusterNames,
                                                    WikibaseRepository.Uris uris,
                                                    int batchSize,
                                                    Instant startTime,
                                                    boolean ignoreStoredOffsets,
                                                    KafkaOffsetsRepository kafkaOffsetsRepository,
                                                    com.codahale.metrics.MetricRegistry metricRegistry)

firstBatch

public KafkaPoller.Batch firstBatch()
                             throws RetryableException

Description copied from interface: Change.Source

Fetch the first batch.

Specified by:

firstBatch in interface Change.Source<KafkaPoller.Batch>

Throws:

RetryableException - is the fetch fails in a retryable way

currentOffsets

public com.google.common.collect.ImmutableMap<org.apache.kafka.common.TopicPartition,Long> currentOffsets()

nextBatch

public KafkaPoller.Batch nextBatch(KafkaPoller.Batch lastBatch)
                            throws RetryableException

Description copied from interface: Change.Source

Fetches the next batch after lastBatch.

Specified by:

nextBatch in interface Change.Source<KafkaPoller.Batch>

Throws:

RetryableException - is the fetch fails in a retryable way

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable