pl.allegro.tech.hermes.consumers.consumer.offset

Class OffsetCommitter

java.lang.Object

pl.allegro.tech.hermes.consumers.consumer.offset.OffsetCommitter

All Implemented Interfaces:

java.lang.Runnable

public class OffsetCommitter
extends java.lang.Object
implements java.lang.Runnable

Note on algorithm used to calculate offsets to actually commit.

The idea behind this algorithm is that we would like to commit: * maximal offset marked as committed * but not larger than smallest inflight offset (smallest inflight - 1)

Important note! This class is Kafka OffsetCommiter, and so it perceives offsets in Kafka way. Most importantly committed offset marks message that is read as first on Consumer restart (offset is inclusive for reading and exclusive for writing).

There are two queues which are used by Consumers to report message state: * inflightOffsets: message offsets that are currently being sent (inflight) * committedOffsets: message offsets that are ready to get committed

This committer class holds internal state in form of inflightOffsets and maxCommittedOffsets collections. * inlfightOffsets are all offsets that are currently in inflight state. * maxCommittedOffsets are offsets (maximum per partition) of already committed messages that could not yet be committed to kafka due to an existing inflight offset on the same partition

In scheduled periods, commit algorithm is run. It has three phases. First one is draining the queues and performing reductions: * drain committedOffsets queue to collection - it needs to be done before draining inflights, so this collection will not grow anymore, resulting in having inflights unmatched by commits; commits are incremented by 1 to match Kafka commit definition * update the maxCommittedOffsets map with largest committed offsets * drain inflightOffsets

Second phase is calculating the offsets:

* calculate maximal committed offset for each subscription and partition * calculate minimal inflight offset for each subscription and partition

Third phase is choosing which offset to commit for each subscription/partition. This is the minimal value of * maximum committed offset * minimum inflight offset

This algorithm is very simple, memory efficient, can be performed in single thread and introduces no locks.

Constructor Summary

Constructors

Constructor and Description
OffsetCommitter(OffsetQueue offsetQueue, ConsumerPartitionAssignmentState partitionAssignmentState, MessageCommitter messageCommitter, int offsetCommitPeriodSeconds, pl.allegro.tech.hermes.common.metric.HermesMetrics metrics)

Method Summary

Modifier and Type	Method and Description
void	run()
void	shutdown()
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OffsetCommitter

public OffsetCommitter(OffsetQueue offsetQueue,
                       ConsumerPartitionAssignmentState partitionAssignmentState,
                       MessageCommitter messageCommitter,
                       int offsetCommitPeriodSeconds,
                       pl.allegro.tech.hermes.common.metric.HermesMetrics metrics)

Method Detail

run

public void run()

Specified by:

run in interface java.lang.Runnable

start

public void start()

shutdown

public void shutdown()