ElasticSearch, Kafka Concurrent Write

We are doing CQRS like design to separate for different domains.

And data ETLed through Kafka, writing into ElasticSearch.

In case of ElasticSearch in a nature, index is immutable, partial updates still lead into replacement of old index into new.

Elastic’s policy on concurrent writing is last-write-wins, which means we need to control over concurrency.

There are two choices on us, one is optimistic locking, the other is by partitioning Kafka message by index id so there will be a single writer for each index, however that will lead to higher throughput, and you don’t necessarily want to use more partitions than needed because increasing partition count simultaneously increases the number of open server files and leads to increased replication latency.