Re: [druid-user] Kafka source and UPSERTs

Hey Mathieu,
Welcome to the Druid community :slight_smile:

While I haven’t used Kafka Streams myself, by reading your description, I think the way to achieve what you’re looking for is using Druid’s roll-up capability (see https://druid.apache.org/docs/latest/tutorials/tutorial-rollup.html).
That way, at ingestion time:

  1. When an event with a new key comes along, a new record will be created in Druid.
  2. When an event with an existing key comes along, Druid will update the existing record by adding the values associated with that key, to the current values.
    This essentially gives you the UPSERT mode I believe you’re looking for.

As per the next step, of how to actually ingest the data into Druid - Druid supports a few ingestion methods (https://druid.apache.org/docs/latest/ingestion/index.html#ingestion-methods).
In your case, I think Kafka ingestion (https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html) would make sense, so basically:

  1. Write the output records from your Kafka Streams app to a Kafka topic
  2. Have Druid read the data from Kafka
    Hope that makes sense, let me know if you have any further questions.

Good luck!