Druid Use Case: Event Sourcing?

Does anyone have any experience / thoughts on using Druid in conjunction with Kafka for an event sourcing architecture?

The idea is that all messages on Kafka topics would end up on Druid (with historical data ending up on deep storage, like s3).

Running OLAP queries to see usage intensity of particular services by particular customers is definitely a plus, but reliability and ability to ‘replay’ (i.e. feed all events from some time period to a temporary kafka topic to rebuild a service) are more important.

cheers.

Hi Charlie, Ingesting from kafka is a pretty common use-case.
Have you looked at http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html ?

Does this help ?

Hi Nishant,

Thanks for pointing me to that, that certainly helps. Good to know integration is there for exactly once consumption.

As for the event sourcing part of things, are you aware of whether its possible to start a kafka producer from a druid query (that would replay the events returned by the query in sequence) in an equally ‘exactly once’ guaranteed sort of way?

The use case is that I have service B that should respond to events emitted by service A (let’s say by altering its DB). In production this happens by B consuming events on a kafka topic that A puts events to. If I want to rebuild the state of B from scratch, I’d like to be able to query all historical events emitted by A (i.e. from Druid), and feed them to a temporary kafka topic that B can feed off to rebuild itself.

This is why I’m asking about Druid putting to kafka (in special cases) as well as ingesting from kafka.

cheers.

Inline.

Hi Nishant,

Thanks for pointing me to that, that certainly helps. Good to know integration is there for exactly once consumption.

As for the event sourcing part of things, are you aware of whether its possible to start a kafka producer from a druid query (that would replay the events returned by the query in sequence) in an equally ‘exactly once’ guaranteed sort of way?

I don’t believe this is possible. You would have to write the code.