Fundamental issues with druid

Hello ,

I have evalauting druid and its been fairly pain ful experience

  • Realy difficult to use realtime indexing : Its such a painful ? Does it really workks in production or its gimmick? How much time its takes to index? I always get out memory , service not available
  • Index get overwritten !! in historical update
  • No dynamic dimension support , you cant add dimension on the fly
  • Huge resource requirement for memory
  • No Join support
    Can community confirm are any solution /work around to above issues ? and why do people use druid ? is for processing logs ?no critical data

Hello ,

I have evalauting druid and its been fairly pain ful experience

  • Realy difficult to use realtime indexing : Its such a painful ? Does it really workks in production or its gimmick? How much time its takes to index? I always get out memory , service not available

it is used in production by at least dozen of big companies like Yahoo, Netflix Metamarket etc

it can be tedious to set it up the first time but once you get it right it works well

can you elaborate more about what is exactly painful ? what is your setup ? what can of error you are getting ? would love to help you to get this up and running

  • Index get overwritten !! in historical update

Not sure what you mean by this ? this should not happen updates can be done in a rolling fashion without any downtime

  • No dynamic dimension support , you cant add dimension on the fly

Druid supports schema less ingestion did you try that ?

  • Huge resource requirement for memory

If you don’t need to serve fast query you don’t have to provide lot of memory, in fact Druid uses memory mapped files so can do swapping out of disk if not enough RAM available

  • No Join support

Support of Join is coming soon maybe next release or so

Currently we do support query time lookup which can be considered as a left hash join

Can community confirm are any solution /work around to above issues ? and why do people use druid ? is for processing logs ?no critical data

We have seen multiple use cases from monitoring to tracking and analyzing user activities, anomaly detection …

Totally disagree.

Druid is very powerful tool, however it needs time and experience to master it optimal configuration and usage.

At Deep.BI We run multi tenant Druid cloud for our customers (mainly big media companies) for real-time data processing (collection, analytics, real-time usage in other systems).

Rest of my comment inline.

Hello ,

I have evalauting druid and its been fairly pain ful experience

  • Realy difficult to use realtime indexing : Its such a painful ? Does it really workks in production or its gimmick? How much time its takes to index? I always get out memory , service not available

How exactly do you index realtime data? There’re two methods: Kafka Indexing Service and Tranquility, The obsolete version of Druid was using real-time nodes, and in fact, they were problematic.

  • Index get overwritten !! in historical update

Never happened. Probably there’s some misunderstanding here.

  • No dynamic dimension support , you cant add dimension on the fly

100% of our customers use dynamic dimensions - they change data schema on the fly, and Druid process it natively.

  • Huge resource requirement for memory

It all depends on your use case. There are three components of Druid that might need machines with 10+ threads and 128-256GB RAM: historicals, brokers and middlemanagers. I highly recommend to turn on Druid metrics and analyze your usage vs. configuration to get optimal settings. BTW, we provide a ready-to-use Druid monitoring tool that can give you immediate answers what’s wrong.

  • No Join support

You can do by the right querying Druid.

Can community confirm are any solution /work around to above issues ? and why do people use druid ? is for processing logs ?no critical data

Our customers use Druid we host for them for very critical data: serving ads, optimize subscriptions, tracking content performance. Just drop me a line if you need some help: druiddeepwatch@deep.bi.

Hi Sebastian.

I’ve been evaluating Druid over the past few weeks and was hoping you would expound on something you said, specifically: “There’re two methods: Kafka Indexing Service and Tranquility, The obsolete version of Druid was using real-time nodes, and in fact, they were problematic.” Are you saying that you don’t need to / shouldn’t configure true realtime nodes anymore in favor of the other 2 methods? Is it even possible to dedicate realtime nodes in the latest version?

Thanks.

Michael

I wouldn’t recommend to use real-time nodes anymore. Now we completely switched to Kafka Indexing Service which the biggest advantage is the ability to ingest both real-time and historical data in the same pipeline / time. You don’t have to fit into recent a window (if events were outside this windows they were dropped). Also, real-time nodes keeps data in-memory and it happens that they they “hang”, so you have to restart it, loosing all the data that were not stored in deep storage (sometimes it was be a month of data). In that case, to recover this data you must have had:

  • raw data backup

  • some indexer (like Kafka Indexing Service)

The reasons that real-time nodes stop working are for example:

  • too heavy data load (peaks)

  • anomalies in data (we don’t use schema, so users can load any data, sometimes thousands of attributes)

  • too heavy querying (note that real-time nodes, as well KIS, is responsible both for ingesting data and handling queries on data that is not yet handed over to deep storage)

We were considering Tranquility too that looks more stable than real-time nodes, however it has the same restriction regarding only recent data, so to load older data you must have an additional indexing component.

So, taking into account all the above we started using:

  • KIS for all data ingestion (current and historical) and query this data

  • Hadoop Index Task - for merging segments that KIS don’t merge (this is a bug to be solved in the future version of Druid)

Hope this helps.