Why realtime config directory (conf/druid/realtime) is missing in druid-0.9.1.1

Hi I have setup a simple druid cluster with 3 machines.

druid-01: Coordinator and Overlord processes, zookeeper

druid-02: Historicals and MiddleManagers, zookeeper, tranquility-kafka

druid-03: Druid Brokers, zookeeper

utility-01: metadataDB (mysql), kafka

I am new to druid and have some confusions about the setup of latest release of druid.

Q.1 : Why realtime config directory (conf/druid/) missing in druid-0.9.1.1?

Q. 2 : Why realtime configuration details are not mentioned in “clustering” document in druid website? Is this normal?

Q. 3 :I have inserted some realtime data through tranquility-kafka in druid cluster. After some time when I am querying data, data is missing from cluster. What is the problem?

I would really appreciate any help.

Hi Jitender, starting with Druid 0.9.0, we are recommending everyone use the indexing service instead of realtime nodes. Some more information inline.

Hi I have setup a simple druid cluster with 3 machines.

druid-01: Coordinator and Overlord processes, zookeeper

druid-02: Historicals and MiddleManagers, zookeeper, tranquility-kafka

druid-03: Druid Brokers, zookeeper

utility-01: metadataDB (mysql), kafka

I am new to druid and have some confusions about the setup of latest release of druid.

Q.1 : Why realtime config directory (conf/druid/) missing in druid-0.9.1.1?

In the indexing service, all realtime ingestion is done through middle manager.

Q. 2 : Why realtime configuration details are not mentioned in “clustering” document in druid website? Is this normal?

Same as above.

Q. 3 :I have inserted some realtime data through tranquility-kafka in druid cluster. After some time when I am querying data, data is missing from cluster. What is the problem?

This is likely because you timestamps are out of band. If you have data in Kafka, you can try the new exactly once kafka ingestion setup available in 0.9.1.1. You can find a tutorial of it here: http://imply.io/docs/latest/tutorial-kafka-indexing-service.html

Hi Fangjin,

Thanks for sharing this useful information.

Hi Fangjin,

Now I have one question. I am ingesting data through tranquility-kafka into druid. But when I am checking mysql metastore, there is no metadata in the druids tables and also segments are not created in hdfs. It means my druid services are not able to create segments. Can you help me in understanding this concept? And what configurations I have to do to fix this?

Regards

Jitender

Hey Jitender,

With realtime ingestion, segments are only created periodically (based on your segmentGranularity and windowPeriod). With the defaults of segmentGranularity = HOUR and windowPeriod = PT10M, you should see segments created in the metadata store and deep storage within a couple of hours. Until that point, realtime indexers serve queries off partially created segments on local storage.

Thanks Gian.

Hi Gian,

I have ingested data 12 hours ago, but still there is no records in metastore tables in MYSQL and no segment created in HDFS. Even no segments are created on local storage. And I am not able to do query on old data (ingested 12 hours ago) using pivot.

M i doing something wrong?

Records in metastore:

For old data, they are outside Druid’s window period and will be discarded. You can either use the new kafka indexing service task to get around this problem, or use batch ingestion.

http://imply.io/docs/latest/tutorial-kafka-indexing-service.html

Hi Fangjin,

Thanks for your answer. You mean tranquility-kafka don’t have fluid (ingest, persist, merge, and handoff steps) functionality which is available in real-times node. Is it correct?

And I am trying to open http://imply.io/docs/latest/tutorial-kafka-indexing-service.html link in multiple browsers but this page is not working for many days. I always get a white blank screen.

Hi Jitender, see https://groups.google.com/forum/#!searchin/druid-development/fangjin$20yang$20"thoughts"/druid-development/aRMmNHQGdhI/muBGl0Xi_wgJ

I’m not sure why you can’t open that page, it seems to work for me. Is your firewall blocking it?