issues with warming period

I recently started using druid and am currently using the setup provided in imply package with slight modification in the configuration. I am trying to ingest real-time data using tranquility.
Following are few important configuration used for tranquility-server:

“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “minute”,
“queryGranularity” : “none”
}

“tuningConfig” : {
“type” : “realtime”,
“maxRowsInMemory” : “100000”,
“intermediatePersistPeriod” : “PT1S”,
“windowPeriod” : “PT1S”
}

“properties” : {
“task.warmingPeriod” : “PT10S”,
“zookeeper.connect” : “localhost”,
“druid.discovery.curator.path” : “/druid/discovery”,
“druid.selectors.indexing.serviceName” : “druid/overlord”,
“druidBeam.firehoseGracePeriod” : “PT1M”,
“http.port” : “8200”,
“http.threads” : “40”
}

Since the segment granularity is a minute, a new task is being created every minute but the new task setup takes some time which blocks the ingestion for a few seconds and hence I had configured a warming up time of 10seconds but still all the tasks were being created only when the segment period started(The creation time in indexing logs is still the segment boundary). Could someone please let me know what the issue might be?

Also I understand that the task shutoff time depends on the segment and window period but it also has an additional component firehoseGracePeriod, as far as I could figure out this time is kind of a buffer for some lingering packets. Please let me know if the interpretation is wrong.

Thanks in advance.

Hi Parth, why do you need minutely tasks?

Hello,

There was a favorable size of segments mentioned in the documents so we have decided to keep the segment granularity to 1 minute based on that. Since the granularity is small(a minute), we wanted that the the next task creation starts slightly before the last task ends so that the data ingestion can continue smoothly across the time segment boundary and were using ‘warming period’ for the same but the result was not as expected.

Does the time config for warming period depend on the segment granularity?

Thanks.

Hi Parth,
Warming period should not depend on segment granularity and can be configured independently of segment granularity in tranquility.
Also please note, that 1 minute is quite small interval for segment and a task. A task creation involves spawning up a new JVM on middlemanager which may take some time.

Have you tried scaling out by increasing your partitions and keeping the segment granularity to 1 hour in order to have segment size near to ideal segment size (i.e 5M rows per partition for 1 hour) ?

Hi,

Thanks a lot for the advice I would definitely investigate the idea of partitioning further.

Currently I wanted to see the concept of warming period in play so I tested it with segment granularity of 15 minutes and warming period of 5 minutes. My expectation was that in case of a current stream of data(lets say from 10:00 to 10:15) the new task creation should start at 10:10 and the transition to the new segment should be smooth but instead the new task gets created at 10:15 only and there is a few seconds of gap before data is added in the segment. Attached is a screenshot showing the gap in the data ingestion. Can you please help with what the issues might be, I have checked the log files for any errors but there arent any.

Thanks.

Hi Parth,
Your expectation about warmingPeriod seems correct.

I wonder if the gap is due to not enough worker.capacity on the middlemanagers to run tasks for current and next set of segments concurrently and tasks being submitted to the overlord are just in pending state on the overlord waiting for a free slot to run the task.

Can you check that you have worker.capacity configured to allow all the tasks to run concurrently ?

Hello Nishant,

I have set the worker.capacity to 15 at the moment to avoid any such issues. Still the task creation time as seen in the overlord log is the segment boundary. I am attaching the respective logs and the tranquility server config file for reference.

Thanks

middleManager.log (390 KB)

overlord.log (394 KB)

server.json (1.88 KB)

tranquility-server.log (52.4 KB)

Hello guys,

Can anyone provide any insights into the matter, its really important for the use case that the transition occur smoothly between the segments?

Thanks,

Parth Shah

Hi Parth,
you seem to be hitting - https://github.com/druid-io/tranquility/pull/174

Could you try with the latest tranquility master ?

Hey Nishant,

Thanks a lot. The issue seemed to be resolved now.