Questions about tranquility ingestion.

Hi Team,

Now we are considering to migrate from real-time node ingestion to tranquility, but we still have some questions remain unclear, could you please help?

Our use case: we have a pool of app hosts to do tranquility ingest, each host runs exact the same code and handles part of the whole traffic.

Questions:
#1, if I config the partitions=1 on each host, then the result data table will have 1*N partitions, correct?
#2, do I need to set -Duser.timezone=UTC on these app?
#3, what will happen if the overlord dies? If I have 2 overlord nodes, will the other live overlord take over the role?
#4, what will happen if the middle manager/peon dies? there will be data loss in this case, correct?
#5, our middle manager nodes are also used to do batch hadoop ingestion, so the “druid.indexer.runner.javaOpts” is different in that case(heap size, -DHADOOP_USER_NAME=xxx, log, etc.), should we isolate the two kinds of task to avoid running on the same middle manager? How to do that?
#6, what’s the optimized size of events that sent each time? druidService.apply(listOfEvents)

thanks a lot!

Hi, please see inline.

Hi Team,

Now we are considering to migrate from real-time node ingestion to tranquility, but we still have some questions remain unclear, could you please help?

Our use case: we have a pool of app hosts to do tranquility ingest, each host runs exact the same code and handles part of the whole traffic.

Questions:
#1, if I config the partitions=1 on each host, then the result data table will have 1*N partitions, correct?

What is N here?

#2, do I need to set -Duser.timezone=UTC on these app?

We recommend in general that you set UTC timezone everywhere. It will make operating your entire system much easier.

#3, what will happen if the overlord dies? If I have 2 overlord nodes, will the other live overlord take over the role?

Yes, the other overlord acts as a backup.

#4, what will happen if the middle manager/peon dies? there will be data loss in this case, correct?

You should replicate your realtime ingestion. A middle manager dies means that tasks on the middle manager fails.

#5, our middle manager nodes are also used to do batch hadoop ingestion, so the “druid.indexer.runner.javaOpts” is different in that case(heap size, -DHADOOP_USER_NAME=xxx, log, etc.), should we isolate the two kinds of task to avoid running on the same middle manager? How to do that?

Middle managers just kick off jobs to a Hadoop cluster so you don’t need to isolate these kinds of tasks.

#6, what’s the optimized size of events that sent each time? druidService.apply(listOfEvents)

If there are no default values, 1k events should be reasonable.

Thanks for the answer.

For #1, N means N tranquility apps, each app runs the same code, same configuration, so there will be N partitions for each data source?

another question, if I want to modify a data source’s tranquility config on the fly, should I close the existing java service and start a new service with modified config?

thanks again.

Hi,

For #1, the “partitions” count is global across all tranquility instances that share the same configuration. So if you set partitions = 1 on N machines, you’ll only get one partition in Druid, and all your machines will write to that one partition.

Yeah, it’s fine to just close the old service and open a new one with a new config. One note there is that you should delay closing the old service until all of its futures have resolved. That’ll make sure that all in-flight messages have actually made it through before the connections are closed.

Hi Gian,

Thanks for the answer, I am waiting for it :slight_smile:

So if all tranquility apps write to the one partition, then only one indexing task is enough is enough, right?

And in my app, I just call services.apply(eventList) and never care the return value since I doubt the Future.get() would harm the throughput, is that justified? Or I can call Future.get() to know if any events are discarded.

thanks!

Yeah, if you set partitions = 1, then only one task per segmentGranularity will get created (you might get more than one task at a time due to task overlap).

I’d definitely suggest looking at the result of the future (by calling Await.result or by adding a callback) since it’s possible that the send will fail in some way, and if it does, information about dropped messages would be in the return value or returned exception.