[Ingestion spec][Middle Manager] load balancing queries

HI All

I have few queries, on the basis of our Druid setup:

Druid version: 0.9.0

  • 2 Overlords in Active Passive mode, with remote workers.
  • 2 Middle Managers with configuration for 4 workers on each
    Middle manager.

We are using Tranquility (version: 0.7.4) for real time ingestion,
and post tasks with:

  • segmentGranularity : Hour
  • queryGranularity : None
  • task.partitions : 2
  • task.replicants : 2

On the basis of above configurations, i have following queries:

  1. Even after setting the queryGranularity = None, is it that
    Druid will still rollup on Milliseconds? I read the following
    excerpt from : http://druid.io/docs/latest/design/index.html
  • *"Phrased another way, the rollup granularity is the minimum granularity you will be able to explore data at and events are floored to this granularity. Hence, Druid ingestion specs define this granularity as the *queryGranularity* **of the data. The lowest supported *queryGranularity* *is millisecond"
  • Is there any configuration in Overlord using which I can set the MiddleManger selection strategy. I see there is a GET request using which you can set the selectstrategy to either: fillCapacity, fillCapacityWithAffinity, equalDistribution and javascript, but is there any corresponding configuration in runtime.properties?
  • In above case we are creating a total of 4 tasks(2 partitions* 2 replications):
  1. <task_name>_partition1_rep1
  2. <task_name>_partition2_rep1
  3. <task_name>_partition1_rep2
  4. <task_name>_partition2_rep2
  • We are planning to use the “equalDistribution” select strategy. Can we ensure that replicative tasks run on different Middle Managers. As if one MM gets down, then we still have all the replicant tasks running on different MM.
  • Question regarding the Druid Integration with other technology page , from the integration diagram on page, I am getting the inference that Tranquility posts the Create Realtime task request to Zookeeper. But I think it is not that but rather Tranquility posts create task request directly to Overlord.

Thanks

Himanshu

Answers inline:

HI All

I have few queries, on the basis of our Druid setup:

Druid version: 0.9.0

  • 2 Overlords in Active Passive mode, with remote workers.
  • 2 Middle Managers with configuration for 4 workers on each
    Middle manager.

We are using Tranquility (version: 0.7.4) for real time ingestion,
and post tasks with:

  • segmentGranularity : Hour
  • queryGranularity : None
  • task.partitions : 2
  • task.replicants : 2

On the basis of above configurations, i have following queries:

  1. Even after setting the queryGranularity = None, is it that
    Druid will still rollup on Milliseconds? I read the following
    excerpt from : http://druid.io/docs/latest/design/index.html
  • *"Phrased another way, the rollup granularity is the minimum granularity you will be able to explore data at and events are floored to this granularity. Hence, Druid ingestion specs define this granularity as the *queryGranularity* **of the data. The lowest supported *queryGranularity* *is millisecond"

Yes, druid will still try to rollup to Milliseconds. You can prevent any rollup by having some unique field in your events. Unfortunately events are still probably-at-least-once (which is part of why the batch side of lambda is needed) so even if you have some unique identifier it is possible that an event makes it in twice during the realtime portion.

  • Is there any configuration in Overlord using which I can set the MiddleManger selection strategy. I see there is a GET request using which you can set the selectstrategy to either: fillCapacity, fillCapacityWithAffinity, equalDistribution and javascript, but is there any corresponding configuration in runtime.properties?

You’re looking for http://druid.io/docs/0.9.0/configuration/indexing-service.html#dynamic-configuration

  1. In above case we are creating a total of 4 tasks(2 partitions* 2 replications):
  2. <task_name>_partition1_rep1
  3. <task_name>_partition2_rep1
  4. <task_name>_partition1_rep2
  5. <task_name>_partition2_rep2
  • We are planning to use the “equalDistribution” select strategy. Can we ensure that replicative tasks run on different Middle Managers. As if one MM gets down, then we still have all the replicant tasks running on different MM.

I don’t remember off the top of my head where that is ensured, but it is the way the MM is supposed to spread out load.

Be warned if you use autoscaling, then using equalDistribution means your nodes won’t ever really scale down.

  • Question regarding the Druid Integration with other technology page , from the integration diagram on page, I am getting the inference that Tranquility posts the Create Realtime task request to Zookeeper. But I think it is not that but rather Tranquility posts create task request directly to Overlord.

I believe you are correct, tranquility should be talking to the overlord directly.

Thanks for the response, few follow up queries :slight_smile:

Is there any configuration in Overlord using which I can set the MiddleManger selection strategy. I see there is a GET request using which you can set the selectstrategy to either: fillCapacity, fillCapacityWithAffinity, equalDistribution and javascript, but is there any corresponding configuration in runtime.properties?

Answers Inline.

Thanks for the response, few follow up queries :slight_smile:

Is there any configuration in Overlord using which I can set the MiddleManger selection strategy. I see there is a GET request using which you can set the selectstrategy to either: fillCapacity, fillCapacityWithAffinity, equalDistribution and javascript, but is there any corresponding configuration in runtime.properties?

You’re looking for http://druid.io/docs/0.9.0/configuration/indexing-service.html#dynamic-configuration.

Above link list the GET request for specifying the select strategy, is there any configuration as well, which i can specify in Overlord runtime.properties file. I don’t want to make a GET request to every Druid setup (test/dev/prod).

These configs are dynamic so that you can modify them without restarting the overlord. Dynamic configs are stored in database in config table. you can set the config in the database and restart the overlords if you don’t wanna do it HTTP way.

In above case we are creating a total of 4 tasks(2 partitions* 2 replications):

  1. <task_name>_partition1_rep1
  2. <task_name>_partition2_rep1
  3. <task_name>_partition1_rep2
  4. <task_name>_partition2_rep2
  • We are planning to use the “equalDistribution” select strategy. Can we ensure that replicative tasks run on different Middle Managers. As if one MM gets down, then we still have all the replicant tasks running on different MM.

I don’t remember off the top of my head where that is ensured, but it is the way the MM is supposed to spread out load.

Is MM will spread irrespective of select strategy type. If we go with selectstartegy=“fillcapacity”, does even then Overlord will spread the replicant tasks to different MM?

yeah, it does NOT depend on worker select strategy,

The spread is controlled by availabilityGroup specified in task.json.

Overlord ensures that all tasks within same availability group are assigned to different middle-managers. When using tranquility to create tasks, tranquility handles setting proper availability group for tasks for you.

Thanks all for the responses.

Hi All,

We want to set middle manger **selectstrategy to **equalDistribution, but it doesn’t seems to be working now.

Let me know if we are doing it wrongly:

root@dr-crd1:~# curl -X POST http://dr-crd1.XXXXXXX:8090/druid/indexer/v1/worker -d ‘{ “selectStrategy”: { “type”: “equalDistribution”} }’
root@dr-crd1:~# curl -X GET http://dr-crd1.XXXXXXX:8090/druid/indexer/v1/worker
root@dr-crd1:~# curl -XGET http://dr-crd1.XXXXXXX:8090/druid/indexer/v1/worker/history?count=5
root@dr-crd1:~#

``

use this to try

curl -XPOST -H ‘X-Druid-Author: lucky’ -H ‘X-Druid-Comment: lucky’ -H ‘Content-Type: application/json’ http://dr-crd1.XXXXXXX:8090/druid/indexer/v1/worker -d ‘{ “selectStrategy”: { “type”: “equalDistribution” } }’