[druid-user] Loadrule not getting triggered by coordinator

Hi all,

We have recently seen an issue where coordinator is not triggering loadrule for around 8 hours even though data is getting inserted continuously.

Please see below logs. We see a pattern where it’s not triggering loadrule for 8 hours. Please assist if anyone has seen similar issue.

We are using Druid version 0.18.

2021-03-29T13:30:18,184 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.rules.LoadRule - Assigning ‘primary’ for segment [abc_2021-03-29T14:00:00.000Z_2021-03-29T15:00:00.000Z
_2021-03-29T13:03:45.535Z_6] to server [xxx] in tier [hot]
2021-03-29T13:30:18,186 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.rules.LoadRule - Assigning ‘replica’ for segment [abc_2021-03-29T14:00:00.000Z_2021-03-29T15:00:00.000Z
_2021-03-29T13:03:45.535Z_6] to server [xxx] in tier [hot]
--------------------------------------------------------- no load rule in coordinator --------------------------------------------------------------------------
2021-03-29T21:34:56,722 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.rules.LoadRule - Assigning ‘primary’ for segment [abc_2021-03-29T22:00:00.000Z_2021-03-29T23:00:00.000Z_2021-03-29T21:03:33.443Z_5] to server [xxx] in tier [hot]
2021-03-29T21:34:56,726 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.rules.LoadRule - Assigning ‘replica’ for segment [abc_2021-03-29T22:00:00.000Z_2021-03-29T23:00:00.000Z_2021-03-29T21:03:33.443Z_5] to server [xxx] in tier [hot]

Hi Shalin,

If you can summarise with brief it would help to understand an issue,

  1. Is it happening for all Datasources?
  2. Have you create new rules?
  3. Have you update existing rules?

Thanks & Regards,
Kishor Patil

Hi Kishore,

Yes. This is happening for all resources. We haven’t created new rules or modified any existing rules. We restarted our servers by increasing memory allocated to coordinator on Saturday and all worked fine until yesterday when we started seeing these issues.

Also, we are not able to find any exceptions in coordinator or historical logs. I am wondering if this has anything to do with below configurations as I am not able to figure out how these 2 parameters should be set at historical end.

Segment storage

druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:4700000000000}]
druid.server.maxSize=4700000000000

Hi Shalin,

You can set mentioned params in Historical runtime.properties as it is.

Thanks & Regards,
Kishor Patil