Issue with Overlord Dynamic Configuration

Hi Druid,

I try to use affinity Configs to make tasks for a dataSource only ever running designated middleManagers. But the replica task violates the affinity rule. Looking forward to your reply.

AffinityRule.json:

{

“selectStrategy”: {

“type”: “equalDistribution”,

“affinityConfig”: {

“affinity”: {

“qpixel_reporting_realtime”: [“ip-10-65-6-111.us-west-2.bds.aws:8092”, “ip-10-65-1-107.us-west-2.bds.aws:8092”]

},

“strong”: true

}

}

}

Unexpected Task Distribution

Partial Datasource ingestion spec:

{

“ioConfig”: {

“topic”: “XXXXX”,

“replicas”: 2,

“taskCount”: 1,

“taskDuration”: “PT1H”,

“completionTimeout”: “PT30M”,

“consumerProperties”: {

“bootstrap.servers”: “XXXX”,

“isolation.level”: “read_committed”

}

}

The Druid version is v16.0

Hi Xuanyi,
I suspect affinity config accepts only IP address but not host names.

Is it possible for you to try with IP address and Port instead of host name and port?

If IP address changes , then also it might not achieve desired result.

Thank you.

–siva

I should be more clear. The official documentation said that Druid Affinity require a ‘host value’ instead IP. And the above hostname does work for the leading task. The leading task could run on designated middleManagers but replica task evenly distributed. I have test it many times and have a strong feeling that it’s Druid internal bug. Please C.F. the official link below https://druid.apache.org/docs/latest/configuration/index.html#affinity.