Coordinator log size; insufficient active servers

Hi all,

My current setup is on a single server, launched with the micro-quickstart configuration. (all the Druid services on the same host, AWS t3.xlarge instance)
I’ve also configured S3 deep storage and basic security.

My problem is with coordinator-overlord.log log size, since the coordinator logs the following message:

2020-09-15T07:55:27,339 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2020-09-15T07:55:32,345 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2020-09-15T07:55:37,352 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2020-09-15T07:55:42,358 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.

I checked the Druid coordinator UI (http://“host”:8081) and the Historical process is available, ingestion jobs and query works fine.

In less a day I’ve a 7GB log file… Is there anything I can tune to fix this?

Thanks,
Lorenzo

Hey Lorenzo - balancing requires more than one server, so WARN is just saying “hey - you don’t have more than one server.” Which is fine… More importantly, are the data sources showing as 100% loaded in the console?

Oh wait! Are you asking can you set some configuration to stop generating such MASSIVE log files?

Hi Peter,

Yes the point is on stop generating so many logs. And yes, datasources are loaded fine and are available for queries. I’ve read about issues/bugs with historical loading queue (but seems to be solved in the latest Druid version, which I’m using), but I know too much little apache Druid to ensure it is my problem.

Thanks,
Lorenzo

No worries! OK hm I will maybe ask around a bit you know to see what you might be able to do…

One of my teammates pointed me at this issue in Github: https://github.com/apache/druid/issues/9452 - it looks like you are not the first to spot this.

For now you are probably best to change the log4j configuration - e.g. to use rolling logs.
https://druid.apache.org/docs/latest/configuration/logging.html

Hi Peter,
I’ll watch the issue on github and meanwhile follow you suggestion

Thanks a lot

Lorenzo

I just spotted this PR … it would be worth adding your weight to this if it helps :slight_smile:

https://github.com/apache/druid/pull/10147