Log Level - which xml to use?

I am completely new to druid. We have this baremetal server where the druid is running. Not sure who set it up. the logs are getting bigger and we are running into disk capacity issue. I was reading this thread - Druid.io (imply-2.2.3): log level settings

When I searched for the xml on my system I found many xml files.

root@dbsystems1:~/druid/druid/var/sv# find / -iname log4j2.xml
/root/druid/druid_bkp/conf/zk/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/medium/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/small/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/micro-quickstart/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/nano-quickstart/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/large/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/single-server/xlarge/_common/log4j2.xml
/root/druid/druid_bkp/conf/druid/cluster/_common/log4j2.xml
/root/druid/druid/conf/zk/log4j2.xml
/root/druid/druid/conf/druid/single-server/medium/_common/log4j2.xml
/root/druid/druid/conf/druid/single-server/small/_common/log4j2.xml
/root/druid/druid/conf/druid/single-server/micro-quickstart/_common/log4j2.xml
/root/druid/druid/conf/druid/single-server/nano-quickstart/_common/log4j2.xml
/root/druid/druid/conf/druid/single-server/large/_common/log4j2.xml
/root/druid/druid/conf/druid/single-server/xlarge/_common/log4j2.xml
/root/druid/druid/conf/druid/cluster/_common/log4j2.xml

I edited the last one to match level to error

<Configuration status="WARN">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
    </Console>
  </Appenders>
  <Loggers>
    <Root level="error">
      <AppenderRef ref="Console"/>
    </Root>

Do I need to edit any more xmls?

Welcome @Makrand! Which version of Druid are you using?

Druid version - 0.22.1

@Mark_Herrera

It might be a middle manager issue:

NOTE: Although the log4j configuration file is shared with Druid’s task peon processes, the appenders in this file DO NOT take effect for peon processes, which always output logs to standard output. Middle Managers redirect task logs from standard output to long-term storage.

However, log level settings do take effect for these task peon processes, which means you can still configure loggers at different logging level for task logs using log4j2.xml.

Can you check the middle manager configuration and runtime properties, specifically the druid.emitter.logging.logLevel? info is the default, and I’ve heard about debug leading to the issue you’re describing.

Adding to @Mark_Herrera – could you confirm which log files are consuming most space? There are two types of logging in Druid – there’s the process logs themselves

and then there are the ingestion task logs.

Last time one of my junior collegue deleted below logs to make spcae available.

1870 rm broker.log
1871 rm coordinator-overlord.log
1872 rm historical.log

All these files are under

/root/druid/druid/var/sv/

The other day historical.log has grown in size (it was few Gb…or below that), since I last edited the xml…its now limited to few Kb

root@dbsystems1:~/druid/druid/var/sv# ls -lht
total 23M
-rw-r–r-- 1 root root 2.7M Aug 4 04:17 zk.log
-rw-r–r-- 1 root root 4.2M Aug 3 17:16 middleManager.log
-rw-r–r-- 1 root root 16M Aug 3 17:16 broker.log
-rw-r–r-- 1 root root 26K Aug 3 17:15 historical.log
-rw-r–r-- 1 root root 420K Dec 29 2021 router.log

@Mark_Herrera @petermarshallio