Configure separate log file for each daemon

I want to configure a separate log file for each druid daemon like middlemanager.log , coordinator.log etc.
I am aware of the fact that druid uses log4j for logging hence i tried creating a log4j.properties for my use case but got stuck while figuring out the logger name for each of these daemons?

From the code of druid server i figured that druid uses the classname as the logger name hence i tried to configure logger like “io.druid.server.coordinator” for coordinator but this doesn’t seem to work.

I also observed that the peons uses the same log4j.properties as used by middlemanager hence both the daemons log to same file,how do i pass a seperate log configuration file to peons?

Thanks

Rohit

Hi Rohit, for each of the Druid processes, once you start them up, they should be generating their own log that you can pipe into a separate log file. Perhaps I am a bit confused about what you are trying to do.

Hello Fangjin,
My objective here is to define a “RollingFile” appender for each daemon.

The default log4j2.xml that is bundled with druid has only “root” logger defined and only console appender.

Hence we need to pipe the output of each daemon to a separate log file but doing so doesn’t provide us the capabilities of defining log retention.

The default log4j2.xml is like this :

<Appenders>
    <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
    </Console>
</Appenders>
<Loggers>
    <Root level="info">
        <AppenderRef ref="Console"/>
    </Root>
</Loggers>

I created a separate log4j2.xml for each daemon and added a Rolling appender to each config file.My log4j2.xml looks like this

<Appenders>
     <RollingFile name="root" fileName="log/coordinator.log" filePattern="log/coordinator.%i.log" >
            <PatternLayout>
                    <pattern>%d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n</pattern>
            </PatternLayout>
            <Policies>
                    <SizeBasedTriggeringPolicy size="1 MB" />
            </Policies>
    <DefaultRolloverStrategy max="9"/>
    </RollingFile>
</Appenders>
<Loggers>
    <Root level="info">
        <AppenderRef ref="root"/>
    </Root>
</Loggers>

The problem with such a approach is that all peons add the entire classpath of middlemanager to its path hence even the log4j2.xml of middlemanager gets added to peons path therefore all the logs of “realtime_indexing” task gets logged to middle manager’s log file.

Whereas with default log4j2.xml all such task logs were logged in separate log file which was present in “{DRUID_HOME}/log” directory.

In the ideal scenario it should be possible for me to declare one single log4j2.xml for all the daemons,which has multiple loggers and appenders(RollingFile) defined 1 for each daemon such that each daemon logs in its own file.

I also tried with this log4j2.xml which has logger defined for all daemons,this didn’t work at all.

<?xml version="1.0" encoding="UTF-8"?> %d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n %d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n

With the above log4j2.xml none of the logs were printed to corresponding files probably the logger names are incorrect.

It would be of great help if you can let me know how to configure druid’s logging such that each daemon has it own log file AND i have the capability to define retention on log files.

Thanks

Rohit

Good question Rohit.

http://logging.apache.org/log4j/2.x/manual/configuration.html#PropertySubstitution

Have you tinkered with property substitution at all? A natural choice would be the service name.

I have answered this similar question here - https://groups.google.com/forum/#!topic/druid-user/b8CjDaIVuKQ