druid indexing server can't generate task log

Hi, ALL:

I set the parameter “druid.indexer.logs.directory” to “/tmp/logs/tasks”, there is no task log generate under this folder,

and can’t see the task log from UI, Can somebody help to me to get these task serving log?

Thanks

What is the value of
druid.indexer.runner.type
`druid.indexer.logs.type

`

`note that logs must be stored on hdfs/s3 if druid.indexer.runner.type=remote .

`

`see http://druid.io/docs/0.7.3/Indexing-Service-Config.html#task-logging

`

`-- Himanshu```

Thanks,Himanshu. I thought it will give the default value for these parameters. I set “druid.indexer.runner.type” to “local” and “druid.indexer.logs.type” to “file”, there are the logs under the folder now.
But the content log is not enough, I think it will logging more details, for example, the rate of map-reducer job progress.

It only have these content:

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/luotao/software/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/luotao/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

15/05/29 05:18:40 INFO util.Version: HV000001: Hibernate Validator 5.1.3.Final

May 29, 2015 5:18:46 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class

May 29, 2015 5:18:46 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering io.druid.server.StatusResource as a root resource class

May 29, 2015 5:18:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate

INFO: Initiating Jersey application, version ‘Jersey: 1.17.1 02/28/2013 12:47 PM’

May 29, 2015 5:18:46 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope “Singleton”

May 29, 2015 5:18:47 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider

May 29, 2015 5:18:47 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider

May 29, 2015 5:18:47 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope “Undefined”

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/luotao/software/druid-0.7.1.1/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/luotao/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/luotao/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

15/05/29 05:18:48 INFO util.Version: HV000001: Hibernate Validator 5.1.3.Final

by the way, are there some parameters for setting the number of retain logs, for example, only retain recent 10 tasks’ log?

Thanks,

Tao

在 2015年5月29日星期五 UTC+8下午12:48:19,Himanshu Gupta写道:

That is strange a bit, I do believe that default values are “local” and “file” for those. Originally, did you try to restart your overlord after setting the “druid.indexer.logs.directory”?

Also, there is no cleanup from druid itself. You will have to manage that externally.

– Himanshu

There are logs in dir now, but the content of logs are simpler than I firstly start it, for example, lack of logging the process of indexing build.

Does it caused by changing the start command? Here is the command to start druid index:

java -server -Xmx8g -Xms8g -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Djava.io.tmpdir=/tmp/io -classpath /home/luotao/hadoop_conf:/home/luotao/software/druid-0.7.1.1/config/_common:/home/luotao/software/druid-0.7.1.1/config/overlord:/home/luotao/software/druid-0.7.1.1/lib/* io.druid.cli.Main server overlord

Thanks,

Tao

在 2015年5月29日星期五 UTC+8下午10:30:17,Himanshu Gupta写道:

Hi Tao, I’m not entirely sure what you are asking. Druid task logs are stored in the local filesystem by default, and should be accessable via the indexing console at overlord_ip:port/console.html. You can also load logs into deep storage by configuration the appropriate configurations. Examples here: http://druid.io/docs/0.7.3/Production-Cluster-Configuration.html