Where would I see error log of an index task from tranquility?

I see that some of the tasks FAILED on the coordinator but I would like to know why.

When I clicked on the log link I only see this

No log was found for this task. The task may not exist, or it may not have begun running yet.
And when I logged on to the the middle manager and look at the log I only see something like this.


3435.504: [GC pause (young), 0.0449170 secs] [Parallel Time: 27.9 ms, GC Workers: 8] [GC Worker Start (ms): Min: 3435505.1, Avg: 3435505.2, Max: 3435505.3, Diff: 0.2] [Ext Root Scanning (ms): Min: 2.6, Avg: 3.0, Max: 4.1, Diff: 1.5, Sum: 24.4] [Update RS (ms): Min: 3.9, Avg: 5.5, Max: 9.7, Diff: 5.8, Sum: 44.0] [Processed Buffers: Min: 0, Avg: 8.0, Max: 19, Diff: 19, Sum: 64] [Scan RS (ms): Min: 0.3, Avg: 2.7, Max: 3.4, Diff: 3.1, Sum: 21.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 2.0] [Object Copy (ms): Min: 14.4, Avg: 15.6, Max: 16.1, Diff: 1.7, Sum: 124.6] [Termination (ms): Min: 0.2, Avg: 0.4, Max: 0.5, Diff: 0.3, Sum: 3.1] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.2, Diff: 0.2, Sum: 1.2] [GC Worker Total (ms): Min: 27.5, Avg: 27.6, Max: 27.8, Diff: 0.3, Sum: 220.8] [GC Worker End (ms): Min: 3435532.7, Avg: 3435532.8, Max: 3435532.9, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Migration: 1.3 ms] [Clear CT: 1.6 ms] [Other: 14.1 ms] [Choose CSet: 0.0 ms] [Ref Proc: 1.0 ms] [Ref Enq: 0.0 ms] [Free CSet: 11.5 ms] [Eden: 7361.0M(7361.0M)->0.0B(7361.0M) Survivors: 22.0M->19.0M Heap: 14.0G(15.6G)->6933.8M(15.6G)] [Times: user=0.20 sys=0.02, real=0.04 secs]

``




And this is what I would see from the overlord


Status{id=index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_0_0, status=FAILED, duration=0}
2016-06-16T21:10:50,645 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Task done: RealtimeIndexTask{id=index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_0_0, type=index_realtime, dataSource=test-sparrow}
2016-06-16T21:10:50,645 INFO [Curator-PathChildrenCache-3] LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-06-16T21:10:50.645Z","service":"druid/overlord","host":"[:8090](http://www.google.com/url?q=http%3A%2F%2F10.90.27.71%3A8090&sa=D&sntz=1&usg=AFQjCNE5JMwLTBsY0KEfsUCFPFgeXBivrQ)","metric":"task/run/time","value":0,"dataSource":"test-sparrow","taskStatus":"FAILED","taskType":"index_realtime"}]
2016-06-16T21:10:50,645 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Task FAILED: RealtimeIndexTask{id=index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_0_0, type=index_realtime, dataSource=test-sparrow} (0 run duration)
2016-06-16T21:10:50,645 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_0_0] went bye bye.
2016-06-16T21:11:13,107 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Worker[[:8080](http://www.google.com/url?q=http%3A%2F%2F10.90.25.146%3A8080&sa=D&sntz=1&usg=AFQjCNF1sxGAPvW24XW3SCNI3KvvzjdYGA)] wrote FAILED status for task: index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1
2016-06-16T21:11:13,107 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Worker[[:8080](http://www.google.com/url?q=http%3A%2F%2F10.90.25.146%3A8080&sa=D&sntz=1&usg=AFQjCNF1sxGAPvW24XW3SCNI3KvvzjdYGA)] completed task[index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1] with status[FAILED]
2016-06-16T21:11:13,107 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1
2016-06-16T21:11:13,107 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1] on worker[[:8080](http://www.google.com/url?q=http%3A%2F%2F10.90.25.146%3A8080&sa=D&sntz=1&usg=AFQjCNF1sxGAPvW24XW3SCNI3KvvzjdYGA)]
2016-06-16T21:11:13,110 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskLockbox - Removing task[index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1] from activeTasks
2016-06-16T21:11:13,110 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskLockbox - Removing task[index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1] from TaskLock[index_realtime_test-sparrow]
2016-06-16T21:11:13,111 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[3592]: TaskLock{groupId=index_realtime_test-sparrow, dataSource=test-sparrow, interval=2016-06-16T19:00:00.000Z/2016-06-16T20:00:00.000Z, version=2016-06-16T20:09:36.944Z}
2016-06-16T21:11:13,127 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1 to status: TaskStatus{id=index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1, status=FAILED, duration=0}
2016-06-16T21:11:13,143 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Task done: RealtimeIndexTask{id=index_realtime_test-sparrow_2016-06-16T19:00:00.000Z_2_1, type=index_realtime, dataSource=test-sparrow}

Are you sure there’s no exceptions in the middle manager logs? If this is a clustered Druid setup, did you set up writing task logs to deep storage?

Hi Fangjin,

I’m also facing the same issue - ie, I’m not able to find any logs for failed tasks.

I’m running a clustered Druid setup. How do I verify if I’ve set up writing task logs to deep storage or not?

Thanks,

Jithin

Hi Jithin,
the configs for writing task logs to deep storage can be found here -

http://druid.io/docs/latest/configuration/indexing-service.html

Look for Task Logging.

Hi, Fangjin Yang, Nishant Bangarwa

I am facing a similar situationg. I can find the log file as I specified in conf file (local file), however, there’s only some GC related log, useless for debugging.

I don’t know if you have solved the problem. I encountered the same issue. In my case i configed the log4j2.xml for druid, to let druid components output logs to specified log file. However, indexing task would inherit the middlemanager’s config, in which case the indexing log was actually output to the middlemanager log.

My solution is define a variable in log4j2:

<Appenders>
    <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
    </Console>
...
<Loggers>
    <Root level="${sys:druid.log.level}">
        <AppenderRef ref="${sys:druid.log.ref}"/>
    </Root>
``` ```

and use different -Ddruid.log.ref configs for middlemanager and task. For middlemanager, just start it with java -cp … -Ddruid.log.ref=<Name_of_appender> …

and for task, use -Ddruid.indexer.fork.property.druid.log.ref=Console instead.