Merge task failed with a java.lang.NullPointerException

I used druid 0.8.1 stable, and use S3 as deep storage.

When merge small segment after batch ingestion(or realtime ingestion), the task failed with a java.lang.NullPointerException:

2015-09-22T20:04:23,384 INFO [task-runner-0] io.druid.storage.s3.S3DataSegmentPuller - Loaded 9378 bytes from [s3://bucket/druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip] to [/mnt/persistent/task/merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00/work/batch_csv2/2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0]

2015-09-22T20:04:23,388 ERROR [task-runner-0] io.druid.indexing.common.task.MergeTaskBase - Exception merging[batch_csv2]: {class=io.druid.indexing.common.task.MergeTaskBase, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, interval=2013-08-28T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00}

java.lang.NullPointerException

at io.druid.indexing.common.task.MergeTask.merge(MergeTask.java:65) ~[druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:146) [druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.8.1.jar:0.8.1]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_85]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_85]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_85]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_85]

2015-09-22T20:04:23,402 INFO [task-runner-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2015-09-22T20:04:23.395-07:00”,“service”:“druid/middleManager”,“host”:“ip-10-15-36-109.us-west-1.compute.internal:8101”,“severity”:“component-failure”,“description”:“Exception merging[batch_csv2]”,“data”:{“class”:“io.druid.indexing.common.task.MergeTaskBase”,“exceptionType”:“java.lang.NullPointerException”,“exceptionMessage”:null,“exceptionStackTrace”:“java.lang.NullPointerException\n\tat io.druid.indexing.common.task.MergeTask.merge(MergeTask.java:65)\n\tat io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:146)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n”,“interval”:“2013-08-28T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00”}}]

2015-09-22T20:04:23,403 INFO [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Removing task directory: /mnt/persistent/task/merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00/work

2015-09-22T20:04:23,418 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“status” : “FAILED”,

“duration” : 785

}

Merge task config (got from task log):

{

“type” : “merge”,

“id” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“dataSource” : “batch_csv2”,

“segments” : [ {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-28T00:00:00.000-07:00/2013-08-29T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",

 "bucket" : "bucket",

 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-28T00:00:00.000-07:00_2013-08-29T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

“size” : 9230,

“identifier” : “batch_csv2_2013-08-28T00:00:00.000-07:00_2013-08-29T00:00:00.000-07:00_2015-09-21T19:48:45.162-07:00”

}, {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-29T00:00:00.000-07:00/2013-08-30T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",

 "bucket" : "bucket",

 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-29T00:00:00.000-07:00_2013-08-30T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

“size” : 9855,

“identifier” : “batch_csv2_2013-08-29T00:00:00.000-07:00_2013-08-30T00:00:00.000-07:00_2015-09-21T19:48:45.162-07:00”

}, {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-30T00:00:00.000-07:00/2013-08-31T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",

 "bucket" : "bucket",

 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-30T00:00:00.000-07:00_2013-08-31T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

“size” : 9608,

“identifier” : “batch_csv2_2013-08-30T00:00:00.000-07:00_2013-08-31T00:00:00.000-07:00_2015-09-21T19:48:45.162-07:00”

}, {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-31T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",

 "bucket" : "bucket",

 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

“size” : 9378,

“identifier” : “batch_csv2_2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00_2015-09-21T19:48:45.162-07:00”

} ],

“aggregations” : null,

“groupId” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“interval” : “2013-08-28T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00”,

“resource” : {

“availabilityGroup” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“requiredCapacity” : 1

}

}

``

I have no idea about this, could you help me?

Hmmm, I wonder if the work on delta ingestion has impacted this feature.

@Himanshu, can you take a look?

Actually I don’t think delta ingestion is related. Can you include the full log including any messages before and after these logs lines to look at? I’m not entirely sure what is happening.

The NPE is happening because you have not provided “aggregations” with the merge task. Druid needs them so it knows how to combine the metric columns in the underlying segments.

Fangjin Yang, Thank you. Attachment is full log.

在 2015年9月25日星期五 UTC+8上午8:30:37,Fangjin Yang写道:

Actually I don’t think delta ingestion is related. Can you include the full log including any messages before and after these logs lines to look at? I’m not entirely sure what is happening.

Hmmm, I wonder if the work on delta ingestion has impacted this feature.

@Himanshu, can you take a look?

I used druid 0.8.1 stable, and use S3 as deep storage.

When merge small segment after batch ingestion(or realtime ingestion), the task failed with a java.lang.NullPointerException:

2015-09-22T20:04:23,384 INFO [task-runner-0] io.druid.storage.s3.S3DataSegmentPuller - Loaded 9378 bytes from [s3://bucket/druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip] to [/mnt/persistent/task/merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00/work/batch_csv2/2013-08-31T00:00:00.000-07:00_2013-09-01T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0]

2015-09-22T20:04:23,388 ERROR [task-runner-0] io.druid.indexing.common.task.MergeTaskBase - Exception merging[batch_csv2]: {class=io.druid.indexing.common.task.MergeTaskBase, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, interval=2013-08-28T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00}

java.lang.NullPointerException

at io.druid.indexing.common.task.MergeTask.merge(MergeTask.java:65) ~[druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:146) [druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.8.1.jar:0.8.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.8.1.jar:0.8.1]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_85]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_85]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_85]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_85]

2015-09-22T20:04:23,402 INFO [task-runner-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2015-09-22T20:04:23.395-07:00”,“service”:“druid/middleManager”,“host”:“ip-10-15-36-109.us-west-1.compute.internal:8101”,“severity”:“component-failure”,“description”:“Exception merging[batch_csv2]”,“data”:{“class”:“io.druid.indexing.common.task.MergeTaskBase”,“exceptionType”:“java.lang.NullPointerException”,“exceptionMessage”:null,“exceptionStackTrace”:“java.lang.NullPointerException\n\tat io.druid.indexing.common.task.MergeTask.merge(MergeTask.java:65)\n\tat io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:146)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n”,“interval”:“2013-08-28T00:00:00.000-07:00/2013-09-01T00:00:00.000-07:00”}}]

2015-09-22T20:04:23,403 INFO [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Removing task directory: /mnt/persistent/task/merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00/work

2015-09-22T20:04:23,418 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“status” : “FAILED”,

“duration” : 785

}

Merge task config (got from task log):

{

“type” : “merge”,

“id” : “merge_batch_csv2_0f464c869e5624c59b97cd39dc94f160e2837c39_2015-09-22T20:04:14.522-07:00”,

“dataSource” : “batch_csv2”,

“segments” : [ {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-28T00:00:00.000-07:00/2013-08-29T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",
 "bucket" : "bucket",
 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-28T00:00:00.000-07:00_2013-08-29T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

“size” : 9230,

“identifier” : “batch_csv2_2013-08-28T00:00:00.000-07:00_2013-08-29T00:00:00.000-07:00_2015-09-21T19:48:45.162-07:00”

}, {

“dataSource” : “batch_csv2”,

“interval” : “2013-08-29T00:00:00.000-07:00/2013-08-30T00:00:00.000-07:00”,

“version” : “2015-09-21T19:48:45.162-07:00”,

“loadSpec” : {

 "type" : "s3_zip",
 "bucket" : "bucket",
 "key" : "druid_v0.8.1/peon/batch_csv2/batch_csv2/2013-08-29T00:00:00.000-07:00_2013-08-30T00:00:00.000-07:00/2015-09-21T19:48:45.162-07:00/0/index.zip"

},

“dimensions” : “region,anonymous,unpatrolled,page,continent,newPage,robot,language,user,namespace,country,city”,

“metrics” : “count,added,deleted,delta”,

“shardSpec” : {

 "type" : "none"

},

“binaryVersion” : 9,

``

在 2015年9月25日星期五 UTC+8上午8:30:37,Fangjin Yang写道:

merge_task_log (64.2 KB)

Do you mean add “metricsSpec” field? (“aggregations” only used in query?)

There is no aggregation field in document: http://druid.io/docs/latest/misc/tasks.html#merge-task
{
“type”: “merge”,
“id”: <task_id>,
“dataSource”: <task_datasource>,
“segments”:
}

``

在 2015年9月25日星期五 UTC+8上午8:32:32,Gian Merlino写道:

Hi Wb,

You have to add an attribute “aggregations” in your task json that will be a list of aggregators similar to what you would put in “metricsSpec” in an indexing task.

That said, I think this merge task is really hard to use given that you have to specify list of segments. If you use hadoop based indexing then re-indexing via “dataSource” inputSpec might be a better option (pls see “dataSource” inputSpec on http://druid.io/docs/0.8.1/ingestion/batch-ingestion.html )

– himanshu

OK, I’ll try both of method (add aggregations and re-index).

Thank you very much, Himanshu.

在 2015年9月25日星期五 UTC+8上午11:24:24,Himanshu Gupta写道: