Compact task are failing with failed to publish the segments

Hi,

Executing compact task daily for 30 days.

For each day one compact task running,

Some compact tasks are failing with below reason,

java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.druid.java.util.common.ISE: Failed to publish segments.

    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

    at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:738) ~[druid-indexing-service-0.12.3.jar:0.12.3]

    at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:264) ~[druid-indexing-service-0.12.3.jar:0.12.3]

    at io.druid.indexing.common.task.CompactionTask.run(CompactionTask.java:213) ~[druid-indexing-service-0.12.3.jar:0.12.3]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.3.jar:0.12.3]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.3.jar:0.12.3]

    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]

    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

Caused by: java.util.concurrent.ExecutionException: io.druid.java.util.common.ISE: Failed to publish segments.

    at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_181]

    at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_181]

    at io.druid.indexing.common.task.IndexTask.awaitPublish(IndexTask.java:762) ~[druid-indexing-service-0.12.3.jar:0.12.3]

    at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:717) ~[druid-indexing-service-0.12.3.jar:0.12.3]

    ... 8 more

Caused by: io.druid.java.util.common.ISE: Failed to publish segments.

    at io.druid.segment.realtime.appenderator.BaseAppenderatorDriver.lambda$publishInBackground$8(BaseAppenderatorDriver.java:577) ~[druid-server-0.12.3.jar:0.12.3]

Please suggest what can be the reason for failing the compact tasks,

Because of the no of segments are increasing ,query execution taking more time.

Thanks in advance.

Hi Banesh,

It’s not apparent from your log snippet what the cause of the failure is. It would be helpful if you could post the full log of a failed compaction task.

I also noticed that you’re still running Druid 0.12.3. There have been a number of compaction advancements since then: an automated scheduled compaction was added in 0.13.0, additional compaction functionality was added in 0.14.0, and a new web console that has one-click compaction configuration was also added (see: https://druid.apache.org/docs/latest/operations/druid-console.html#datasources). Druid 0.15.0 was just released so it may be a good time to upgrade to take advantage of these new features.