Failed to publish segments

Hi,

I have a distributed setup with 3 middle managers, 3 historicals, and co-ordinator/overlord running in the same container. Druid version is 0.15.0-incubating

I am trying to consume messages from a kafka topic into a single datasource.

The indexing task is able to fetch payloads from kafka but when it comes time to publishing the segments, things go downhill.

Following are the log messages from the indexing task. Any help is appreciated. I am only running a single indexing task. Let me know if any more logs are required to troubleshoot.

org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!].
	at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.lambda$publishInBackground$8(BaseAppenderatorDriver.java:602) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-07-29T19:54:56,632 ERROR [publish-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=0, sequenceName='index_kafka_s1mme_26ab5987c8650ac_0', assignments=[], startOffsets={0=0, 1=0, 2=0, 3=0}, exclusiveStartPartitions=[], endOffsets={0=100483, 1=84960, 2=95422, 3=113598}, sentinel=false, checkpointed=true}]
org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!].
	at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.lambda$publishInBackground$8(BaseAppenderatorDriver.java:602) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-07-29T19:54:56,639 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Shutting down immediately...
2019-07-29T19:54:56,648 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[s1mme_2019-07-29T00:00:00.000Z_2019-07-30T00:00:00.000Z_2019-07-29T18:27:32.739Z] at path[/druid/segments/10.0.1.172:8100/10.0.1.172:8100_indexer-executor__default_tier_2019-07-29T19:44:06.547Z_5268521a0d874a539822c2e8c12ee2c60]
2019-07-29T19:54:56,649 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - unannouncing [/druid/segments/10.0.1.172:8100/10.0.1.172:8100_indexer-executor__default_tier_2019-07-29T19:44:06.547Z_5268521a0d874a539822c2e8c12ee2c60]
2019-07-29T19:54:56,691 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Unregistering chat handler[index_kafka_s1mme_26ab5987c8650ac_knilaidp]
2019-07-29T19:54:56,691 INFO [task-runner-0-priority-0] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannouncing [DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/middleManager', host='10.0.1.172', bindOnHost=false, port=-1, plaintextPort=8100, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType='PEON', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, type=indexer-executor, priority=0}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}].
2019-07-29T19:54:56,691 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - unannouncing [/druid/internal-discovery/PEON/10.0.1.172:8100]
2019-07-29T19:54:56,708 INFO [task-runner-0-priority-0] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced [DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/middleManager', host='10.0.1.172', bindOnHost=false, port=-1, plaintextPort=8100, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType='PEON', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, type=indexer-executor, priority=0}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}].
2019-07-29T19:54:56,709 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - unannouncing [/druid/announcements/10.0.1.172:8100]
2019-07-29T19:54:56,712 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
java.util.concurrent.ExecutionException: org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!].
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:753) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Aborting transaction!].
	at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.lambda$publishInBackground$8(BaseAppenderatorDriver.java:602) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
	... 4 more
2019-07-29T19:54:56,719 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_s1mme_26ab5987c8650ac_knilaidp] status changed to [FAILED].
2019-07-29T19:54:56,721 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_s1mme_26ab5987c8650ac_knilaidp",
  "status" : "FAILED",
  "duration" : 651319,
  "errorMsg" : "java.util.concurrent.ExecutionException: org.apache.druid.java.util.common.ISE: Failed to publish se..."
}

Please use ` index_kafka_s1mme_26ab5987c8650ac_0` and check what’s happening in overlord log and the middlemanager logs. That could give further clue on what’s happening.

Now I have the same problem,what happened to this problem later?

在 2019年7月30日星期二 UTC+8上午5:37:26,Ankush写道: