Indexing task failure/Segments are not published

Hi
I am running six node Druid cluster with nfs as deep storage. I have ingested a Kafka topic and can see all the segments are available in the druid(datasource) console. The zookeeper(1 instance) is attached with master 1.

However, i am experiencing indexing task failure(around 70%) and the segments are never published. Few indexing task also completed successfully. One the indexing task is over i can see all the segments are available as realtime(true) and is_published is false.

Another thing i have noticed that the druid storage directory which is configured as var/druid_share/segments which is nfs mount is empty.

Task log

{
  "id": "index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel",
  "groupId": "index_kafka_cdc_icx_sessions",
  "type": "index_kafka",
  "createdTime": "2022-08-31T00:20:14.086Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": -1,
  "location": {
    "host": "10.10.0.50",
    "port": 8100,
    "tlsPort": -1
  },
  "dataSource": "cdc_icx_sessions",
  "errorMsg": "No task in pending completion taskGroup[0] succeeded before completion timeout elapsed"
}

Complete task logs

....
....
2022-08-31T01:22:40,750 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2019-10-04T00:00:00.000Z_2019-10-05T00:00:00.000Z_2022-08-20T15:45:18.652Z_124] spill[0] to disk in [17,521] ms (1 rows).
2022-08-31T01:22:59,502 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2021-06-03T00:00:00.000Z_2021-06-04T00:00:00.000Z_2022-08-20T15:46:11.322Z_124] spill[0] to disk in [18,734] ms (6 rows).
2022-08-31T02:04:02,895 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2019-10-21T00:00:00.000Z_2019-10-22T00:00:00.000Z_2022-08-20T15:45:18.978Z_124] spill[0] to disk in [15,212] ms (1 rows).
2022-08-31T02:04:20,879 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2019-11-24T00:00:00.000Z_2019-11-25T00:00:00.000Z_2022-08-20T15:45:20.173Z_124] spill[0] to disk in [17,971] ms (1 rows).
2022-08-31T02:04:38,074 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2020-12-15T00:00:00.000Z_2020-12-16T00:00:00.000Z_2022-08-20T15:45:56.388Z_124] spill[0] to disk in [17,187] ms (1 rows).
2022-08-31T02:04:55,193 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2021-07-17T00:00:00.000Z_2021-07-18T00:00:00.000Z_2022-08-20T15:46:14.517Z_124] spill[0] to disk in [17,108] ms (2 rows).

2022-08-31T02:20:17,277 INFO [[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data for segment[cdc_icx_sessions_2022-08-19T00:00:00.000Z_2022-08-20T00:00:00.000Z_2022-08-20T15:46:27.607Z_124] spill[0] to disk in [15,882] ms (8 rows).
2022-08-31T02:20:28,912 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
2022-08-31T02:20:28,914 INFO [Thread-35] org.apache.druid.cli.CliPeon - Running shutdown hook
2022-08-31T02:20:28,915 INFO [Thread-35] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2022-08-31T02:20:28,926 INFO [Thread-35] org.apache.druid.curator.announcement.Announcer - Unannouncing [/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,125 INFO [Thread-35] org.apache.druid.curator.announcement.Announcer - Unannouncing [/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,150 INFO [Thread-35] org.apache.druid.curator.announcement.Announcer - Unannouncing [/data_fluent_test/internal-discovery/PEON/10.10.0.50:8100]
2022-08-31T02:20:29,171 INFO [Thread-35] org.apache.druid.curator.announcement.Announcer - Unannouncing [/data_fluent_test/announcements/10.10.0.50:8100]
2022-08-31T02:20:29,192 INFO [Thread-35] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2022-08-31T02:20:29,203 INFO [Thread-35] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@7846913f{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2022-08-31T02:20:29,204 INFO [Thread-35] org.eclipse.jetty.server.session - node0 Stopped scavenging
2022-08-31T02:20:29,206 INFO [Thread-35] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@351e86b2{/,null,STOPPED}
2022-08-31T02:20:29,211 INFO [Thread-35] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2022-08-31T02:20:29,211 INFO [Thread-35] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='10.10.0.50:8100', hostAndPort='10.10.0.50:8100', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2022-08-31T02:20:29,211 INFO [Thread-35] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2022-08-31T02:20:29,211 INFO [Thread-35] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2022-08-31T02:20:29,214 INFO [Thread-35] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_cdc_icx_sessions_42c958f14d7f228_oamhpiel].
2022-08-31T02:20:29,214 INFO [Thread-35] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [PUBLISHING])
2022-08-31T02:20:29,214 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-kafka-supervisor-jjgcdoec-1, groupId=kafka-supervisor-jjgcdoec] Resetting generation due to: consumer pro-actively leaving the group
2022-08-31T02:20:29,214 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-kafka-supervisor-jjgcdoec-1, groupId=kafka-supervisor-jjgcdoec] Request joining group due to: consumer pro-actively leaving the group
2022-08-31T02:20:29,215 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed
2022-08-31T02:20:29,215 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2022-08-31T02:20:29,215 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Metrics reporters closed
2022-08-31T02:20:29,223 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-kafka-supervisor-jjgcdoec-1 unregistered
2022-08-31T02:20:29,229 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-06-04T00:00:00.000Z_2021-06-05T00:00:00.000Z_2022-08-20T15:46:11.537Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:04.355Z_dc200c59cebd444e851944031f3a99db6]
2022-08-31T02:20:29,230 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-11-17T00:00:00.000Z_2021-11-18T00:00:00.000Z_2022-08-20T15:46:22.741Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:16.691Z_a6ebd55c5e31456d9b3e76beee5d5ba37]
2022-08-31T02:20:29,232 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-02-15T00:00:00.000Z_2020-02-16T00:00:00.000Z_2022-08-20T15:45:29.600Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:44.655Z_c1e04c124248424483ad4813791aff811]
2022-08-31T02:20:29,234 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-06-07T00:00:00.000Z_2021-06-08T00:00:00.000Z_2022-08-20T15:46:12.095Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:04.355Z_dc200c59cebd444e851944031f3a99db6]
2022-08-31T02:20:29,236 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-07-02T00:00:00.000Z_2021-07-03T00:00:00.000Z_2022-08-20T15:46:13.381Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:04.355Z_dc200c59cebd444e851944031f3a99db6]
2022-08-31T02:20:29,238 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-12-19T00:00:00.000Z_2020-12-20T00:00:00.000Z_2022-08-20T15:45:56.761Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:37.205Z_4489764f17ed498d86d8cd36744bcbb24]
2022-08-31T02:20:29,238 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2022-08-31T02:20:29,239 INFO [Thread-35] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
2022-08-31T02:20:29,239 INFO [Thread-35] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
2022-08-31T02:20:29,239 INFO [Thread-35] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
2022-08-31T02:20:29,239 INFO [Thread-35] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
2022-08-31T02:20:29,241 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-05-29T00:00:00.000Z_2021-05-30T00:00:00.000Z_2022-08-20T15:46:10.485Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,242 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2022-08-31T02:20:29,244 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2019-10-04T00:00:00.000Z_2019-10-05T00:00:00.000Z_2022-08-20T15:45:18.652Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,246 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-06-03T00:00:00.000Z_2021-06-04T00:00:00.000Z_2022-08-20T15:46:11.322Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,249 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2010-12-07T00:00:00.000Z_2010-12-08T00:00:00.000Z_2022-08-20T15:45:12.537Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,251 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-11-27T00:00:00.000Z_2020-11-28T00:00:00.000Z_2022-08-20T15:45:53.782Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:37.205Z_4489764f17ed498d86d8cd36744bcbb24]
2022-08-31T02:20:29,254 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-03-03T00:00:00.000Z_2020-03-04T00:00:00.000Z_2022-08-20T15:45:32.779Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:02.951Z_284bed7e7c7644e0beb943d43d2c03732]
2022-08-31T02:20:29,256 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-04-16T00:00:00.000Z_2021-04-17T00:00:00.000Z_2022-08-20T15:46:04.328Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,258 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2022-04-19T00:00:00.000Z_2022-04-20T00:00:00.000Z_2022-08-20T15:46:25.728Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:16.691Z_a6ebd55c5e31456d9b3e76beee5d5ba37]
2022-08-31T02:20:29,261 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-08-06T00:00:00.000Z_2020-08-07T00:00:00.000Z_2022-08-20T15:45:48.417Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,264 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-02-28T00:00:00.000Z_2020-02-29T00:00:00.000Z_2022-08-20T15:45:32.034Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:02.951Z_284bed7e7c7644e0beb943d43d2c03732]
2022-08-31T02:20:29,266 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-03-09T00:00:00.000Z_2020-03-10T00:00:00.000Z_2022-08-20T15:45:33.680Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:02.951Z_284bed7e7c7644e0beb943d43d2c03732]
2022-08-31T02:20:29,268 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2022-01-27T00:00:00.000Z_2022-01-28T00:00:00.000Z_2022-08-20T15:46:23.878Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:16.691Z_a6ebd55c5e31456d9b3e76beee5d5ba37]
2022-08-31T02:20:29,270 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-06-16T00:00:00.000Z_2020-06-17T00:00:00.000Z_2022-08-20T15:45:47.611Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,272 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-05-14T00:00:00.000Z_2020-05-15T00:00:00.000Z_2022-08-20T15:45:44.872Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,275 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-08-18T00:00:00.000Z_2020-08-19T00:00:00.000Z_2022-08-20T15:45:48.842Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,277 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-04-14T00:00:00.000Z_2021-04-15T00:00:00.000Z_2022-08-20T15:46:03.891Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,279 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-01-13T00:00:00.000Z_2020-01-14T00:00:00.000Z_2022-08-20T15:45:23.206Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:44.655Z_c1e04c124248424483ad4813791aff811]
2022-08-31T02:20:29,281 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-05-20T00:00:00.000Z_2020-05-21T00:00:00.000Z_2022-08-20T15:45:45.725Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:21.023Z_bacc3f34cad6496d99d6fe09a71692953]
2022-08-31T02:20:29,284 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2010-10-26T00:00:00.000Z_2010-10-27T00:00:00.000Z_2022-08-20T15:45:11.776Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,286 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-05-17T00:00:00.000Z_2021-05-18T00:00:00.000Z_2022-08-20T15:46:09.209Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,288 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-12-07T00:00:00.000Z_2020-12-08T00:00:00.000Z_2022-08-20T15:45:55.359Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:37.205Z_4489764f17ed498d86d8cd36744bcbb24]
2022-08-31T02:20:29,290 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-11-11T00:00:00.000Z_2021-11-12T00:00:00.000Z_2022-08-20T15:46:22.302Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:16.691Z_a6ebd55c5e31456d9b3e76beee5d5ba37]
2022-08-31T02:20:29,292 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2020-03-01T00:00:00.000Z_2020-03-02T00:00:00.000Z_2022-08-20T15:45:32.412Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:02.951Z_284bed7e7c7644e0beb943d43d2c03732]
2022-08-31T02:20:29,294 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2019-09-11T00:00:00.000Z_2019-09-12T00:00:00.000Z_2022-08-20T15:45:16.918Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,297 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2019-11-26T00:00:00.000Z_2019-11-27T00:00:00.000Z_2022-08-20T15:45:20.546Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:44.655Z_c1e04c124248424483ad4813791aff811]
2022-08-31T02:20:29,299 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-02-20T00:00:00.000Z_2021-02-21T00:00:00.000Z_2022-08-20T15:46:00.448Z_124] at path[
2022-08-31T02:20:29,387 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2011-01-04T00:00:00.000Z_2011-01-05T00:00:00.000Z_2022-08-20T15:45:13.655Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31T02:20:29,388 INFO [Thread-35] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
2022-08-31T02:20:29,388 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2022-08-15T00:00:00.000Z_2022-08-16T00:00:00.000Z_2022-08-20T15:46:27.403Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:22:16.691Z_a6ebd55c5e31456d9b3e76beee5d5ba37]
2022-08-31 02:20:29,388 Thread-35 DEBUG Stopping LoggerContext[name=7852e922, org.apache.logging.log4j.core.LoggerContext@61230f6a]
2022-08-31 02:20:29,389 Thread-35 DEBUG Stopping LoggerContext[name=7852e922, org.apache.logging.log4j.core.LoggerContext@61230f6a]...
2022-08-31T02:20:29,390 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2021-04-29T00:00:00.000Z_2021-04-30T00:00:00.000Z_2022-08-20T15:46:06.851Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:21:51.466Z_96578f779a094fa296321dee7653f62a5]
2022-08-31T02:20:29,391 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[cdc_icx_sessions_2012-01-09T00:00:00.000Z_2012-01-10T00:00:00.000Z_2022-08-20T15:45:13.984Z_124] at path[/data_fluent_test/segments/10.10.0.50:8100/10.10.0.50:8100_indexer-executor__default_tier_2022-08-31T00:20:24.310Z_334371b8e33141658ff2350f032e2e030]
2022-08-31 02:20:29,392 Thread-35 DEBUG Shutting down RollingRandomAccessFileManager /u01/single-server/druid/bin/../log/middleManager.log
2022-08-31 02:20:29,392 Thread-35 DEBUG Shutting down RollingFileManager /u01/single-server/druid/bin/../log/middleManager.log
2022-08-31 02:20:29,393 Thread-35 DEBUG All asynchronous threads have terminated
2022-08-31 02:20:29,393 Thread-35 DEBUG RollingFileManager shutdown completed with status true
2022-08-31 02:20:29,393 Thread-35 DEBUG Shut down RollingRandomAccessFileManager /u01/single-server/druid/bin/../log/middleManager.log, all resources released: true
2022-08-31 02:20:29,393 Thread-35 DEBUG Appender FileAppender stopped with status true
2022-08-31 02:20:29,393 Thread-35 DEBUG Shutting down OutputStreamManager SYSTEM_OUT.false.false
2022-08-31 02:20:29,394 Thread-35 DEBUG OutputStream closed
2022-08-31 02:20:29,394 Thread-35 DEBUG Shut down OutputStreamManager SYSTEM_OUT.false.false, all resources released: true
2022-08-31 02:20:29,394 Thread-35 DEBUG Appender Console stopped with status true
2022-08-31 02:20:29,394 Thread-35 DEBUG Stopped OverrideConfiguration[location=/u01/single-server/druid/conf/druid/cluster/_common/log4j2.xml] OK
2022-08-31 02:20:29,394 Thread-35 DEBUG Stopped LoggerContext[name=7852e922, org.apache.logging.log4j.core.LoggerContext@61230f6a] with status true

Common runtime

druid.extensions.loadList=[“druid-hdfs-storage”, “druid-kafka-indexing-service”, “druid-datasketches”, “mysql-metadata-storage”, “druid-avro-extensions”, “druid-basic-security”]

druid.zk.service.host=10.10.0.44:2181
druid.zk.paths.base=/data_fluent_test
druid.zk.service.sessionTimeoutMs=60000
druid.zk.service.connectionTimeoutMs=30000
druid.zk.service.sessionTimeoutMs=60000
druid.zk.service.connectionTimeoutMs=30000

druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://10.10.0.13:3306/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=password

druid.storage.type=local
druid.storage.storageDirectory=var/druid_share/segments

druid.indexer.logs.type=file
druid.indexer.logs.directory=var/druid_share/indexing-logs

druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator

druid.monitoring.monitors=[“org.apache.druid.java.util.metrics.JvmMonitor”]
druid.emitter=noop
druid.emitter.logging.logLevel=debug

druid.indexing.doubleStorage=double

druid.server.hiddenProperties=[“druid.s3.accessKey”,“druid.s3.secretKey”,“druid.metadata.storage.connector.password”]

druid.sql.enable=true

druid.lookup.enableLookupSyncOnStartup=false

druid.auth.authenticatorChain=[“MyBasicMetadataAuthenticator”]
druid.auth.authenticator.MyBasicMetadataAuthenticator.type=basic
druid.auth.authenticator.MyBasicMetadataAuthenticator.initialAdminPassword=password2
druid.auth.authenticator.MyBasicMetadataAuthenticator.initialInternalClientPassword=password2
druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialsValidator.type=metadata
druid.auth.authenticator.MyBasicMetadataAuthenticator.skipOnFailure=false
druid.auth.authenticator.MyBasicMetadataAuthenticator.authorizerName=MyBasicMetadataAuthorizer

druid.escalator.type=basic
druid.escalator.internalClientUsername=druid_system
druid.escalator.internalClientPassword=password2
druid.escalator.authorizerName=MyBasicMetadataAuthorizer

druid.auth.authorizers=[“MyBasicMetadataAuthorizer”]
druid.auth.authorizer.MyBasicMetadataAuthorizer.type=basic

druid.expressions.useStrictBooleans=true

druid.global.http.eagerInitialization=false

Historical

druid.service=druid/historical
druid.plaintextPort=8083

druid.server.http.numThreads=22

druid.processing.buffer.sizeBytes=200MiB
druid.processing.numMergeBuffers=2
druid.processing.numThreads=15

druid.processing.tmpDir=var/druid_share/processing

druid.segmentCache.locations=[{“path”:“var/druid_share/segment-cache”,“maxSize”:“32g”}]

druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=100MiB

Middle manager

druid.service=druid/middleManager
druid.plaintextPort=8091

druid.worker.capacity=15

druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid_share/task

druid.server.http.numThreads=22

druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=200MiB
druid.indexer.fork.property.druid.processing.numThreads=1

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

Coordinator

druid.service=druid/coordinator
druid.plaintextPort=8081

druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S

druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord

druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

=====================================================

Historical/Middle manager log: https://file.io/64LHDXC0k4js

Can any one point out where could be the issue ?

Welcome @Mitchel!

For my own clarification, are some of the segments are being published? Which ones? Are they from the same Kafka topic?

Did something recently change in your cluster or the Kafka topic?

Best,

Mark