historical.log

Hi,

I am trying to load historical segment to s3 for deep storage. S3 access is done via EC2 assume role. So no need to put access key and secret key there on the conf file. I point both deep storage and index log to the same bucket. The index log got written to the s3, but the deep storage did not. There are errors in historical log. Some errors found on index log (attached) . No errors log found for zk.log and coordinator.log.

Have anyone have any experience on setting up S3 and any idea what might have done wrong on my config? (config files attached)

Any feedback will be appreciated. Thank you!

Historical log:

NOTE: HIstorical node was loading up fine, but very soon it throws error on Failed to load segment for dataSource: … loadSpec={type=>local

2019-04-12T03:47:06,012 INFO [main] org.apache.druid.cli.CliHistorical - Starting up with processors[4], memory[1,029,177,344], maxMemory[1,029,177,344].

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * aws.region: us-east-1

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * awt.toolkit: sun.awt.X11.XToolkit

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.emitter: logging

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.emitter.logging.logLevel: info

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.extensions.loadList: [“postgresql-metadata-storage”,“druid-s3-extensions”]

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.host: localhost

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.indexer.logs.s3Bucket:

2019-04-12T03:47:06,022 INFO [main] org.apache.druid.cli.CliHistorical - * druid.indexer.logs.s3Prefix: druid/indexing-logs

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.indexer.logs.type: s3

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.indexing.doubleStorage: double

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.metadata.storage.connector.connectURI: jdbc:postgresql://…us-east-1.rds.amazonaws.com:5432/druid

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.metadata.storage.connector.password:

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.metadata.storage.connector.user: druid

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.metadata.storage.type: postgresql

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.monitoring.monitors: [“org.apache.druid.java.util.metrics.JvmMonitor”]

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.plaintextPort: 8083

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.processing.buffer.sizeBytes: 256000000

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.processing.numThreads: 2

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.segmentCache.locations: [{“path”:“var/druid/segment-cache”,“maxSize”:300000000000}]

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.selectors.coordinator.serviceName: druid/coordinator

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.selectors.indexing.serviceName: druid/overlord

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.server.http.numThreads: 9

2019-04-12T03:47:06,023 INFO [main] org.apache.druid.cli.CliHistorical - * druid.server.maxSize: 300000000000

2019-04-12T03:47:06,052 INFO [main] org.apache.druid.cli.CliHistorical - * druid.service: druid/historical

2019-04-12T03:47:06,052 INFO [main] org.apache.druid.cli.CliHistorical - * druid.sql.enable: true

2019-04-12T03:47:06,052 INFO [main] org.apache.druid.cli.CliHistorical - * druid.startup.logging.logProperties: true

2019-04-12T03:47:06,052 INFO [main] org.apache.druid.cli.CliHistorical - * druid.storage.baseKey: druid/segments

2019-04-12T03:47:06,052 INFO [main] org.apache.druid.cli.CliHistorical - * druid.storage.bucket:

2019-04-12T03:47:06,053 INFO [main] org.apache.druid.cli.CliHistorical - * druid.storage.type: s3

2019-04-12T03:47:06,053 INFO [main] org.apache.druid.cli.CliHistorical - * druid.zk.paths.base: /druid

2019-04-12T03:47:06,053 INFO [main] org.apache.druid.cli.CliHistorical - * druid.zk.service.host: localhost

2019-04-12T03:47:19,894 ERROR [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[event_stats_2019-04-11T17:00:00.000Z_2019-04-11T18:00:00.000Z_2019-04-11T17:59:03.410Z], segment=DataSegment{size=2077, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[count], dimensions=[eventName, carrierId, source], version=‘2019-04-11T17:59:03.410Z’, loadSpec={type=>local, path=>/home/ec2-user/workspace/apache-druid-0.13.0-incubating/var/druid/segments/event_stats/2019-04-11T17:00:00.000Z_2019-04-11T18:00:00.000Z/2019-04-11T17:59:03.410Z/0/index.zip}, interval=2019-04-11T17:00:00.000Z/2019-04-11T18:00:00.000Z, dataSource=‘event_stats’, binaryVersion=‘9’}}

org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[event_stats_2019-04-11T17:00:00.000Z_2019-04-11T18:00:00.000Z_2019-04-11T17:59:03.410Z]

   at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]

   at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

   at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

   at org.apache.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]

   at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]

   at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]

   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]

   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]

   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]

   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]

   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]

   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]

   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

``

index.log (11.7 KB)

historical.runtime.properties (360 Bytes)

common.runtime.properties (1.14 KB)

coordinator.runtime.properties (123 Bytes)

Hi
I don’t see the extensions section in your common properties file, did you explicitly leave it out?

Regards,

No, it was a copy and paste mistake. This is my line for extension loading. Thanks.

druid.extensions.loadList=[“postgresql-metadata-storage”,“druid-s3-extensions”]

common.runtime.properties (1.22 KB)

I found out the new segments are being inserted to S3 deep storage finally.

The actual mystery is this history log below has the latest timestamp, but the interval was few days back (before i switch over to the s3. I have cleared the /tmp and druid/var/tmp/ before i switch over. What else do I have to clear in order to clear up these zombie events? thanks.

historical.log:2019-04-12T16:20:49,305 ERROR [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[event_stats_2019-04-09T18:00:00.000Z_2019-04-09T19:00:00.000Z_2019-04-09T18:02:03.293Z], segment=DataSegment{size=3098, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[count], dimensions=[eventName, carrierId, source], version=‘2019-04-09T18:02:03.293Z’, loadSpec={type=>local, path=>/home/ec2-user/workspace/apache-druid-0.13.0-incubating/var/druid/segments/event_stats/2019-04-09T18:00:00.000Z_2019-04-09T19:00:00.000Z/2019-04-09T18:02:03.293Z/0/index.zip}, interval=2019-04-09T18:00:00.000Z/2019-04-09T19:00:00.000Z, dataSource=‘event_stats’, binaryVersion=‘9’}}

``