Problem with druid cluster

Hello everybody !

I have just installed a druid cluster with one Master Server, one Query Server and one data Server.

I wanted to add a second node (i.e. a second data Server). When I add it, it only add a MiddleManager. As you can see in the picture I still only have one historical. How can it be ?

If someone knows it would be cool !

Thanks for advance !

If you login to the data server and check for processes most likely the historical process is not running. This could be related to memory. Check the historical log.

Eric Graham

Solutions Engineer -** **Imply

**cell: **303-589-4581

email: eric.graham@imply.io

www.imply.io

Hi Eric,

When I start the druid cluster, the console shows 2 historicals but when I load data (the wikipedia pack from the tutorial) ti only shows one.

When I go into the log of the second we can see this, what do you think :

2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-01T14:16:12.677Z/0/index.zip}, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, dataSource=‘tutorial’, binaryVersion=‘9’}}
org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[tutorial_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-01T14:16:12.677Z]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:268) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:312) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.ZkCoordinator.lambda$childAdded$0(ZkCoordinator.java:152) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.lang.IllegalArgumentException: Instantiation of [simple type, class org.apache.druid.segment.loading.LocalLoadSpec] value failed: [/home/clement/Documents/apache-druid-0.15.0-incubating/var/druid/segments/tutorial/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-01T14:16:12.677Z/0/index.zip] does not exist
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3459) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3378) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:235) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:224) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:187) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:164) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:131) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:264) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
… 8 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.druid.segment.loading.LocalLoadSpec] value failed: [/home/clement/Documents/apache-druid-0.15.0-incubating/var/druid/segments/tutorial/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-01T14:16:12.677Z/0/index.zip] does not exist
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:399) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:231) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:442) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1099) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:296) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:122) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:93) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:131) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:42) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3454) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3378) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:235) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:224) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:187) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:164) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:131) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:264) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
… 8 more
Caused by: java.lang.IllegalArgumentException: [/home/clement/Documents/apache-druid-0.15.0-incubating/var/druid/segments/tutorial/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-01T14:16:12.677Z/0/index.zip] does not exist
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:148) ~[guava-16.0.1.jar:?]
at org.apache.druid.segment.loading.LocalLoadSpec.(LocalLoadSpec.java:51) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at sun.reflect.GeneratedConstructorAccessor50.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:227) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:442) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1099) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:296) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:122) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:93) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:131) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:42) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3454) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3378) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:235) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:224) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:187) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:164) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:131) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:264) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]
… 8 more

Take a look at your overlord/coordinator logs.

Eric Graham

Solutions Engineer -** **Imply

**cell: **303-589-4581

email: eric.graham@imply.io

www.imply.io

I don’t have any error in the coordinator’s log. This is why I don’t understand…

Hi Clem, I would suggest checking your deep storage too. It is possible that somehow that segment(s) being missed from deep storage itself.

Hi, I’am using the local storage. For the moment I just want to try it in local.

Le lun. 1 juil. 2019 à 21:11, Jvalant Patel jvalant.patel@menlosecurity.com a écrit :

From the error log you shared,

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.druid.segment.loading.LocalLoadSpec] value failed: [/home/clement/Documents/apache-druid-0.15.0-incubating/var/druid/segments/tutorial/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-01T14:16:12.677Z/0/index.zip] does not exist

Does the above file exist ? Also check if your segment cache location (/var/druid/segments) has proper access permissions.

Hi Sashidhar, thanks for your help but I still get this error saying that the folder is missing.
How do I check the permission accesses ?

Just do a ls command on the folder. For more info see below

https://askubuntu.com/questions/528411/how-do-you-view-file-permissions And make sure the id thats running the druid process has access to those folders.

If you are using local storage as deep storage, I don’t think you can use a cluster setup for historicals. You need to have a common storage like S3 for Hadoop for instance.