Getting error in lookups for huge Number of Dictionaries

We need to create a lookup for a dimension that has around 120 thousand distinct data points. While Loading the lookups I am getting an error:

2020-01-02 06:04:03,752 DEBG ‘druid-coordinator’ stdout output:

06:04:03.751 [qtp1058994646-101] ERROR org.apache.druid.server.http.LookupCoordinatorResource - Error creating new lookups

java.lang.RuntimeException: Unknown error updating configuration

at org.apache.druid.server.http.LookupCoordinatorResource.updateAllLookups(LookupCoordinatorResource.java:170) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [druid-services-0.15.2-incubating-SNAPSHOT-selfcontained.jar:0.15.2-incubating-SNAPSHOT]

Also in a completely different scenario, we are facing performance issues while fetching data for lookup dimensions. It is taking almost a minute to fetch the results.

Hey Abhishek,

Do you see anything else suspicious in your logs? Maybe warning messages with the text “Failed to set”?