Historical Exception when running a GroupBy query (druid 13.0)

Running: druid 0.13.0-incubating

I submitted a query to the Broker for a given time window (let’s say, 5 days), and got this error:

{
“error”: “Unknown exception”,
“errorMessage”: “Can not deserialize instance of java.lang.String out of VALUE_NULL token\n at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259]”,
“errorClass”: “com.fasterxml.jackson.databind.RuntimeJsonMappingException”,
“host”: “(redacted)”
}

``

Initially I thought it might be a data corruption issue but when I reduce the span of the query, it works. The error only occurs when I increase the amount of data being sent. After digging around the logs, I found this stack trace in the Historical:

2019-05-08T21:16:35,187 ERROR [qtp269685385-140[groupBy_[(redacted)]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.fasterxml.jackson.databind.RuntimeJsonMappingException, exceptionMessage=Can not deserialize instance of java.lang.String out of VALUE_NULL token

at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], exception=com.fasterxml.jackson.databind.RuntimeJsonMappingException: Cannot deserialize instance of java.lang.String out of VALUE_NULL token at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259], query=GroupByQuery{dataSource=’(redacted)’, querySegmentSpec=MultipleSpecificSegmentSpec{descriptors=[SegmentDescriptor{interval=2019-05-01T00:00:00.000Z/2019-05-05T00:00:00.000Z, version=‘2019-05-08T12:01:15.823Z’, partitionNumber=0}]}, virtualColumns=, limitSpec=NoopLimitSpec, dimFilter=((redacted)}

com.fasterxml.jackson.databind.RuntimeJsonMappingException: Can not deserialize instance of java.lang.String out of VALUE_NULL token

at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259]

at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:194) ~[jackson-databind-2.6.7.jar:2.6.7]

at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.mergeAndGetDictionary(SpillingGrouper.java:223) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.tryMergeDictionary(ConcurrentGrouper.java:392) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.iterator(ConcurrentGrouper.java:320) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.(CloseableGrouperIterator.java:44) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:426) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.makeGrouperIterator(RowBasedGrouperHelper.java:414) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:282) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:158) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:64) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80) ~[druid-common-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.server.QueryResource.doPost(QueryResource.java:201) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.3.jar:1.19.3]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:76) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:85) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:60) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:88) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of VALUE_NULL token

at [Source: LZ4BlockInputStream(in=java.io.FileInputStream@5441412, decompressor=LZ4JNIFastDecompressor, checksum=StreamingXXHash32JNI(seed=-1756908916)); line: -1, column: 1259]

at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:857) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:62) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:277) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:192) ~[jackson-databind-2.6.7.jar:2.6.7]

… 86 more

``

I’m thinking it could be due to a memory limit in the Historical config, but unsure what specific limit it would be. Any ideas? Thanks!

Cheers,

David

This looks like it could be a bug when disk spilling is enabled in GroupBy V2.

To get the query going, you could try setting druid.query.groupBy.maxOnDiskStorage to 0 (to disable spilling) and then increase druid.query.groupBy.maxMergingDictionarySize (the max merging dictionary size on heap) until your query has enough resources.

Opened an issue at https://github.com/apache/incubator-druid/issues/7690