Unexpected behavior of hyperUnique aggregator in timeseries queries

Hello Druid.io mailinglist,

I have a bit of a weird behavior of druid (version 0.9.2) where I could use some advice on.

I have a druid source which has a few dimensions and one metric, which is a longSum, called requests and have some data ingested into it. Let’s say for simplicity this is only one segment, which I will call A.

Now I extend my ingestion and write another segment with an additional second metric (metric_two) which is present in the next segment (segment B), but which is not present in segment A.

So setup is like this:

SegmentA: requests,

SegmentB: requests, metric_two

If I make this metric_two a longSum, call it requestsTest, and make a timeseries query, for segmentA requestsTest is returning 0 as expected:

Query:

{

“queryType”: “timeseries”,

“aggregations”: [

{

  "name": "requests",

  "fieldName": "requests",

  "type": "longSum"

},

{

  "type": "longSum",

  "name": "requestsTest",

  "fieldName": "requestsTest"

}

],

“intervals”: [

"2017-06-30T10:00:00.000Z/2017-06-30T18:00:00.000Z"

],

“dataSource”: “pixel_broken2-minute”,

“granularity”: “hour”

}

Result:

[ {

“timestamp” : “2017-06-30T13:00:00.000Z”,

“result” : {

"requests" : 1,

"requestsTest" : 0

}

}, {

“timestamp” : “2017-06-30T15:00:00.000Z”,

“result” : {

"requests" : 19,

"requestsTest" : 190

}

} ]

However, if I make metric_two a hyperUnique (and of course ingest it as hyperUnique as well) and make this call:

Query:

{

“queryType”: “timeseries”,

“aggregations”: [

{

  "name": "requests",

  "fieldName": "requests",

  "type": "longSum"

},

{

  "type": "hyperUnique",

  "name": "hyper",

  "fieldName": "uniqueUsers"

}

],

“intervals”: [

"2017-06-30T10:00:00.000Z/2017-06-30T18:00:00.000Z"

],

“dataSource”: “pixel_broken1-minute”,

“granularity”: “hour”

}

I get an error:

{

“error” : “Unknown exception”,

“errorMessage” : "com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START

_OBJECT\n at [Source: java.io.SequenceInputStream@37aded27; line: -1, column: 87] (through reference chai

n: java.util.ArrayList[0])",

“errorClass” : “java.lang.RuntimeException”,

“host” : null

}

and broker logs an error as well, see below.

Interestingly this only happens for timeseries queries. topN and groupBy queries work as expected and return 0 when the hyperUnique metric is missing:

Query:

{

“queryType”: “groupBy”,

“dataSource”: “pixel_broken1-minute”,

“dimensions”: [

{

  "type": "default",

  "dimension": "pixelId",

  "outputName": "pixelId"

}

],

“granularity”: “hour”,

“aggregations”: [

{

  "name": "requests",

  "fieldName": "requests",

  "type": "longSum"

},

{

  "type": "hyperUnique",

  "name": "hyper",

  "fieldName": "uniqueUsers"

}

],

“intervals”: [

"2017-06-30T10:00:00.000Z/2017-06-30T18:00:00.000Z"

],

“filter”: {

“type”: “selector”,

“dimension”: “pixelId”,

“value”: “0314fec6-52d7-4d37-b271-e0a8185708d9”

}

}

Result:

[ {

“version” : “v1”,

“timestamp” : “2017-06-30T13:00:00.000Z”,

“event” : {

"pixelId" : "0314fec6-52d7-4d37-b271-e0a8185708d9",

"requests" : 5,

"hyper" : 0.0

}

}, {

“version” : “v1”,

“timestamp” : “2017-06-30T14:00:00.000Z”,

“event” : {

"pixelId" : "0314fec6-52d7-4d37-b271-e0a8185708d9",

"requests" : 71,

"hyper" : 20.098296726168925

}

}, {

“version” : “v1”,

“timestamp” : “2017-06-30T15:00:00.000Z”,

“event” : {

"pixelId" : "0314fec6-52d7-4d37-b271-e0a8185708d9",

"requests" : 338,

"hyper" : 45.5017482108523

}

}, {

“version” : “v1”,

“timestamp” : “2017-06-30T16:00:00.000Z”,

“event” : {

"pixelId" : "0314fec6-52d7-4d37-b271-e0a8185708d9",

"requests" : 63,

"hyper" : 16.062827440181

}

} ]

Also when I restrict my hyperUnique query to the segments that were indexed with this metric, the error disappears and everything works as expected:

Query:

{

“queryType”: “timeseries”,

“aggregations”: [

{

  "name": "requests",

  "fieldName": "requests",

  "type": "longSum"

},

{

  "type": "hyperUnique",

  "name": "hyper",

  "fieldName": "uniqueUsers"

}

],

“intervals”: [

"2017-06-30T15:00:00.000Z/2017-06-30T18:00:00.000Z"

],

“dataSource”: “pixel_broken1-minute”,

“granularity”: “hour”

}

Result:

[ {

“timestamp” : “2017-06-30T15:00:00.000Z”,

“result” : {

"hyper" : 44.479537059108665,

"requests" : 338

}

}, {

“timestamp” : “2017-06-30T16:00:00.000Z”,

“result” : {

"hyper" : 10.024493827539368,

"requests" : 44

}

} ]

So I would conclude that this error has something to do with the segments (segmentA) which were not indexed with this hyperUnique metric. But I cannot figure out why this throws an error instead of returning just 0, as it does in the case of other querytypes (topN, groupBy) or other metrictypes (eg longSum).

Thank you for helping in advance!

Kind regards,

Sönke

Broker error message:

==> /var/log/druid/broker.log <==

2017-06-30 16:06:41,964 ERROR [qtp1280977839-47[timeseries_pixel_broken1-minute_808e1990-a620-4411-81d4-04eeb12f50e1]] logger.Logger (Logger.java:97) - Exception handling request: {class=io.druid.server.QueryResource, exceptionType=class java.lang.RuntimeException, exceptionMessage=com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

at [Source: java.io.SequenceInputStream@5c8a61f8; line: -1, column: 87] (through reference chain: java.util.ArrayList[0]), exception=java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

at [Source: java.io.SequenceInputStream@5c8a61f8; line: -1, column: 87] (through reference chain: java.util.ArrayList[0]), query=TimeseriesQuery{dataSource=‘pixel_broken1-minute’, querySegmentSpec=LegacySegmentSpec{intervals=[2017-06-30T10:00:00.000Z/2017-06-30T18:00:00.000Z]}, descending=false, dimFilter=pixelId = 0314fec6-52d7-4d37-b271-e0a8185708d9, granularity=‘DurationGranularity{length=3600000, origin=0}’, aggregatorSpecs=[LongSumAggregatorFactory{fieldName=‘requests’, name=‘requests’}, HyperUniquesAggregatorFactory{name=‘hyper’, fieldName=‘uniqueUsers’}], postAggregatorSpecs=, context={queryId=808e1990-a620-4411-81d4-04eeb12f50e1, timeout=300000}}, peer=78.47.68.101}

java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

at [Source: java.io.SequenceInputStream@5c8a61f8; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.client.DirectDruidClient$JsonParserIterator.next(DirectDruidClient.java:463) ~[druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.MergeSequence.toYielder(MergeSequence.java:59) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:66) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:62) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.MergeSequence.toYielder(MergeSequence.java:59) ~[java-util-0.27.10.jar:?]

at com.metamx.common.guava.LazySequence.toYielder(LazySequence.java:43) ~[java-util-0.27.10.jar:?]

at io.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:106) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-common-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.metamx.common.guava.MappedSequence.toYielder(MappedSequence.java:46) ~[java-util-0.27.10.jar:?]

at io.druid.query.CPUTimeMetricQueryRunner$1.toYielder(CPUTimeMetricQueryRunner.java:93) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.metamx.common.guava.Sequences$1.toYielder(Sequences.java:98) ~[java-util-0.27.10.jar:?]

at io.druid.server.QueryResource.doPost(QueryResource.java:223) [druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

at [Source: java.io.SequenceInputStream@5c8a61f8; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:189) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3038) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1681) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.client.DirectDruidClient$JsonParserIterator.next(DirectDruidClient.java:458) ~[druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

… 59 more

Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input in START_OBJECT

at [Source: java.io.SequenceInputStream@5c8a61f8; line: -1, column: 87]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:445) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:441) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.ParserBase.loadMoreGuaranteed(ParserBase.java:408) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.dataformat.smile.SmileParser._handleFieldName(SmileParser.java:1309) ~[jackson-dataformat-smile-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.dataformat.smile.SmileParser.nextToken(SmileParser.java:567) ~[jackson-dataformat-smile-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:120) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3038) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1681) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.client.DirectDruidClient$JsonParserIterator.next(DirectDruidClient.java:458) ~[druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

… 59 more

Hi, we are having the same issue here. The aggregator should be 0 if metric is not present on the segment, however it fails the query.

Had any resolution on this?

Thanks!

Hi Federico,

sadly the answer is no. Our current plan is to migrate to Druid 0.10 and see if the problem disappears. If not we will have to dive into Druid internals and try to figure out what’s wrong and fix it. If we make any progress in this direction I’ll post here so other users can see it.

What version of Druid are you using?

Kind regards,

Sönke

The weirdly cryptic message “com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT\n at [Source: java.io.SequenceInputStream@37aded27; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])” means that the broker got an error from a historical node that it doesn’t understand. Some of these are going to be improved in 0.10.x so the errors will look nicer. But at any rate, if you check out the historical node, you’ll find the real error and that will provide some clues.

Hi Gian,

thanks for taking a look at this. The error on historical nodes is “Null key for a Map not allowed in JSON (use a converting NullKeySerializer?).” You’ll find the complete stacktrace at the end of this message.

I was able to pinpoint the problem so far that on failing queries (ie on time-ranges including a segment without hyperUnique index) yielder.get() results in the following (I have inserted the log statement right above https://github.com/druid-io/druid/blob/49c0fe0e8bbbc685a3e8281c4b9239ddf6e7d742/server/src/main/java/io/druid/server/QueryResource.java#L249):

Result{timestamp=2017-09-01T11:00:00.000Z, value=TimeseriesResultValue{value=MetricValueExtractor{value={null=null, hyper=0, requests=5325828}}}}

whereas for segments that were indexed using hyperUnique the result is

Result{timestamp=2017-09-01T12:00:00.000Z, value=TimeseriesResultValue{value=MetricValueExtractor{value={hyper=631279.7615525584, requests=5573652}}}}

Hence, for whatever reason for non-hyperUnique segments a “null” key is added to the map inside MetricValueExtractor. I did not look substantially further, so I have no idea, why this happens yet. If you have any ideas where to start investigating, please let me know.

I used this simple query to produce the log messages above:

{

“queryType”: “timeseries”,

“aggregations”: [

{

  "name": "requests",

  "fieldName": "requests",

  "type": "longSum"

},

{

  "type": "hyperUnique",

  "name": "hyper",

  "fieldName": "uniqueUsers"

}

],

“intervals”: [

"2017-09-01T11:01:00.000Z/2017-09-01T12:58:00.000Z"

],

“dataSource”: “hyper-unique-test-minute”,

“granularity”: “hour”

}

Thanks again for helping and have a nice weekend everyone!

Sönke

Here is the full historical-node stacktrace:

2017-09-01 12:29:23,638 WARN [qtp763008902-47] servlet.ServletHandler (ServletHandler.java:628) - /druid/v2/

com.fasterxml.jackson.core.JsonGenerationException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?)

at com.fasterxml.jackson.databind.ser.impl.FailingSerializer.serialize(FailingSerializer.java:36) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:466) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:412) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:27) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.JsonValueSerializer.serialize(JsonValueSerializer.java:170) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:128) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:1902) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:280) ~[jackson-core-2.4.6.jar:2.4.6]

at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:139) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:130) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:128) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:800) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:642) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.server.QueryResource$2.write(QueryResource.java:249) ~[druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) ~[jersey-core-1.19.jar:1.19]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) ~[jersey-core-1.19.jar:1.19]

at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.jar:1.19]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

2017-09-01 12:29:23,639 WARN [qtp763008902-47] server.HttpChannel (HttpChannel.java:395) - /druid/v2/

com.fasterxml.jackson.core.JsonGenerationException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?)

at com.fasterxml.jackson.databind.ser.impl.FailingSerializer.serialize(FailingSerializer.java:36) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:466) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:412) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:27) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.JsonValueSerializer.serialize(JsonValueSerializer.java:170) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:128) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:1902) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:280) ~[jackson-core-2.4.6.jar:2.4.6]

at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:139) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:130) ~[druid-processing-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:128) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:800) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:642) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.server.QueryResource$2.write(QueryResource.java:249) ~[druid-server-0.9.2-mbr2.jar:0.9.2-mbr2]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) ~[jersey-core-1.19.jar:1.19]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) ~[jersey-core-1.19.jar:1.19]

at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.jar:1.19]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.handle(Server.java:497) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

2017-09-01 12:29:23,640 WARN [qtp763008902-47] server.HttpChannel (HttpChannel.java:481) - Could not send response error 500: com.fasterxml.jackson.core.JsonGenerationException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?)

Hi Sönke, how are you?

We are using Druid version 0.9.2.

We made some changes on the ingestion side, and it seems to be working well right now. On the ingestion side, we are using json upload. We have a metric configured on a dimension that should be on every json record. Supposedly, if the metric isn’t present on the json record, the metric should take that record as null and work without problem. However it isn’t what is really happening.

So what we added that dimension (from which the metric is being calculated) on every row that is being sent to druid. If the dimension does not exists, we push it as null value on the json. It seems to be working ok now, but can’t confirm that it is actually definitely. If this doesn’t work, we are also planning to push the dimension as an empty string. Will try to add more insights as the days go by.

Regards,

Ah thanks for posting the details. This was a bug, and is fixed in 0.10.0 by https://github.com/druid-io/druid/pull/3572.

Hello Gian, hello Federico,

@Gian Thank you very much for taking the time and your help! We’ll then migrate to 0.10 soon :slight_smile:

@Federico: Great that you found something that works for you for now and thanks also for describing your approach. I think the problem for us with your approach would be that we’d have to reingest all data, which is not easily possible in our setup.

Have a nice weekend everyone!

Sönke