Failure getting results from datasource.. Exception trace in post

I have two identical datasources (same ingestion spec; only difference is that one of the datasources contain only 1k rows while the other contains 200M rows)

Can someone tell me what this exception actually means and how I can go about correcting it?

{

“error”: “Unknown exception”,

“errorMessage”: "Failure getting results for query[c8ed0ff6-bb69-40ba-91b7-f04c130acc8e] url[http://<IP_RETRACTED>:8083/druid/v2/] because of [Unexpected end-of-input: was expecting closing quote for a string value\n at [Source: {“missingSegments”:[{“itvl”:“2002-02-18T00:00:00.000Z/2002-02-25T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:6},{“itvl”:“2003-01-20T00:00:00.000Z/2003-01-27T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:7},{“itvl”:“2003-01-20T00:00:00.000Z/2003-01-27T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:8},{“itvl”:“2003-11-10T00:00:00.000Z/2003-11-17T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:13},{“itvl”:“2003-11-10T00:00:00.000Z/2003-11-17T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:12},{“itvl”:“2003-11-24T00:00:00.000Z/2003-12-01T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:9},{“itvl”:“2004-01-12T00:00:00.000Z/2004-01-19T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:10},{“itvl”:“2004-01-12T00:00:00.000Z/2004-01-19T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:9},{“itvl”:“2004-01-05T00:00:00.000Z/2004-01-12T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:16},{“itvl”:“2004-04-05T00:00:00.000Z/2004-04-12T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:16},{“itvl”:“2004-05-24T00:00:00.000Z/2004-05-31T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:11},{“itvl”:“2004-05-24T00:00:00.000Z/2004-05-31T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:10},{“itvl”:“2004-06-07T00:00:00.000Z/2004-06-14T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:13},{“itvl”:“2004-05-24T00:00:00.000Z/2004-05-31T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:13},{“itvl”:“2004-04-05T00:00:00.000Z/2004-04-12T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:15},{“itvl”:“2004-12-20T00:00:00.000Z/2004-12-27T00:00:00.000Z”,“ver”:“2019-07-10T08:35:49.874Z”,“part”:14},{“itvl”:“2005-01-17T00:00:00.000Z/2005-01-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:24},{“itvl”:“2005-01-17T00:00:00.000Z/2005-01-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:23},{“itvl”:“2005-01-17T00:00:00.000Z/2005-01-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:25},{“itvl”:“2005-01-17T00:00:00.000Z/2005-01-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:26},{“itvl”:“2003-12-15T00:00:00.000Z/2003-12-22T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:14},{“itvl”:“2003-12-15T00:00:00.000Z/2003-12-22T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:13},{“itvl”:“2003-11-24T00:00:00.000Z/2003-12-01T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:10},{“itvl”:“2005-03-14T00:00:00.000Z/2005-03-21T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:22},{“itvl”:“2005-05-23T00:00:00.000Z/2005-05-30T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:22},{“itvl”:“2005-05-23T00:00:00.000Z/2005-05-30T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:24},{“itvl”:“2005-05-23T00:00:00.000Z/2005-05-30T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:21},{“itvl”:“2005-05-30T00:00:00.000Z/2005-06-06T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:23},{“itvl”:“2005-07-25T00:00:00.000Z/2005-08-01T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:18},{“itvl”:“2005-11-21T00:00:00.000Z/2005-11-28T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:15},{“itvl”:“2006-02-20T00:00:00.000Z/2006-02-27T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:13},{“itvl”:“2006-01-30T00:00:00.000Z/2006-02-06T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:18},{“itvl”:“2006-01-30T00:00:00.000Z/2006-02-06T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:17},{“itvl”:“2006-06-26T00:00:00.000Z/2006-07-03T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:16},{“itvl”:“2006-05-08T00:00:00.000Z/2006-05-15T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:24},{“itvl”:“2006-05-08T00:00:00.000Z/2006-05-15T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:23},{“itvl”:“2006-12-04T00:00:00.000Z/2006-12-11T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:27},{“itvl”:“2007-01-08T00:00:00.000Z/2007-01-15T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:21},{“itvl”:“2007-04-09T00:00:00.000Z/2007-04-16T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:26},{“itvl”:“2007-05-28T00:00:00.000Z/2007-06-04T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:26},{“itvl”:“2007-06-04T00:00:00.000Z/2007-06-11T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:29},{“itvl”:“2007-07-09T00:00:00.000Z/2007-07-16T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:27},{“itvl”:“2007-09-03T00:00:00.000Z/2007-09-10T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:16},{“itvl”:“2007-11-19T00:00:00.000Z/2007-11-26T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:31},{“itvl”:“2007-11-26T00:00:00.000Z/2007-12-03T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:38},{“itvl”:“2007-12-24T00:00:00.000Z/2007-12-31T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:24},{“itvl”:“2008-01-07T00:00:00.000Z/2008-01-14T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:26},{“itvl”:“2008-03-31T00:00:00.000Z/2008-04-07T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:29},{“itvl”:“2008-04-14T00:00:00.000Z/2008-04-21T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:23},{“itvl”:“2008-04-14T00:00:00.000Z/2008-04-21T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:31},{“itvl”:“2008-04-21T00:00:00.000Z/2008-04-28T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:22},{“itvl”:“2008-04-21T00:00:00.000Z/2008-04-28T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:23},{“itvl”:“2008-06-30T00:00:00.000Z/2008-07-07T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:28},{“itvl”:“2008-08-11T00:00:00.000Z/2008-08-18T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:22},{“itvl”:“2008-09-22T00:00:00.000Z/2008-09-29T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:38},{“itvl”:“2007-08-27T00:00:00.000Z/2007-09-03T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:21},{“itvl”:“2008-11-17T00:00:00.000Z/2008-11-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:36},{“itvl”:“2008-12-29T00:00:00.000Z/2009-01-05T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:31},{“itvl”:“2009-01-05T00:00:00.000Z/2009-01-12T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:29},{“itvl”:“2009-01-26T00:00:00.000Z/2009-02-02T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:27},{“itvl”:“2009-02-02T00:00:00.000Z/2009-02-09T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:33},{“itvl”:“2009-02-02T00:00:00.000Z/2009-02-09T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:42},{“itvl”:“2009-02-02T00:00:00.000Z/2009-02-09T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:41},{“itvl”:“2009-05-04T00:00:00.000Z/2009-05-11T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:33},{“itvl”:“2009-06-01T00:00:00.000Z/2009-06-08T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:26},{“itvl”:“2009-06-01T00:00:00.000Z/2009-06-08T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:27},{“itvl”:“2009-08-17T00:00:00.000Z/2009-08-24T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:45},{“itvl”:“2009-09-07T00:00:00.000Z/2009-09-14T00:00:00.000Z”,“ver”:“2019-07-10T08:30:48.190Z”,“part”:44},{“itvl”:“2009-10-26T00:00:00.000Z/2009-11-02T00:00:00.000Z”,“ver”:“2019-07-10T08:; line: 1, column: 14337]]”,

“errorClass”: “org.apache.druid.java.util.common.RE”,

“host”: null

}

``

The query is a simple “select count(*) from datasource”

I just checked my coordinator console and one of my historicals is not showing up (I have two historicals)… But on the one which doesn’t show up on console. I have the data for my datasource and I just restarted it for good measure and I do see a lot of segment loads

2019-07-11T10:28:23,213 INFO [Segment-Load-Startup-2] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[2198/32458][abc_2019-02-18T00:00:00.000Z_2019-02-25T00:00:00.000Z_2019-07-10T08:35:49.874Z_115]

``

I just restarted the historical which is showing up on the console… This has a lot of errors in the logs -

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/loadQueue/:8083/abc_2017-09-11T00:00:00.000Z_2017-09-18T00:00:00.000Z_2019-07-10T08:35:49.874Z_129

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at org.apache.druid.server.coordination.ZkCoordinator$1$1.execute(ZkCoordinator.java:142) ~[druid-server-0.14.2-incubating.jar:0.14.2-incubating]

at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:331) ~[druid-server-0.14.2-incubating.jar:0.14.2-incubating]

at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) ~[druid-server-0.14.2-incubating.jar:0.14.2-incubating]

at org.apache.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.14.2-incubating.jar:0.14.2-incubating]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.1.0.jar:4.1.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.1.0.jar:4.1.0]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) [curator-client-4.1.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.1.0.jar:4.1.0]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.1.0.jar:4.1.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.1.0.jar:4.1.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/loadQueue/2:8083/abc_2017-09-11T00:00:00.000Z_2017-09-18T00:00:00.000Z_2019-07-10T08:35:49.874Z_129

at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:876) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274) ~[curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268) ~[curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[curator-client-4.1.0.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[curator-client-4.1.0.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265) ~[curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:249) ~[curator-framework-4.1.0.jar:4.1.0]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:34) ~[curator-framework-4.1.0.jar:4.1.0]

at org.apache.druid.server.coordination.ZkCoordinator$1$1.execute(ZkCoordinator.java:129) ~[druid-server-0.14.2-incubating.jar:0.14.2-incubating]

… 18 more

``

I have no idea whats going on… Any help will be greatly appreciated.

Hi Karthik:

Those ‘NoNode’ error are pointing to znodes not found on zookeepers. Can you verify if the zookeepers are healthy, and both historical nodes have finished loading segments? Is the datasource fully ready from your coordinator console?

Thanks

Yeah. The data source is ready on the coordinator console. The deep storage (S3) has all the segments. The historical nodes have the segments in their local storage as well.

One thing I forgot to mention is that I have a cluster of three zk nodes. I have given the comma separated list of all zk nodes in my common.runtime.properties on all my cluster nodes. I will double check on that.

It’s that one historical didn’t have all three zookeepers in common properties. Fixed that and restarted historicals and it’s all fixed now.