Broker node returns empty json

I tried to set up a cluster in VM. While I eventually will use hdfs as deep storage, currently I used the shared folder to store segments as local. Then I can query realtime node for data but broker node only returns an empty json: . Is hdfs mandatory or I did not set up the cluster right. Below is the relevant info:

  1. Realtime node + zookeeper
  2. Broker
  3. Coordinator
  4. Historical
  5. Mysql

Hi Gyrokinetic,

HDFS is not mandatory and many deep storages are supported. If you are able to query for something on the realtime node you should be able to query it on the broker as the broker is just forwarding queries and merging results. A few questions to understand more about what is happening.

  1. Are you issuing the exact same query to the realtime node and the broker?

  2. Are there any exceptions on the broker? Generally query exceptions should get returned as part of the query object, so I suspect you are not seeing any exceptions.

  3. In your broker logs, do you see messages pertaining to segments being announced? The messages state which node is serving which segment.

Here is the answer:

  1. Yes, the exact query with the exception to hit different server in the json query.
  2. No exceptions in the broker, only an empty array
  3. Got the following error when broker node log, indicating connection issues with the zookeeper (which is also in the realtime node):

2015-04-09T19:21:22,048 INFO [main-SendThread(druid-realtime:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server druid-realtime/10.9.162.10:2181. Will not attempt to authenticate using SASL (unknown error)
2015-04-09T19:21:22,049 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@65c38146].

from broker node I was able to use telnet to connect to druid-realtime 2181 so it is not a firewall issue.

Is it possible to try and simple timeBoundary query to see if you get the same error with all queries?

I changed the zookeeper hosts entry to include the port, after that the error went away but broker node still returned me the empty array: Here is the full broker log:

2015-04-09T21:17:27,022 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-04-09T21:17:27,033 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2015-04-09T21:17:28,047 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:mysql-metadata-storage], defaultVersion=‘0.7.0’, localRepository=’/home/druid/.m2/repository’, remoteRepositories=[http://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]
2015-04-09T21:17:28,274 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.cli.CliCommandCreator]
2015-04-09T21:17:31,088 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/io/druid/extensions/druid-examples/0.7.0/druid-examples-0.7.0.jar]
2015-04-09T21:17:31,088 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar]
2015-04-09T21:17:31,088 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/twitter4j/twitter4j-async/3.0.3/twitter4j-async-3.0.3.jar]
2015-04-09T21:17:31,089 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar]
2015-04-09T21:17:31,095 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/commons-validator/commons-validator/1.4.0/commons-validator-1.4.0.jar]
2015-04-09T21:17:31,095 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar]
2015-04-09T21:17:31,095 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar]
2015-04-09T21:17:31,095 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar]
2015-04-09T21:17:31,097 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.cli.CliCommandCreator]
2015-04-09T21:17:31,515 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/io/druid/extensions/druid-kafka-eight/0.7.0/druid-kafka-eight-0.7.0.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/apache/kafka/kafka-clients/0.8.2.0/kafka-clients-0.8.2.0.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/xerial/snappy/snappy-java/1.1.1.6/snappy-java-1.1.1.6.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar]
2015-04-09T21:17:31,519 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar]
2015-04-09T21:17:31,520 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar]
2015-04-09T21:17:31,520 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar]
2015-04-09T21:17:31,520 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar]
2015-04-09T21:17:31,520 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/net/sf/jopt-simple/jopt-simple/3.2/jopt-simple-3.2.jar]
2015-04-09T21:17:31,520 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar]
2015-04-09T21:17:31,530 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.cli.CliCommandCreator]
2015-04-09T21:17:32,653 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/io/druid/extensions/mysql-metadata-storage/0.7.0/mysql-metadata-storage-0.7.0.jar]
2015-04-09T21:17:32,654 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/mysql/mysql-connector-java/5.1.33/mysql-connector-java-5.1.33.jar]
2015-04-09T21:17:32,654 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/druid/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar]
2015-04-09T21:17:32,865 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:32,871 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.examples.ExamplesDruidModule] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:32,871 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:32,874 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.firehose.kafka.KafkaEightDruidModule] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:32,874 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:32,876 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class[io.druid.initialization.DruidModule]
2015-04-09T21:17:33,858 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class com.metamx.emitter.core.LoggingEmitterConfig] from props[druid.emitter.logging.] as [LoggingEmitterConfig{loggerClass=‘com.metamx.emitter.core.LoggingEmitter’, logLevel=‘info’}]
2015-04-09T21:17:33,958 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@464b8634]
2015-04-09T21:17:33,973 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=}]
2015-04-09T21:17:34,020 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName=‘broker’, host=‘druid-broker’, port=8084}]
2015-04-09T21:17:34,057 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.cache.CacheProvider] from props[druid.cache.] as [io.druid.client.cache.LocalCacheProvider@47ee14d4]
2015-04-09T21:17:34,059 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.client.cache.CacheMonitor@17e02401]
2015-04-09T21:17:34,105 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=40, maxIdleTime=PT5M}]
2015-04-09T21:17:34,114 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.QueryConfig] from props[druid.query.] as [io.druid.query.QueryConfig@84a4410]
2015-04-09T21:17:34,121 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@2734c0a4]
2015-04-09T21:17:34,128 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [io.druid.query.groupby.GroupByQueryConfig@42ec6271]
2015-04-09T21:17:34,136 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]
2015-04-09T21:17:34,138 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [1] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]
2015-04-09T21:17:34,138 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [{base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()] 2015-04-09T21:17:34,142 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [{base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]
2015-04-09T21:17:34,349 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@106d3f9d]
2015-04-09T21:17:34,367 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.broker.http.] as [io.druid.guice.http.DruidHttpClientConfig@37ad4823]
2015-04-09T21:17:34,457 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@76c1b876]
2015-04-09T21:17:34,463 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@34109a9d]
2015-04-09T21:17:34,468 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2015-04-09T21:17:34,497 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.ServerInventoryViewProvider] from props[druid.announcer.] as [io.druid.client.BatchServerInventoryViewProvider@664e7f56]
2015-04-09T21:17:34,510 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.selector.ServerSelectorStrategy] from props[druid.broker.balancer.] as [io.druid.client.selector.RandomServerSelectorStrategy@463c5a19]
2015-04-09T21:17:34,519 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.selector.TierSelectorStrategy] from props[druid.broker.select.] as [io.druid.client.selector.HighestPriorityTierSelectorStrategy@76cae8d9]
2015-04-09T21:17:34,527 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.cache.CacheConfig] from props[druid.broker.cache.] as [io.druid.client.cache.CacheConfig@1033827c]
2015-04-09T21:17:34,531 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.RetryQueryRunnerConfig] from props[druid.broker.retryPolicy.] as [io.druid.query.RetryQueryRunnerConfig@793172b2]
2015-04-09T21:17:34,535 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@5d28bb2a]
2015-04-09T21:17:34,541 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@24bbab7]
2015-04-09T21:17:34,711 INFO [main] org.eclipse.jetty.util.log - Logging initialized @8977ms
2015-04-09T21:17:34,800 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.core.LoggingEmitter.start()] on object[com.metamx.emitter.core.LoggingEmitter@4558ffd5].
2015-04-09T21:17:34,801 INFO [main] com.metamx.emitter.core.LoggingEmitter - Start: started [true]
2015-04-09T21:17:34,801 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@5e934dd].
2015-04-09T21:17:34,808 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@eaf60a7].
2015-04-09T21:17:34,812 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@15b230fc].
2015-04-09T21:17:34,812 INFO [main] io.druid.curator.CuratorModule - Starting Curator
2015-04-09T21:17:34,812 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2015-04-09T21:17:34,821 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-04-09T21:17:34,821 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid-broker
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_75
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/java/jdk1.7.0_75/jre
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=config/_common:config/broker:lib/druid-indexing-service-0.7.0.jar:lib/slf4j-api-1.6.4.jar:lib/jackson-core-2.4.4.jar:lib/commons-io-2.0.1.jar:lib/curator-client-2.7.0.jar:lib/rhino-1.7R4.jar:lib/aopalliance-1.0.jar:lib/http-client-1.0.0.jar:lib/derby-10.11.1.1.jar:lib/disruptor-3.3.0.jar:lib/jsr305-2.0.1.jar:lib/commons-cli-1.2.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/java-util-0.26.14.jar:lib/netty-3.9.5.Final.jar:lib/wagon-provider-api-2.4.jar:lib/druid-indexing-hadoop-0.7.0.jar:lib/maven-aether-provider-3.1.1.jar:lib/guice-4.0-beta.jar:lib/asm-3.1.jar:lib/extendedset-1.3.8.jar:lib/java-xmlbuilder-0.4.jar:lib/server-metrics-0.1.0.jar:lib/google-http-client-jackson2-1.15.0-rc.jar:lib/aether-spi-0.9.0.M2.jar:lib/jdbi-2.32.jar:lib/lz4-1.3.0.jar:lib/javax.el-3.0.0.jar:lib/maven-model-builder-3.1.1.jar:lib/jets3t-0.9.1.jar:lib/jackson-datatype-guava-2.4.4.jar:lib/maven-settings-builder-3.1.1.jar:lib/derbyclient-10.11.1.1.jar:lib/jersey-guice-1.17.1.jar:lib/commons-pool2-2.2.jar:lib/aws-java-sdk-core-1.8.11.jar:lib/derbynet-10.11.1.1.jar:lib/log4j-1.2.16.jar:lib/maxminddb-0.2.0.jar:lib/plexus-interpolation-1.19.jar:lib/commons-logging-1.1.1.jar:lib/commons-lang-2.6.jar:lib/zookeeper-3.4.6.jar:lib/jetty-security-9.2.5.v20141112.jar:lib/google-http-client-1.15.0-rc.jar:lib/bcprov-jdk15-1.46.jar:lib/maven-settings-3.1.1.jar:lib/jackson-core-asl-1.9.12.jar:lib/jackson-jaxrs-json-provider-2.4.4.jar:lib/druid-processing-0.7.0.jar:lib/airline-0.6.jar:lib/jetty-util-9.2.5.v20141112.jar:lib/log4j-core-2.1.jar:lib/log4j-api-2.1.jar:lib/javax.inject-1.jar:lib/curator-recipes-2.7.0.jar:lib/aether-impl-0.9.0.M2.jar:lib/jackson-datatype-joda-2.4.4.jar:lib/mail-1.4.7.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/jetty-server-9.2.5.v20141112.jar:lib/aws-java-sdk-1.8.11.jar:lib/jersey-core-1.17.1.jar:lib/guice-multibindings-4.0-beta.jar:lib/mapdb-1.0.6.jar:lib/druid-common-0.7.0.jar:lib/httpclient-4.2.jar:lib/druid-api-0.3.4.jar:lib/geoip2-0.4.0.jar:lib/icu4j-4.8.1.jar:lib/curator-framework-2.7.0.jar:lib/druid-aws-common-0.7.0.jar:lib/spymemcached-2.11.4.jar:lib/commons-codec-1.7.jar:lib/plexus-utils-3.0.15.jar:lib/hibernate-validator-5.1.3.Final.jar:lib/jline-0.9.94.jar:lib/validation-api-1.1.0.Final.jar:lib/javax.servlet-api-3.1.0.jar:lib/mx4j-3.0.2.jar:lib/jackson-databind-2.4.4.jar:lib/jackson-dataformat-smile-2.4.4.jar:lib/jetty-proxy-9.2.5.v20141112.jar:lib/druid-services-0.7.0.jar:lib/jersey-server-1.17.1.jar:lib/activation-1.1.1.jar:lib/commons-dbcp2-2.0.1.jar:lib/guice-servlet-4.0-beta.jar:lib/aether-util-0.9.0.M2.jar:lib/org.abego.treelayout.core-1.0.1.jar:lib/jboss-logging-3.1.3.GA.jar:lib/guava-16.0.1.jar:lib/classmate-1.0.0.jar:lib/config-magic-0.9.jar:lib/xpp3-1.1.4c.jar:lib/irc-api-1.0-0011.jar:lib/jetty-http-9.2.5.v20141112.jar:lib/protobuf-java-2.5.0.jar:lib/jetty-client-9.2.5.v20141112.jar:lib/bytebuffer-collections-0.1.5.jar:lib/okhttp-1.0.2.jar:lib/jackson-jaxrs-smile-provider-2.4.4.jar:lib/javax.el-api-3.0.0.jar:lib/jetty-io-9.2.5.v20141112.jar:lib/jersey-servlet-1.17.1.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/jetty-servlets-9.2.5.v20141112.jar:lib/emitter-0.3.0.jar:lib/joda-time-2.6.jar:lib/httpcore-4.2.jar:lib/maven-repository-metadata-3.1.1.jar:lib/jackson-jaxrs-base-2.4.4.jar:lib/commons-pool-1.6.jar:lib/tesla-aether-0.0.5.jar:lib/RoaringBitmap-0.4.5.jar:lib/antlr4-runtime-4.0.jar:lib/compress-lzf-1.0.3.jar:lib/jetty-continuation-9.2.5.v20141112.jar:lib/druid-server-0.7.0.jar:lib/maven-model-3.1.1.jar:lib/log4j-slf4j-impl-2.1.jar:lib/opencsv-2.3.jar:lib/jetty-servlet-9.2.5.v20141112.jar:lib/aether-api-0.9.0.M2.jar:lib/curator-x-discovery-2.7.0.jar:lib/jackson-module-jaxb-annotations-2.4.4.jar:lib/jackson-annotations-2.4.4.jar
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.32-504.8.1.el6.x86_64
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/druid
2015-04-09T21:17:34,822 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/druid/poc/druid-0.7.0
2015-04-09T21:17:34,823 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=druid-realtime:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@4e1740fa
2015-04-09T21:17:34,846 INFO [main-SendThread(druid-realtime:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server druid-realtime/10.9.162.10:2181. Will not attempt to authenticate using SASL (unknown error)
2015-04-09T21:17:34,847 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@65c38146].
2015-04-09T21:17:34,862 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.coordination.broker.DruidBroker.start()] on object[io.druid.server.coordination.broker.DruidBroker@133cc81d].
2015-04-09T21:17:34,864 INFO [main-SendThread(druid-realtime:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to druid-realtime/10.9.162.10:2181, initiating session
2015-04-09T21:17:34,869 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112
2015-04-09T21:17:34,878 INFO [main-SendThread(druid-realtime:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server druid-realtime/10.9.162.10:2181, sessionid = 0x14c9567dbde0067, negotiated timeout = 30000
2015-04-09T21:17:34,884 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering io.druid.server.StatusResource as a root resource class
Apr 09, 2015 9:17:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version ‘Jersey: 1.17.1 02/28/2013 12:47 PM’
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope “Singleton”
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.http.BrokerResource to GuiceInstantiatedComponentProvider
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.ClientInfoResource to GuiceInstantiatedComponentProvider
Apr 09, 2015 9:17:35 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope “Undefined”
2015-04-09T21:17:35,780 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7c46148{/,null,AVAILABLE}
2015-04-09T21:17:35,787 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@16786ec6{HTTP/1.1}{0.0.0.0:8084}
2015-04-09T21:17:35,787 INFO [main] org.eclipse.jetty.server.Server - Started @10056ms
2015-04-09T21:17:35,929 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Inventory Initialized
2015-04-09T21:17:35,939 INFO [ServerInventoryView-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘broker’, host=‘druid-broker’, port=8084}]

2015-04-09T21:17:41,328 INFO [qtp488334200-23] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-04-09T21:17:41.309Z”,“service”:“broker”,“host”:“druid-broker:8084”,“metric”:“query/time”,“value”:13,“user2”:“druidtest_160”,“user4”:“timeseries”,“user5”:[“2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z”],“user6”:“false”,“user7”:“4 aggs”,“user8”:“d93931e9-cba8-4948-9bad-9525017fdedd”,“user9”:“PT5258880M”}]
2015-04-09T21:17:41,329 INFO [qtp488334200-23] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-04-09T21:17:41.328Z”,“service”:“broker”,“host”:“druid-broker:8084”,“metric”:“query/wait”,“value”:8,“user2”:“druidtest_160”,“user4”:“timeseries”,“user5”:[“2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z”],“user6”:“false”,“user7”:“4 aggs”,“user8”:“d93931e9-cba8-4948-9bad-9525017fdedd”,“user9”:“PT5258880M”}]
2015-04-09T21:17:41,337 INFO [qtp488334200-23] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-04-09T21:17:41.337Z”,“service”:“broker”,“host”:“druid-broker:8084”,“metric”:“request/time”,“value”:152,“user2”:“druidtest_160”,“user3”:"{“queryId”:“d93931e9-cba8-4948-9bad-9525017fdedd”,“timeout”:300000}",“user4”:“timeseries”,“user5”:[“2010-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z”],“user6”:“false”,“user7”:“10.9.162.10”,“user8”:“d93931e9-cba8-4948-9bad-9525017fdedd”,“user9”:“PT5258880M”}]

Hi, are you including the same common configuration file on the classpath of both the realtime and broker node?

The two common config files are almost identical, here is the diff. In the realtime node, I did not put the port 2181 and its has a specific monitor setting:

[druid@druid-realtime druid-0.7.0]$ diff config/_common/common.runtime.properties ~/.
22c22
< druid.zk.service.host=druid-realtime

The timeBoundary query in realtime node returns:

[ {
“timestamp” : “2015-04-07T22:09:56.000Z”,
“result” : {
“minTime” : “2015-04-07T22:09:56.000Z”,
“maxTime” : “2015-04-09T12:57:07.000Z”
}
} ]

In broker node returns:

Looking at your logs, it seems the broker does not see the realtime node, which leads me to suspect it is a zookeeper issue.

Can you use the same common configuration across both nodes? Since the realtime node seems to work, you might want to use the common configuration there (I suspect the config druid.zk.service.host=druid-realtime may be problematic). If you want to override configs in the common configuration for a particular node, you can do so in the runtime.properties associated with that node.

Let me know if that helps.

– FJ

I found the cause of the issues – the host name in config file is localhost. It should be the real hostname since broker node is in a separate host.

I take it back, ran the wrong query and it hit the realtime node.
BTW, I used the same common config across nodes but the issue persists.

Can you send me the full logs of realtime and broker? We’re not quite sure what is going on. We suspect there’s some sort of ZK connection problem, but having the full logs would help here.

FJ,

Thank you for agreeing to take time to help me look at the issue. The two files are large so I attached them here instead. For both services set druid to emit logging.

Please let me know if you need further info.

Gyrokinetic

broker.log (28.4 KB)

realtime.log (92.5 KB)

Most of this was discussed over IRC, but the realtime that is being run is the example realtime that doesn’t announce itself in ZK. We should fix the examples to use the actual nodes and remove example realtime.

Through IRC, FJ helped me to identify the issue, when starting realtime node, we should use server realtime, instead I used the recycled example realtime. Thus the realtime node was not announced to the cluster and broker was not aware of it.