issues with ingesting csv data via kafka-0.8

A simple kafka producer generated the following log like data as shown in the debug window:

topic=druidtest msg=2015-03-27 14:41:24,col_1:186,col_2:31,472.0,108.0,161.0

topic=druidtest msg=2015-03-27 14:41:24,col_1:43,col_2:98,111.0,343.0,952.0

topic=druidtest msg=2015-03-27 14:41:24,col_1:99,col_2:65,158.0,195.0,140.0

topic=druidtest msg=2015-03-27 14:41:24,col_1:183,col_2:77,870.0,799.0,409.0

topic=druidtest msg=2015-03-27 14:41:24,col_1:56,col_2:249,541.0,95.0,841.0

The kafka consumer can get data without issues:

2015-03-27 14:41:24,col_1:186,col_2:31,472.0,108.0,161.0

2015-03-27 14:41:24,col_1:43,col_2:98,111.0,343.0,952.0

2015-03-27 14:41:24,col_1:99,col_2:65,158.0,195.0,140.0

2015-03-27 14:41:24,col_1:183,col_2:77,870.0,799.0,409.0

2015-03-27 14:41:24,col_1:56,col_2:249,541.0,95.0,841.0

but not via druid. The spec file is listed below. Did I put something wrong? Thanks.

[{

“dataSchema”: {

“dataSource”: “druidtest”,

“parser”: {

“type”:“string”,

“parseSpec”: {

“type”: “csv”,

“columns”: [“timestamp”, “col_1”, “col_2”, “col_3”, “col_4”, “col_5” ],

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

},

“dimensionsSpec”: {

“dimensions”: [ “col_1”, “col_2” ],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec”: [ { “type”: “count”, “name”: “count” }, { “type”: “doubleSum”, “name”: “sum_3”, “fieldName”: “col_3” }, { “type”: “doubleSum”, “name”: “sum_4”, “fieldName”: “col_4” }, { “type”: “doubleSum”, “name”: “sum_5”, “fieldName”: “col_5” } ],

“granularitySpec”: { “type”: “uniform”, “segmentGranularity”: “HOUR”, “queryGranularity”: “NONE” }

},

“ioConfig”: {

“type”:“realtime”,

“firehose”:{

“type”:“kafka-0.8”,

“consumerProps” : { “zookeeper.connect” : “localhost:2181”,

“zookeeper.connection.timeout.ms” : “15000”,

“zookeeper.session.timeout.ms” : “15000”,

“zookeeper.sync.time.ms” : “5000”,

“group.id” : “druidtest_group”,

“fetch.message.max.bytes” : “1048586”,

“auto.offset.reset” : “largest”,

“auto.commit.enable” : “false” },

“feed”:“druidtest”

},

“plumber”: {

“type”:“realtime”

}

},

“tuningConfig”: {

“type”:“realtime”,

“maxRowsInMemory”:500000,

“intermediatePersistPeriod”:“PT10m”,

“windowPeriod”:“PT10m”,

“basePersistDirectory”:"/tmp/realtime/basePersist",

“rejectionPolicy”:{“type”:“serverTime”}

}

}]

Hi Xiaoming,

Try changing “type”:“csv” to “format”:“csv”.

Fangjin,

Thanks for the quick reply. I made the change but still have the same issue.

Xiaoming

Let me document what I used in the study:

  1. druid-0.7.0

  2. kafka_2.11-0.8.2.1

  3. zookeeper-3.4.6

The traffic seemed not to be routed to Druid, currently set up as a single node application. The Druid server reported the following unknown error, but sub sequent log seems to indicate the connection was made successfully:

2015-03-28T01:32:43,242 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-03-28T01:32:43,242 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session

The study is run in Centos 6.4 in a Virtualbox.

Hi Xiaomi,
Can you post more about the log from the realtime node ?
also you runtime properties file.

The realtime.properties file should be simple (the default one comments out the host and port and I tried both without luck)

druid.host=localhost

druid.port=8084

druid.service=realtime

druid.processing.buffer.sizeBytes=100000000

druid.processing.numThreads=2

Here is the logs

This will run a stand-alone version of Druid

Running command:

  • java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/home/dev/src/druid-0.7.0/test/test.realtime.spec -classpath ‘/home/dev/src/druid-0.7.0/…/config/realtime:/home/dev/src/druid-0.7.0/test:/home/dev/src/druid-0.7.0/config/_common:/home/dev/src/druid-0.7.0/config/realtime:/home/dev/src/druid-0.7.0/lib/*’ io.druid.cli.Main example realtime

2015-03-28T17:05:24,426 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2015-03-28T17:05:24,429 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

2015-03-28T17:05:25,417 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:mysql-metadata-storage], defaultVersion=‘0.7.0’, localRepository=’/home/dev/.m2/repository’, remoteRepositories=[http://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]

2015-03-28T17:05:25,643 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/druid-examples/0.7.0/druid-examples-0.7.0.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-async/3.0.3/twitter4j-async-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-validator/commons-validator/1.4.0/commons-validator-1.4.0.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar]

2015-03-28T17:05:27,804 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/druid-kafka-eight/0.7.0/druid-kafka-eight-0.7.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/kafka/kafka-clients/0.8.2.0/kafka-clients-0.8.2.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/xerial/snappy/snappy-java/1.1.1.6/snappy-java-1.1.1.6.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/net/sf/jopt-simple/jopt-simple/3.2/jopt-simple-3.2.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar]

2015-03-28T17:05:28,180 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:29,206 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/mysql-metadata-storage/0.7.0/mysql-metadata-storage-0.7.0.jar]

2015-03-28T17:05:29,207 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/mysql/mysql-connector-java/5.1.33/mysql-connector-java-5.1.33.jar]

2015-03-28T17:05:29,207 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar]

2015-03-28T17:05:29,410 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,411 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.examples.ExamplesDruidModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,411 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,412 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.firehose.kafka.KafkaEightDruidModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,412 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,413 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:30,235 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@33cdefce]

2015-03-28T17:05:30,252 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=}]

2015-03-28T17:05:30,276 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName=‘realtime’, host=‘localhost’, port=8084}]

2015-03-28T17:05:30,329 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=40, maxIdleTime=PT5M}]

2015-03-28T17:05:30,334 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.RealtimeManagerConfig] from props[druid.realtime.] as [io.druid.guice.RealtimeManagerConfig@7cc8ff30]

2015-03-28T17:05:30,556 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.QueryConfig] from props[druid.query.] as [io.druid.query.QueryConfig@3399ceef]

2015-03-28T17:05:30,575 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@7ff2a3d6]

2015-03-28T17:05:30,581 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [io.druid.query.groupby.GroupByQueryConfig@3171c7df]

2015-03-28T17:05:30,587 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]

2015-03-28T17:05:30,589 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]

2015-03-28T17:05:30,589 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]

2015-03-28T17:05:30,590 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]

2015-03-28T17:05:30,727 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@9f1e040]

2015-03-28T17:05:30,742 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageTablesConfig] from props[druid.metadata.storage.tables.] as [io.druid.metadata.MetadataStorageTablesConfig@609a34a0]

2015-03-28T17:05:30,749 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageConnectorConfig] from props[druid.metadata.storage.connector.] as [DbConnectorConfig{createTables=true, connectURI=‘jdbc:mysql://localhost:3306/druid’, user=‘druid’, password=****}]

2015-03-28T17:05:30,893 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@58d3f4be]

2015-03-28T17:05:30,897 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@6015465]

2015-03-28T17:05:30,901 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29

2015-03-28T17:05:30,957 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.FilteredServerViewProvider] from props[druid.announcer.] as [io.druid.client.FilteredBatchServerViewProvider@10b7415b]

2015-03-28T17:05:30,987 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@3055764c]

2015-03-28T17:05:30,993 INFO [main] org.eclipse.jetty.util.log - Logging initialized @7558ms

2015-03-28T17:05:31,070 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@32d91d5].

2015-03-28T17:05:31,070 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@24c13894].

2015-03-28T17:05:31,070 INFO [main] io.druid.curator.CuratorModule - Starting Curator

2015-03-28T17:05:31,070 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid.icsv.dev

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_75

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/java/jdk1.7.0_75/jre

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/dev/src/druid-0.7.0/…/config/realtime:/home/dev/src/druid-0.7.0/test:/home/dev/src/druid-0.7.0/config/_common:/home/dev/src/druid-0.7.0/config/realtime:/home/dev/src/druid-0.7.0/lib/aws-java-sdk-1.8.11.jar:/home/dev/src/druid-0.7.0/lib/jboss-logging-3.1.3.GA.jar:/home/dev/src/druid-0.7.0/lib/activation-1.1.1.jar:/home/dev/src/druid-0.7.0/lib/curator-framework-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/server-metrics-0.1.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-io-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/spymemcached-2.11.4.jar:/home/dev/src/druid-0.7.0/lib/jetty-security-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jetty-proxy-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/druid-processing-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/guice-multibindings-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/javax.inject-1.jar:/home/dev/src/druid-0.7.0/lib/jackson-jaxrs-smile-provider-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-core-asl-1.9.12.jar:/home/dev/src/druid-0.7.0/lib/commons-codec-1.7.jar:/home/dev/src/druid-0.7.0/lib/druid-api-0.3.4.jar:/home/dev/src/druid-0.7.0/lib/jersey-core-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/maven-settings-builder-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jersey-servlet-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/hibernate-validator-5.1.3.Final.jar:/home/dev/src/druid-0.7.0/lib/jline-0.9.94.jar:/home/dev/src/druid-0.7.0/lib/commons-pool-1.6.jar:/home/dev/src/druid-0.7.0/lib/aether-connector-file-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/jetty-server-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/curator-x-discovery-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/commons-logging-1.1.1.jar:/home/dev/src/druid-0.7.0/lib/aether-impl-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/javax.el-api-3.0.0.jar:/home/dev/src/druid-0.7.0/lib/okhttp-1.0.2.jar:/home/dev/src/druid-0.7.0/lib/RoaringBitmap-0.4.5.jar:/home/dev/src/druid-0.7.0/lib/extendedset-1.3.8.jar:/home/dev/src/druid-0.7.0/lib/log4j-1.2.16.jar:/home/dev/src/druid-0.7.0/lib/maxminddb-0.2.0.jar:/home/dev/src/druid-0.7.0/lib/mail-1.4.7.jar:/home/dev/src/druid-0.7.0/lib/druid-common-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/asm-3.1.jar:/home/dev/src/druid-0.7.0/lib/commons-cli-1.2.jar:/home/dev/src/druid-0.7.0/lib/validation-api-1.1.0.Final.jar:/home/dev/src/druid-0.7.0/lib/aether-connector-okhttp-0.0.9.jar:/home/dev/src/druid-0.7.0/lib/httpcore-4.2.jar:/home/dev/src/druid-0.7.0/lib/classmate-1.0.0.jar:/home/dev/src/druid-0.7.0/lib/aopalliance-1.0.jar:/home/dev/src/druid-0.7.0/lib/guava-16.0.1.jar:/home/dev/src/druid-0.7.0/lib/commons-pool2-2.2.jar:/home/dev/src/druid-0.7.0/lib/opencsv-2.3.jar:/home/dev/src/druid-0.7.0/lib/mapdb-1.0.6.jar:/home/dev/src/druid-0.7.0/lib/java-xmlbuilder-0.4.jar:/home/dev/src/druid-0.7.0/lib/joda-time-2.6.jar:/home/dev/src/druid-0.7.0/lib/jackson-core-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-datatype-joda-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/maven-model-builder-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/plexus-interpolation-1.19.jar:/home/dev/src/druid-0.7.0/lib/curator-client-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-servlet-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/plexus-utils-3.0.15.jar:/home/dev/src/druid-0.7.0/lib/druid-aws-common-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/jdbi-2.32.jar:/home/dev/src/druid-0.7.0/lib/derbynet-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/derbyclient-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/bcprov-jdk15-1.46.jar:/home/dev/src/druid-0.7.0/lib/java-util-0.26.14.jar:/home/dev/src/druid-0.7.0/lib/bytebuffer-collections-0.1.5.jar:/home/dev/src/druid-0.7.0/lib/org.abego.treelayout.core-1.0.1.jar:/home/dev/src/druid-0.7.0/lib/http-client-1.0.0.jar:/home/dev/src/druid-0.7.0/lib/disruptor-3.3.0.jar:/home/dev/src/druid-0.7.0/lib/mx4j-3.0.2.jar:/home/dev/src/druid-0.7.0/lib/druid-services-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/antlr4-runtime-4.0.jar:/home/dev/src/druid-0.7.0/lib/maven-aether-provider-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jackson-dataformat-smile-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/javax.el-3.0.0.jar:/home/dev/src/druid-0.7.0/lib/derby-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/curator-recipes-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/irc-api-1.0-0011.jar:/home/dev/src/druid-0.7.0/lib/zookeeper-3.4.6.jar:/home/dev/src/druid-0.7.0/lib/guice-servlet-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/emitter-0.3.0.jar:/home/dev/src/druid-0.7.0/lib/icu4j-4.8.1.jar:/home/dev/src/druid-0.7.0/lib/netty-3.9.5.Final.jar:/home/dev/src/druid-0.7.0/lib/druid-indexing-service-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/google-http-client-jackson2-1.15.0-rc.jar:/home/dev/src/druid-0.7.0/lib/aether-spi-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/druid-indexing-hadoop-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/commons-io-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/slf4j-api-1.6.4.jar:/home/dev/src/druid-0.7.0/lib/commons-dbcp2-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/jetty-continuation-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jersey-server-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/maven-settings-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/druid-server-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/aether-api-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/geoip2-0.4.0.jar:/home/dev/src/druid-0.7.0/lib/guice-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/protobuf-java-2.5.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-util-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/javax.servlet-api-3.1.0.jar:/home/dev/src/druid-0.7.0/lib/wagon-provider-api-2.4.jar:/home/dev/src/druid-0.7.0/lib/jetty-client-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jetty-http-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/maven-model-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jersey-guice-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/commons-lang-2.6.jar:/home/dev/src/druid-0.7.0/lib/maven-repository-metadata-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jackson-datatype-guava-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/log4j-slf4j-impl-2.1.jar:/home/dev/src/druid-0.7.0/lib/config-magic-0.9.jar:/home/dev/src/druid-0.7.0/lib/jetty-servlets-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/xpp3-1.1.4c.jar:/home/dev/src/druid-0.7.0/lib/rhino-1.7R4.jar:/home/dev/src/druid-0.7.0/lib/jets3t-0.9.1.jar:/home/dev/src/druid-0.7.0/lib/httpclient-4.2.jar:/home/dev/src/druid-0.7.0/lib/jsr305-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/airline-0.6.jar:/home/dev/src/druid-0.7.0/lib/jackson-annotations-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/aether-util-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/log4j-core-2.1.jar:/home/dev/src/druid-0.7.0/lib/aws-java-sdk-core-1.8.11.jar:/home/dev/src/druid-0.7.0/lib/jackson-mapper-asl-1.9.13.jar:/home/dev/src/druid-0.7.0/lib/jackson-databind-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-module-jaxb-annotations-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-jaxrs-base-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/lz4-1.3.0.jar:/home/dev/src/druid-0.7.0/lib/google-http-client-1.15.0-rc.jar:/home/dev/src/druid-0.7.0/lib/jackson-jaxrs-json-provider-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/compress-lzf-1.0.3.jar:/home/dev/src/druid-0.7.0/lib/tesla-aether-0.0.5.jar:/home/dev/src/druid-0.7.0/lib/log4j-api-2.1.jar

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.32-504.el6.x86_64

2015-03-28T17:05:31,080 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=dev

2015-03-28T17:05:31,080 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/dev

2015-03-28T17:05:31,080 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/dev/src/druid-0.7.0

2015-03-28T17:05:31,080 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@783308c4

2015-03-28T17:05:31,097 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@610e2b80].

2015-03-28T17:05:31,112 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-03-28T17:05:31,120 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session

2015-03-28T17:05:31,124 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.segment.realtime.RealtimeManager.start() throws java.io.IOException] on object[io.druid.segment.realtime.RealtimeManager@3cd07ac].

2015-03-28T17:05:31,125 INFO [main] io.druid.segment.realtime.RealtimeManager - Calling the FireDepartment and getting a Firehose.

2015-03-28T17:05:31,146 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14c5731493a00a4, negotiated timeout = 30000

2015-03-28T17:05:31,148 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-03-28T17:05:31,522 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=15000 watcher=org.I0Itec.zkclient.ZkClient@78e0e606

2015-03-28T17:05:31,524 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-03-28T17:05:31,525 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

2015-03-28T17:05:31,526 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14c5731493a00a5, negotiated timeout = 15000

2015-03-28T17:05:32,014 INFO [main] io.druid.segment.realtime.RealtimeManager - Firehose acquired!

2015-03-28T17:05:32,014 INFO [main] io.druid.segment.realtime.RealtimeManager - Someone get us a plumber!

2015-03-28T17:05:32,042 INFO [main] io.druid.segment.realtime.plumber.RealtimePlumber - Creating plumber using rejectionPolicy[serverTime-PT10M]

2015-03-28T17:05:32,042 INFO [main] io.druid.segment.realtime.RealtimeManager - We have our plumber!

2015-03-28T17:05:32,045 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-03-28T17:05:32,058 INFO [chief-druidtest] io.druid.segment.realtime.RealtimeManager - FireChief[druidtest] state ok.

2015-03-28T17:05:32,088 INFO [chief-druidtest] io.druid.segment.realtime.plumber.RealtimePlumber - Expect to run at [2015-03-28T18:10:00.000Z]

2015-03-28T17:05:32,121 INFO [chief-druidtest] io.druid.segment.realtime.plumber.RealtimePlumber - Starting merge and push.

2015-03-28T17:05:32,122 INFO [chief-druidtest] io.druid.segment.realtime.plumber.RealtimePlumber - Found [0] sinks. minTimestamp [2015-03-28T16:00:00.000Z]

2015-03-28T17:05:32,122 INFO [chief-druidtest] io.druid.segment.realtime.plumber.RealtimePlumber - Found [0] sinks to persist and merge

2015-03-28T17:05:32,208 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Inventory Initialized

Mar 28, 2015 5:05:32 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class

Mar 28, 2015 5:05:32 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering io.druid.server.StatusResource as a root resource class

Mar 28, 2015 5:05:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate

INFO: Initiating Jersey application, version ‘Jersey: 1.17.1 02/28/2013 12:47 PM’

Mar 28, 2015 5:05:32 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope “Singleton”

Mar 28, 2015 5:05:33 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider

Mar 28, 2015 5:05:33 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider

Mar 28, 2015 5:05:33 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope “Undefined”

2015-03-28T17:05:33,058 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@4006c95c{/,null,AVAILABLE}

2015-03-28T17:05:33,065 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@22c3b003{HTTP/1.1}{0.0.0.0:8084}

2015-03-28T17:05:33,065 INFO [main] org.eclipse.jetty.server.Server - Started @9632ms

do you have this in your runtime properties file

druid.publish.type
Where to publish segments. Choices are “noop” or “metadata”.
metadata

This is not required and it defaults to metadata right? But I put this in anyway but its presence did not make any difference (tried both noop and metadata)

Hi Xiaoming,

If you enable the RealtimeMetricsMonitor on your node, you will start seeing stats of events ingested, rejected, etc. To enable the realtime metrics monitor, set the following on your realtime node:

druid.monitoring.monitors=["io.druid.segment.realtime.RealtimeMetricsMonitor"]

Also set in your common configuration:

druid.emitter=logging


Afterwards, every minute or so you should see logs about events. I suspect that the timestamp of the events is either malformed, or the events are outside the window period and being rejected. These are the two most common problems I see with not being able to ingest events via realtime ingestion.

Thanks,

FJ

Thanks Fangjin,

The additional debug revealed that the records were thrown away, but it did not give particular reason for it. I used 10 minutes window periods so the data should be able to fit in.

2015-03-29T18:31:55,905 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.904Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/thrownAway”,“value”:0,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,906 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.906Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/unparseable”,“value”:11,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,906 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.906Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/processed”,“value”:0,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,907 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.907Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“rows/output”,“value”:0,“user2”:“druidtest_source”}]

After more careful look at the log, seems the event cannot be parsed, and there are total eleven of them. However, my producer only created 10 of them. Not sure why there is an extra one or if it is related to the cause of the issue. The kafka consumer get the message copied below. I tried to specify with or without millisecond in the timestamp string. Neither works.

2015-03-27 14:41:24,col_1:186,col_2:31,472.0,108.0,161.0

2015-03-27 14:41:24,col_1:43,col_2:98,111.0,343.0,952.0

2015-03-27 14:41:24,col_1:99,col_2:65,158.0,195.0,140.0

2015-03-27 14:41:24,col_1:183,col_2:77,870.0,799.0,409.0

2015-03-27 14:41:24,col_1:56,col_2:249,541.0,95.0,841.0

Finally got it figure out: the time format between date and time should contain ‘T’, i.e.

After more careful look at the log, seems the event cannot be parsed, and there are total eleven of them. However, my producer only created 10 of them. Not sure why there is an extra one or if it is related to the cause of the issue. The kafka consumer get the message copied below. I tried to specify with or without millisecond in the timestamp string. Neither works.

2015-03-27 14:41:24,col_1:186,col_2:31,472.0,108.0,161.0

2015-03-27 14:41:24,col_1:43,col_2:98,111.0,343.0,952.0

2015-03-27 14:41:24,col_1:99,col_2:65,158.0,195.0,140.0

2015-03-27 14:41:24,col_1:183,col_2:77,870.0,799.0,409.0

2015-03-27 14:41:24,col_1:56,col_2:249,541.0,95.0,841.0

Thanks Fangjin,

The additional debug revealed that the records were thrown away, but it did not give particular reason for it. I used 10 minutes window periods so the data should be able to fit in.

2015-03-29T18:31:55,905 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.904Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/thrownAway”,“value”:0,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,906 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.906Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/unparseable”,“value”:11,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,906 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.906Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“events/processed”,“value”:0,“user2”:“druidtest_source”}]

2015-03-29T18:31:55,907 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-29T18:31:55.907Z”,“service”:“realtime”,“host”:“localhost:8084”,“metric”:“rows/output”,“value”:0,“user2”:“druidtest_source”}]

Hi Xiaoming,

If you enable the RealtimeMetricsMonitor on your node, you will start seeing stats of events ingested, rejected, etc. To enable the realtime metrics monitor, set the following on your realtime node:

druid.monitoring.monitors=["io.druid.segment.realtime.RealtimeMetricsMonitor"]


Also set in your common configuration:

druid.emitter=logging

Afterwards, every minute or so you should see logs about events. I suspect that the timestamp of the events is either malformed, or the events are outside the window period and being rejected. These are the two most common problems I see with not being able to ingest events via realtime ingestion.


Thanks,
FJ

This is not required and it defaults to metadata right? But I put this in anyway but its presence did not make any difference (tried both noop and metadata)

do you have this in your runtime properties file

druid.publish.type
Where to publish segments. Choices are “noop” or “metadata”.
metadata

The realtime.properties file should be simple (the default one comments out the host and port and I tried both without luck)

druid.host=localhost

druid.port=8084

druid.service=realtime

druid.processing.buffer.sizeBytes=100000000

druid.processing.numThreads=2

Here is the logs

This will run a stand-alone version of Druid

Running command:

  • java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/home/dev/src/druid-0.7.0/test/test.realtime.spec -classpath ‘/home/dev/src/druid-0.7.0/…/config/realtime:/home/dev/src/druid-0.7.0/test:/home/dev/src/druid-0.7.0/config/_common:/home/dev/src/druid-0.7.0/config/realtime:/home/dev/src/druid-0.7.0/lib/*’ io.druid.cli.Main example realtime

2015-03-28T17:05:24,426 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2015-03-28T17:05:24,429 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

2015-03-28T17:05:25,417 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:mysql-metadata-storage], defaultVersion=‘0.7.0’, localRepository=’/home/dev/.m2/repository’, remoteRepositories=[http://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]

2015-03-28T17:05:25,643 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/druid-examples/0.7.0/druid-examples-0.7.0.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-async/3.0.3/twitter4j-async-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-validator/commons-validator/1.4.0/commons-validator-1.4.0.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar]

2015-03-28T17:05:27,788 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar]

2015-03-28T17:05:27,804 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/druid-kafka-eight/0.7.0/druid-kafka-eight-0.7.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/kafka/kafka-clients/0.8.2.0/kafka-clients-0.8.2.0.jar]

2015-03-28T17:05:28,160 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/xerial/snappy/snappy-java/1.1.1.6/snappy-java-1.1.1.6.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/net/sf/jopt-simple/jopt-simple/3.2/jopt-simple-3.2.jar]

2015-03-28T17:05:28,161 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar]

2015-03-28T17:05:28,180 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.cli.CliCommandCreator]

2015-03-28T17:05:29,206 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/io/druid/extensions/mysql-metadata-storage/0.7.0/mysql-metadata-storage-0.7.0.jar]

2015-03-28T17:05:29,207 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/mysql/mysql-connector-java/5.1.33/mysql-connector-java-5.1.33.jar]

2015-03-28T17:05:29,207 INFO [main] io.druid.initialization.Initialization - Added URL[file:/home/dev/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar]

2015-03-28T17:05:29,410 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,411 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.examples.ExamplesDruidModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,411 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,412 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.firehose.kafka.KafkaEightDruidModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,412 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:29,413 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class[io.druid.initialization.DruidModule]

2015-03-28T17:05:30,235 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@33cdefce]

2015-03-28T17:05:30,252 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=}]

2015-03-28T17:05:30,276 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName=‘realtime’, host=‘localhost’, port=8084}]

2015-03-28T17:05:30,329 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=40, maxIdleTime=PT5M}]

2015-03-28T17:05:30,334 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.RealtimeManagerConfig] from props[druid.realtime.] as [io.druid.guice.RealtimeManagerConfig@7cc8ff30]

2015-03-28T17:05:30,556 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.QueryConfig] from props[druid.query.] as [io.druid.query.QueryConfig@3399ceef]

2015-03-28T17:05:30,575 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@7ff2a3d6]

2015-03-28T17:05:30,581 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [io.druid.query.groupby.GroupByQueryConfig@3171c7df]

2015-03-28T17:05:30,587 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]

2015-03-28T17:05:30,589 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]

2015-03-28T17:05:30,589 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]

2015-03-28T17:05:30,590 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]

2015-03-28T17:05:30,727 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@9f1e040]

2015-03-28T17:05:30,742 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageTablesConfig] from props[druid.metadata.storage.tables.] as [io.druid.metadata.MetadataStorageTablesConfig@609a34a0]

2015-03-28T17:05:30,749 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageConnectorConfig] from props[druid.metadata.storage.connector.] as [DbConnectorConfig{createTables=true, connectURI=‘jdbc:mysql://localhost:3306/druid’, user=‘druid’, password=****}]

2015-03-28T17:05:30,893 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@58d3f4be]

2015-03-28T17:05:30,897 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@6015465]

2015-03-28T17:05:30,901 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29

2015-03-28T17:05:30,957 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.FilteredServerViewProvider] from props[druid.announcer.] as [io.druid.client.FilteredBatchServerViewProvider@10b7415b]

2015-03-28T17:05:30,987 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@3055764c]

2015-03-28T17:05:30,993 INFO [main] org.eclipse.jetty.util.log - Logging initialized @7558ms

2015-03-28T17:05:31,070 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@32d91d5].

2015-03-28T17:05:31,070 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@24c13894].

2015-03-28T17:05:31,070 INFO [main] io.druid.curator.CuratorModule - Starting Curator

2015-03-28T17:05:31,070 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid.icsv.dev

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_75

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/java/jdk1.7.0_75/jre

2015-03-28T17:05:31,079 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/dev/src/druid-0.7.0/…/config/realtime:/home/dev/src/druid-0.7.0/test:/home/dev/src/druid-0.7.0/config/_common:/home/dev/src/druid-0.7.0/config/realtime:/home/dev/src/druid-0.7.0/lib/aws-java-sdk-1.8.11.jar:/home/dev/src/druid-0.7.0/lib/jboss-logging-3.1.3.GA.jar:/home/dev/src/druid-0.7.0/lib/activation-1.1.1.jar:/home/dev/src/druid-0.7.0/lib/curator-framework-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/server-metrics-0.1.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-io-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/spymemcached-2.11.4.jar:/home/dev/src/druid-0.7.0/lib/jetty-security-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jetty-proxy-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/druid-processing-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/guice-multibindings-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/javax.inject-1.jar:/home/dev/src/druid-0.7.0/lib/jackson-jaxrs-smile-provider-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-core-asl-1.9.12.jar:/home/dev/src/druid-0.7.0/lib/commons-codec-1.7.jar:/home/dev/src/druid-0.7.0/lib/druid-api-0.3.4.jar:/home/dev/src/druid-0.7.0/lib/jersey-core-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/maven-settings-builder-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jersey-servlet-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/hibernate-validator-5.1.3.Final.jar:/home/dev/src/druid-0.7.0/lib/jline-0.9.94.jar:/home/dev/src/druid-0.7.0/lib/commons-pool-1.6.jar:/home/dev/src/druid-0.7.0/lib/aether-connector-file-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/jetty-server-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/curator-x-discovery-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/commons-logging-1.1.1.jar:/home/dev/src/druid-0.7.0/lib/aether-impl-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/javax.el-api-3.0.0.jar:/home/dev/src/druid-0.7.0/lib/okhttp-1.0.2.jar:/home/dev/src/druid-0.7.0/lib/RoaringBitmap-0.4.5.jar:/home/dev/src/druid-0.7.0/lib/extendedset-1.3.8.jar:/home/dev/src/druid-0.7.0/lib/log4j-1.2.16.jar:/home/dev/src/druid-0.7.0/lib/maxminddb-0.2.0.jar:/home/dev/src/druid-0.7.0/lib/mail-1.4.7.jar:/home/dev/src/druid-0.7.0/lib/druid-common-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/asm-3.1.jar:/home/dev/src/druid-0.7.0/lib/commons-cli-1.2.jar:/home/dev/src/druid-0.7.0/lib/validation-api-1.1.0.Final.jar:/home/dev/src/druid-0.7.0/lib/aether-connector-okhttp-0.0.9.jar:/home/dev/src/druid-0.7.0/lib/httpcore-4.2.jar:/home/dev/src/druid-0.7.0/lib/classmate-1.0.0.jar:/home/dev/src/druid-0.7.0/lib/aopalliance-1.0.jar:/home/dev/src/druid-0.7.0/lib/guava-16.0.1.jar:/home/dev/src/druid-0.7.0/lib/commons-pool2-2.2.jar:/home/dev/src/druid-0.7.0/lib/opencsv-2.3.jar:/home/dev/src/druid-0.7.0/lib/mapdb-1.0.6.jar:/home/dev/src/druid-0.7.0/lib/java-xmlbuilder-0.4.jar:/home/dev/src/druid-0.7.0/lib/joda-time-2.6.jar:/home/dev/src/druid-0.7.0/lib/jackson-core-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/jackson-datatype-joda-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/maven-model-builder-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/plexus-interpolation-1.19.jar:/home/dev/src/druid-0.7.0/lib/curator-client-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-servlet-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/plexus-utils-3.0.15.jar:/home/dev/src/druid-0.7.0/lib/druid-aws-common-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/jdbi-2.32.jar:/home/dev/src/druid-0.7.0/lib/derbynet-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/derbyclient-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/bcprov-jdk15-1.46.jar:/home/dev/src/druid-0.7.0/lib/java-util-0.26.14.jar:/home/dev/src/druid-0.7.0/lib/bytebuffer-collections-0.1.5.jar:/home/dev/src/druid-0.7.0/lib/org.abego.treelayout.core-1.0.1.jar:/home/dev/src/druid-0.7.0/lib/http-client-1.0.0.jar:/home/dev/src/druid-0.7.0/lib/disruptor-3.3.0.jar:/home/dev/src/druid-0.7.0/lib/mx4j-3.0.2.jar:/home/dev/src/druid-0.7.0/lib/druid-services-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/antlr4-runtime-4.0.jar:/home/dev/src/druid-0.7.0/lib/maven-aether-provider-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jackson-dataformat-smile-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/javax.el-3.0.0.jar:/home/dev/src/druid-0.7.0/lib/derby-10.11.1.1.jar:/home/dev/src/druid-0.7.0/lib/curator-recipes-2.7.0.jar:/home/dev/src/druid-0.7.0/lib/irc-api-1.0-0011.jar:/home/dev/src/druid-0.7.0/lib/zookeeper-3.4.6.jar:/home/dev/src/druid-0.7.0/lib/guice-servlet-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/emitter-0.3.0.jar:/home/dev/src/druid-0.7.0/lib/icu4j-4.8.1.jar:/home/dev/src/druid-0.7.0/lib/netty-3.9.5.Final.jar:/home/dev/src/druid-0.7.0/lib/druid-indexing-service-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/google-http-client-jackson2-1.15.0-rc.jar:/home/dev/src/druid-0.7.0/lib/aether-spi-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/druid-indexing-hadoop-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/commons-io-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/slf4j-api-1.6.4.jar:/home/dev/src/druid-0.7.0/lib/commons-dbcp2-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/jetty-continuation-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jersey-server-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/maven-settings-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/druid-server-0.7.0.jar:/home/dev/src/druid-0.7.0/lib/aether-api-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/geoip2-0.4.0.jar:/home/dev/src/druid-0.7.0/lib/guice-4.0-beta.jar:/home/dev/src/druid-0.7.0/lib/protobuf-java-2.5.0.jar:/home/dev/src/druid-0.7.0/lib/jetty-util-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/javax.servlet-api-3.1.0.jar:/home/dev/src/druid-0.7.0/lib/wagon-provider-api-2.4.jar:/home/dev/src/druid-0.7.0/lib/jetty-client-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/jetty-http-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/maven-model-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jersey-guice-1.17.1.jar:/home/dev/src/druid-0.7.0/lib/commons-lang-2.6.jar:/home/dev/src/druid-0.7.0/lib/maven-repository-metadata-3.1.1.jar:/home/dev/src/druid-0.7.0/lib/jackson-datatype-guava-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/log4j-slf4j-impl-2.1.jar:/home/dev/src/druid-0.7.0/lib/config-magic-0.9.jar:/home/dev/src/druid-0.7.0/lib/jetty-servlets-9.2.5.v20141112.jar:/home/dev/src/druid-0.7.0/lib/xpp3-1.1.4c.jar:/home/dev/src/druid-0.7.0/lib/rhino-1.7R4.jar:/home/dev/src/druid-0.7.0/lib/jets3t-0.9.1.jar:/home/dev/src/druid-0.7.0/lib/httpclient-4.2.jar:/home/dev/src/druid-0.7.0/lib/jsr305-2.0.1.jar:/home/dev/src/druid-0.7.0/lib/airline-0.6.jar:/home/dev/src/druid-0.7.0/lib/jackson-annotations-2.4.4.jar:/home/dev/src/druid-0.7.0/lib/aether-util-0.9.0.M2.jar:/home/dev/src/druid-0.7.0/lib/log4j-core-2.1.jar:/home/dev/src/druid-0.7.0/lib/aws-java-sdk-core-1.8.11.jar:/home/dev/src/druid-0.7.0/lib/

2015-03-27T14:41:24,col_1:186,col_2:31,472.0,108.0,161.0

I found one more issue: not all Aggregation works: I defined three of them, along with count:

“metricsSpec”: [ { “type”: “count”, “name”: “count” }, { “type”: “doubleSum”, “name”: “sum_3”, “fieldName”: “col_3” }, { “type”: “doubleSum”, “name”: “sum_4”, “fieldName”: “col_4” }, { “type”: “doubleSum”, “name”: “sum_5”, “fieldName”: “col_5” } ],

None of the doubleSum worked: all return zero for the following query:

curl -XPOST -H’Content-type: application/json’ “http://localhost:8084/druid/v2/?pretty” -d’{“queryType”: “timeseries”, “dataSource”: “druidtest_source”, “intervals”: [ “2010-01-01/2020-01-01” ], “granularity”: “all”, “aggregations”: [ {“type”: “count”, “name”: “total”}, { “type”: “doubleSum”, “name”: “sum_3”, “fieldName”: “col_3” }, { “type”: “doubleSum”, “name”: “sum_4”, “fieldName”: “col_4” }, { “type”: “doubleSum”, “name”: “sum_5”, “fieldName”: “col_5” } ] }’

The returned json showed all doubleSum is zero. Total count seems to be the only correct one:

[ {

“timestamp” : “2015-03-29T19:03:26.997Z”,

“result” : {

“sum_4” : 0.0,

“total” : 96,

“sum_5” : 0.0,

“sum_3” : 0.0

}

} ]

After talking over IRC, learned that we should query again the renamed column name so that fieldName should be sum_3, sum_4, sum_5. So an example of the query is:

curl -XPOST -H’Content-type: application/json’ “http://localhost:8084/druid/v2/?pretty” -d’{“queryType”: “timeseries”, “dataSource”: “druidtest_source”, “intervals”: [ “2010-01-01/2020-01-01” ], “granularity”: “all”, “aggregations”: [ {“type”: “count”, “name”: “total”}, { “type”: “doubleSum”, “name”: “col_3”, “fieldName”: “sum_3” }, { “type”: “doubleSum”, “name”: “col_4”, “fieldName”: “sum_4” }, { “type”: “doubleSum”, “name”: “col_5”, “fieldName”: “sum_5” } ] }’