Data source not being created using tranquility.

Hi All,

I was facing packet drops in 0.6.170 in our setup and so we moved to 0.7.0. With the new version, my datasource is not being created nor is my data showing up in druid.

Configs:

I have druid-0.7.0 local setup with 1 - overlord, 1- coordinator and 1 - broker node. I am using com.metamx:tranquility_2.10:0.3.1

I am trying to setup our data pipeline using kakfa-storm-druid. I have 2 vms for druid cluster setup. 1 vm has all the druid nodes running and in the other I have mysql and zookeeper-3.4.6 running. I have resolved the following issues in my setup:

  1. Addition of “druid.indexer.task.chathandler.type=announce” property into my overlord configurations.
  2. Change of timestamp to ISO 8601 date format. - yyyy-MM-dd’T’HH:mm:ss’Z’
  3. Set my events with a timestamp name/value pair.
  4. Modified my indexService to match my overlord configs service path.
  5. Made all my VMs have the same timezone. We use PDT standard time zone for this.
    What I have tried:
  1. Check the console if the task is being registered and starting off.

I see this happening as well as the overlord console showing the task. In one of the discussions

  1. I don’t see any errors logs in my tranquility logs or task logs.

  2. I also tried looking into my zookeeper logs. The only thing I could figure out from these logs were the highlighted part. Not sure if that’s
    normal.
    2015-03-13 00:16:41,068 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.244.200.103:54699
    2015-03-13 00:16:41,069 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
    attempting to establish new session at /10.244.200.103:54699
    2015-03-13
    00:16:41,071 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14c0b9a8a4e000d with negotiated timeout 30000 for client /10.244.200.103:54699
    2015-03-13 00:18:39,368 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - ** Got
    user-level KeeperException when processing sessionid:0x14c0b9a8a4e000d type:create cxid:0x9 zxid:0x1d13 txntype:-1 reqpath:n/a Error Path:/tranquility/beams/overlord/newtopic1 Error:KeeperErrorCode = NoNode for /tranquility/beams/overlord/newtopic1**
    2015-03-13 00:18:39,521 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14c0b9a8a4e000d type:create cxid:0x10 zxid:0x1d16 txntype:-1 reqpath:n/a Error Path:/tranquility/beams/overlord/newtopic1/mutex/locks Error:KeeperErrorCode = NoNode for /tranquility/beams/overlord/newtopic1/mutex/locks
    2015-03-13 00:18:39,742 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14c0b9a8a4e000d type:create cxid:0x1b zxid:0x1d1a txntype:-1 reqpath:n/a Error Path:/tranquility/beams/overlord/newtopic1/mutex/leases Error:KeeperErrorCode = NoNode for /tranquility/beams/overlord/newtopic1/mutex/leases
    2015-03-13 00:18:45,769 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.65.204.10:58718
    2015-03-13 00:18:45,774 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
    attempting to establish new session at /10.65.204.10:58718
    2015-03-13
    00:18:45,777 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x14c0b9a8a4e000e with negotiated timeout 30000 for client /10.65.204.10:58718
    2015-03-13 00:18:48,132 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14c0b9a8a4e000e type:create cxid:0x27 zxid:0x1d24 txntype:-1 reqpath:n/a Error Path:/druid/discovery/newtopic1-18-0000-0000 Error:KeeperErrorCode = NoNode for /druid/discovery/newtopic1-18-0000-0000
    2015-03-13 00:25:00,498 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for
    sessionid: 0x14c0b9a8a4e000e
    When I tried looking up the datasource newtopic1 in my broker node /druid/v2/datasources/ (end point) I found nothing there.

  3. Gian mentioned to review the service discovery for task logs. Please find those logs below:

2015-03-13T07:18:48,094 INFO [task-runner-0] io.druid.segment.realtime.firehose.EventReceiverFirehoseFactory - Found chathandler of class[io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider]
2015-03-13T07:18:48,094 INFO [task-runner-0] io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Registering Eventhandler[druid:firehose:newtopic1-18-0000-0000]
2015-03-13T07:18:48,097 INFO [task-runner-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘druid:firehose:newtopic1-18-0000-0000’, host=‘10.65.204.10’, port=8088}]
2015-03-13T07:18:48,131 INFO [task-runner-0] io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Registering Eventhandler[newtopic1-18-0000-0000]
2015-03-13T07:18:48,132 INFO [task-runner-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘newtopic1-18-0000-0000’, host=‘10.65.204.10’, port=8088}]
2015-03-13T07:18:48,147 INFO [task-runner-0] io.druid.data.input.FirehoseFactory - Firehose created, will shut down at: 2015-03-13T00:25:00.000-07:00
2015-03-13T07:18:48,161 INFO [task-runner-0] io.druid.segment.realtime.plumber.RealtimePlumber - Creating plumber using rejectionPolicy[serverTime-PT1M]
Here is the POST request for the task:

{
  "type" : "index_realtime",
  "id" : "index_realtime_newtopic1_2015-03-13T00:18:00.000-07:00_0_0_hkffidca",
  "resource" : {
    "availabilityGroup" : "newtopic1-18-0000",
    "requiredCapacity" : 1
  },
  "spec" : {
    "dataSchema" : {
      "dataSource" : "newtopic1",
      "parser" : {
        "type" : "map",
        "parseSpec" : {
          "format" : "json",
          "timestampSpec" : {
            "column" : "timestamp",
            "format" : "auto"
          },
          "dimensionsSpec" : {
            "dimensions" : [ "expt_id", "msg_id", "name", "tr_id" ],
            "dimensionExclusions" : [ "timestamp", "value" ],
            "spatialDimensions" : [ ]
          }
        }
      },
      "metricsSpec" : [ {
        "type" : "count",
        "name" : "cnt"
      }, {
        "type" : "min",
        "name" : "max_value",
        "fieldName" : "value"
      } ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "MINUTE",
        "queryGranularity" : {
          "type" : "duration",
          "duration" : 60000,
          "origin" : "1970-01-01T00:00:00.000Z"
        },
        "intervals" : null
      }
    },
    "ioConfig" : {
      "type" : "realtime",
      "firehose" : {
        "type" : "clipped",
        "delegate" : {
          "type" : "timed",
          "delegate" : {
            "type" : "receiver",
            "serviceName" : "druid:firehose:newtopic1-18-0000-0000",
            "bufferSize" : 100000
          },
          "shutoffTime" : "2015-03-13T00:25:00.000-07:00"
        },
        "interval" : "2015-03-13T00:18:00.000Z/2015-03-13T00:19:00.000Z"
      }
    },
    "tuningConfig" : {
      "type" : "realtime",
      "maxRowsInMemory" : 75000,
      "intermediatePersistPeriod" : "PT10M",
      "windowPeriod" : "PT1M",
      "basePersistDirectory" : "/var/folders/x8/xw0rndr55svfmlkm2mq41kf0393hyj/T/1426231120517-0",
      "versioningPolicy" : {
        "type" : "intervalStart"
      },
      "rejectionPolicy" : {
        "type" : "serverTime"
      },
      "maxPendingPersists" : 1,
      "shardSpec" : {
        "type" : "linear",
        "partitionNum" : 0
      },
      "persistInHeap" : false,
      "ingestOffheap" : false,
      "bufferSize" : 134217728
    }
  },
  "groupId" : "index_realtime_newtopic1",
  "dataSource" : "newtopic1"
}

My message format: {timestamp=2015-03-13T07:18:46Z, name=abc, msg_id=2, attr=1426231126791, tr_id=123, expt_id=111}

Hi Dhawan, I believe this is the same issue as in this thread: https://groups.google.com/forum/#!topic/druid-user/eFr467HScaM. Is that right? If so let’s discuss there.

Yes Gian, that’s totally cool.