Realtime Data Ingestion

Hi,
I am new to druid and i was trying out the tutorial for realtime data ingestion. But whenever i try out the wikipedia example data is ingested properly and segments are created but why i tried out for my custom data somehow druid is not ingesting data as all messages are pending for consumer. My spec file :

[

{

“dataSchema” : {

“dataSource” : “Titanic”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “timestamp”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions”: [“passengerid”, “survived”, “pcalss”, “name”, “sex”, “age”, “sibsp”, “parch”, “ticket”, “fare”, “cabin”, “embarked”],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec”: [

{

“type”: “count”,

“name”: “count”

},

{

“type”: “doubleSum”,

“name”: “sum_fare”,

“fieldName”: “fare”

}

],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “DAY”,

“queryGranularity” : “NONE”

}

},

“ioConfig” : {

“type” : “realtime”,

“firehose”: {

“type”: “kafka-0.8”,

“consumerProps”: {

“zookeeper.connect”: “localhost:2181”,

“zookeeper.connection.timeout.ms” : “15000”,

“zookeeper.session.timeout.ms” : “15000”,

“zookeeper.sync.time.ms” : “5000”,

“group.id”: “druid-example”,

“fetch.message.max.bytes” : “1048586”,

“auto.offset.reset”: “largest”,

“auto.commit.enable”: “false”

},

“feed”: “testing”

},

“plumber”: {

“type”: “realtime”

}

},

“tuningConfig”: {

“type” : “realtime”,

“maxRowsInMemory”: 500000,

“intermediatePersistPeriod”: “PT10m”,

“windowPeriod”: “PT10m”,

“basePersistDirectory”: “/tmp/realtime/basePersist”,

“rejectionPolicy”: {

“type”: “messageTime”

}

}

}

]

Hi,

Do you see any exceptions in realtime node logs ?

can you share the logs too ?

One other thing that would be useful it to set druid.emitter=logging in your common.runtime.properties and include the RealtimeMetricsMonitor in your realtime runtime.properties. I am curious about the metrics around events ingested/dropped/etc