realtime node ingestion rate

Hi all,

I use realtime node ingestion from kafka. But the ingestion rate is only 100 events/s.

The kafka topic is set to 30 partitions and 3 replications.

The realtime node is in one machine with 60G memory and shardings is set to 20 .

How to tune the realtime ingestion performance?Anyone can help?

Thanks.

My realtime node spec is:

[

{

“dataSchema” : {

“dataSource” : “order”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “timestamp”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions”: [“area”,“status”],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec” : [{

“type” : “count”,

“name” : “count”

}, {

“type” : “longSum”,

“name” : “bonus”,

“fieldName” : “bonus”

}],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “HOUR”,

“queryGranularity” : “NONE”

}

},

“ioConfig” : {

“type” : “realtime”,

“firehose”: {

“type”: “kafka-0.8”,

“consumerProps”: {

“zookeeper.connect”: “localhost:2181”,

“zookeeper.connection.timeout.ms” : “15000”,

“zookeeper.session.timeout.ms” : “15000”,

“zookeeper.sync.time.ms” : “5000”,

“group.id”: “didi_monitor”,

“fetch.message.max.bytes” : “1048586”,

“auto.offset.reset”: “smallest”,

“auto.commit.enable”: “false”

},

“feed”: “order”

},

“plumber”: {

“type”: “realtime”

}

},

“tuningConfig”: {

“type” : “realtime”,

“maxRowsInMemory”: 500000,

“intermediatePersistPeriod”: “PT1m”,

“windowPeriod”: “PT1m”,

“basePersistDirectory”: “/data/logs/druid/realtime/basePersist”,

“rejectionPolicy”: {

“type”: “messageTime”

},

“shardSpec”: {

“type”: “linear”,

“partitionNum”: 0

}

}

“shardSpec”: {

“type”: “linear”,

“partitionNum”:1

}

“shardSpec”: {

“type”: “linear”,

“partitionNum”:19

}

},

That is incredibly slow. Do you mind posting your metrics for your realtime ingestion?

If you are using the latest stable (0.8.0), there should metrics printed in the logs with “ingest/*”. Those would be interesting

Hi Fangjin,

I enable the druid self metric,but the logs are very strange: all the “ingest” events’ values are “0”. But I can query total_number from http api.

the metric logs:

2015-07-23T20:53:06,477 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-07-23T20:53:06.477+08:00”,“service”:“realtime”,“host”:“localhost:8080”,“metric”:“ingest/events/processed”,“value”:0,“dataSource”:“order”}]

2015-07-23T20:53:06,477 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-07-23T20:53:06.477+08:00”,“service”:“realtime”,“host”:“localhost:8080”,“metric”:“ingest/rows/output”,“value”:0,“dataSource”:“order”}]

2015-07-23T20:53:06,477 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-07-23T20:53:06.477+08:00”,“service”:“realtime”,“host”:“localhost:8080”,“metric”:“ingest/persists/count”,“value”:0,“dataSource”:“order”}]

2015-07-23T20:53:06,477 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-07-23T20:53:06.477+08:00”,“service”:“realtime”,“host”:“localhost:8080”,“metric”:“ingest/persists/time”,“value”:0,“dataSource”:“order”}]

The query is:

curl -XPOST -H’Content-type: application/json’ “http://localhost:8080/druid/v2/?pretty” -d '{

“queryType”: “timeseries”,

“dataSource”: “order”,

“granularity”: {“type”: “duration”, “duration”: 86400000, “timeZone”:“Asia/Shanghai”, “origin”: “2015-07-15T00:00:00Z”},

“aggregations”: [

{

“type”: “count”,

“name”: “count”

}

],

“intervals”: [

“2015-07-15T00:00:00.000/2015-07-30T00:00:00.000”

]

}’

the query result is:

[ {

“timestamp” : “2015-07-14T00:00:00.000Z”,

“result” : {

“count” : 144660

}

}, {

“timestamp” : “2015-07-15T00:00:00.000Z”,

“result” : {

“count” : 16322566

}

} ]

Andy, this query is not counting the number of events ingested. This is counting the number of Druid rows after rollup has occurred (http://druid.io/docs/latest/design/index.html). What you want to do is use a count aggregator in your ingestion spec and a longSum aggregator at query time.