Druid Live Ingestion stopped after 1 day

Hi We are currently working on a POC, where in we used Druid to Ingest data from Kafka, Some reason ingestion is stopped after 1 day, previous segments are published to deepstorage (s3) and status is success, but new tasks are created and are running state, no error logs in coordinator, middle manager and historical node, but live query gives empty result. below screenshots are for reference.

{
“type”: “kafka”,
“spec”: {
“ioConfig”: {
“type”: “kafka”,
“consumerProperties”: {
“bootstrap.servers”: “XXXX:9092”
},
“topic”: “liveDataTopic”,
“inputFormat”: {
“type”: “json”
},
“useEarliestOffset”: true,
“taskCount”: 44
},
“tuningConfig”: {
“type”: “kafka”,
“maxRowsPerSegment”: 9000000,
“workerThreads”: 10,
“chatThreads”: 10
},
“dataSchema”: {
“dataSource”: “liveDataTopic”,
“timestampSpec”: {
“column”: “published_at”,
“format”: “iso”
},
“dimensionsSpec”: {
“dimensions”: [
“transmission_gear_position”,
{
“type”: “long”,
“name”: “vehicle_speed”
},
{
“type”: “double”,
“name”: “torque_at_transmission”
},
{
“type”: “double”,
“name”: “engine_speed”
},
{
“type”: “double”,
“name”: “fuel_consumed_since_restart”
},
{
“type”: “double”,
“name”: “odometer”
},
{
“type”: “double”,
“name”: “fuel_level”
},
{
“type”: “double”,
“name”: “latitude”
},
{
“type”: “double”,
“name”: “longitude”
},
{
“type”: “long”,
“name”: “accelerator”
},
{
“type”: “long”,
“name”: “brake”
},
{
“type”: “long”,
“name”: “angle”
},
“parking_brake_status”,
“engine_running”,
“ignition_status”,
“brake_pedal_status”,
{
“type”: “long”,
“name”: “accelerator_pedal_position”
},
“gear_lever_position”,
“window_details”,
“tire_pressure”,
“manual_trans”,
“vin”,
“event_type”,
{
“type”: “long”,
“name”: “transmission_gear_int”
},
“consumed_at”
]
},
“granularitySpec”: {
“queryGranularity”: “none”,
“rollup”: false,
“segmentGranularity”: “hour”
}
}
}
}

Welcome @Nagaraja_h! Sorry for the long delay in replying, but several of us were offsite.

What druid.emitter.logging.logLevel are you using? Maybe debug logging can shed some light on the behavior you’re reporting?

Best,

Mark

Are the segments from the tasks that completed “available”? If not, there may be a problem in the coordinator or Historicals. you can start debugging this by looking at their logs for any Errors.

I referred to configured parameters, I have not set those parameters

yes completed segments are available, but newly created segments says not realtime

I’m not sure I understand the problem you are facing. When you say “but live query gives empty result” what query are you using to test this?

A few thoughts:

Segments are only “Is realtime”=true while they are being ingested in real-time. Segments created by streaming ingestion tasks will show up as “realtime” while they are being created, segments that have already been published and loaded into historicals will show realtime=false. All segments, whether realtime or not, are queryable.

One thing I noticed is that your segments are very small, about 50000 rows each. You could probably use a lot less tasks (taskCount parameter) and produce bigger segments. Each task will produce a segment for each time granularity, so you are creating 44 segments per hour. If you have 44 partitions in kafka you could try 22 tasks or even 11, multiple kafka partitions will be assigned to each task, such that you still keep up with the streaming throughput needs but produce less segments that are larger.

It is also a good practice to setup auto-compaction to further merge small segments after streaming ingestion. By combining streaming ingestion with many tasks to drive ingestion throughput and compaction to optimize segment size, you improve both real-time throughput and historical data query performance.