[druid-user] Kafka Ingestion Failed Tasks

Hi Team,

I have been seeing failures with Kafka Ingestion. The tasks are in ‘Failed’

state. The supervisor is in ‘UNHEALTHY’ state. The ingestion is not complete and none of the new tasks are successful.

I have my doubts on the consistent logging of ‘Flushed in-memory data for segment …’ but I am not sure what to make out of it. Attached Log for reference.

The configuration is ‘conf/druid/single-server/micro-quickstart’ . The machine has 8GB ram. There is enough space on HDD to assimilate data. Historical usage in only 0.5% at the moment.

Please help to debug this. Thanks for your time.

(Attachment index_kafka_tbt_data2_690d6435b564613_cihaeipo.log is missing)


Sorry to hear that your ingestion isn’t complete, and thanks for attaching the log.

I looked for warnings in the log. One particular warning jumped out:

2022-03-12T22:35:24,476 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS]

I wonder if you hit this issue?

Regarding the “Flushed in-memory data for segment …” message, it looks like it’s part of the Appenderator class, and it looks like it returns the number of rows persisted.