Druid Suddenly failed to ingestion

Hi all,

I have a serious ingestion issue in my production server. My druid suddenly failed to ingestion

Attached is the log file. Appreciate some urgent help

error.log (104 KB)

Hi Ls Poh,
I see this error in the ingestion log :

2020-04-25T01:04:58,663 ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [1] unparseable events discarded. Turn on debug logging to see exception stack trace.

Based on your ingestion spec [ “reportParseExceptions” : false] which means unparseable rows and fields will be skipped and hence yor job should proceed . I don’t see any thing else in the log.

Could you check the middlemanager and overlord log grepping this task id to know what has happened with this task.

Thanks and Regards,

Vaibhav

Thanks for the reply. This morning i tried again and it showed me out of memory error now.

Before this if I ingest data by month the date will delay 1 day, example, count for 22/4 will become 21/4… Therefore I decided to ingest day by day and data is tally… but after run for 780 segments i hits this issue

How can I resolved this?

druid.png

error_log.txt (2.49 MB)

Thanks. This will be great. Let me know when will be the best time for this. Really appreciate it

druid.png

Where do you see the OOM error? Could you share the log?

Thanks, and regards,
Vaibhav

druid.png

Finally issue resolved after upgraded to druid version 0.17.1… Thanks so much for the help :smile:

druid.png