Index task are getting failed

Hi Team,

Indexing task of raw data getting failed,previously it was working fine getting success ,but after increasing data volume tasks always getting failed.

In indexing and overlord logs only able to see ,

:00:00.000Z_2019-11-19T13:00:00.000Z_2019-11-19T12:20:23.961Z_2] at existing path[/druid/segments/10.54.55.120:8102/10.54.55.120:8102_indexer-executor__default_tier_2019-11-19T11:50:59.754Z_13c423a5092f49a096f705589a2b58c90]

2019-11-19T12:20:37.657+0000: 1788.292: [GC (Allocation Failure) 2019-11-19T12:20:37.657+0000: 1788.292: [ParNew: 996800K->104368K(996800K), 0.0680385 secs] 3620703K->2795177K(4017452K), 0.0682037 secs] [Times: user=0.65 sys=0.01, real=0.07 secs]

2019-11-19T12:20:40.634+0000: 1791.269: [GC (Allocation Failure) 2019-11-19T12:20:40.634+0000: 1791.269: [ParNew: 990448K->105700K(996800K), 0.0885451 secs] 3681257K->2867790K(4017452K), 0.0886972 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]

2019-11-19T12:20:45.381+0000: 1796.016: [GC (Allocation Failure) 2019-11-19T12:20:45.381+0000: 1796.016: [ParNew: 991780K->110720K(996800K), 0.0914713 secs] 3753870K->2945896K(4017452K), 0.0916234 secs] [Times: user=0.47 sys=0.00, real=0.09 secs]

2019-11-19T12:20:45.473+0000: 1796.108: [GC (CMS Initial Mark) [1 CMS-initial-mark: 2835176K(3020652K)] 2963483K(4017452K), 0.0126342 secs] [Times: user=0.08 sys=0.00, real=0.02 secs]

2019-11-19T12:20:45.486+0000: 1796.121: [CMS-concurrent-mark-start]

2019-11-19T12:20:46.130+0000: 1796.765: [CMS-concurrent-mark: 0.644/0.644 secs] [Times: user=2.80 sys=0.01, real=0.64 secs]

2019-11-19T12:20:46.130+0000: 1796.765: [CMS-concurrent-preclean-start]

2019-11-19T12:20:46.136+0000: 1796.771: [CMS-concurrent-preclean: 0.006/0.006 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

2019-11-19T12:20:46.136+0000: 1796.771: [CMS-concurrent-abortable-preclean-start]

2019-11-19T12:20:48.294+0000: 1798.929: [GC (Allocation Failure) 2019-11-19T12:20:48.294+0000: 1798.929: [ParNew: 996800K->110720K(996800K), 0.0988626 secs] 3831976K->3017234K(4017452K), 0.0990927 secs] [Times: user=0.52 sys=0.02, real=0.10 secs]

CMS: abort preclean due to time 2019-11-19T12:20:52.031+0000: 1802.666: [CMS-concurrent-abortable-preclean: 5.395/5.895 secs] [Times: user=7.64 sys=0.02, real=5.89 secs]

2019-11-19T12:20:52.032+0000: 1802.667: [GC (CMS Final Remark) [YG occupancy: 251812 K (996800 K)]2019-11-19T12:20:52.032+0000: 1802.667: [Rescan (parallel) , 0.0277243 secs]2019-11-19T12:20:52.060+0000: 1802.695: [weak refs processing, 0.0000601 secs]2019-11-19T12:20:52.060+0000: 1802.695: [class unloading, 0.0145241 secs]2019-11-19T12:20:52.074+0000: 1802.709: [scrub symbol table, 0.0088184 secs]2019-11-19T12:20:52.083+0000: 1802.718: [scrub string table, 0.0007976 secs][1 CMS-remark: 2906514K(3020652K)] 3158327K(4017452K), 0.0521266 secs] [Times: user=0.29 sys=0.00, real=0.06 secs]

2019-11-19T12:20:52.084+0000: 1802.719: [CMS-concurrent-sweep-start]

Hi Kishor,
Most likely looks like resource issue as you mentioned issue started after data volume increased and we are also seeing GC issues in the logs.

Some of the things you can try are