realtime node Full GC

hi all,

I set up a druid cluster, and on the realtime node, the Full GC is very frequency and the cpu load average is also very high. (the broker and the history node is ok)

What’s the problem causing high GC rate and high cpu load?

I set some java attributs like this:
java -Xmx50g -Xms50g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:MaxDirectMemorySize=50g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

the log:

22827.470: [Full GC22827.470: [CMS: 631528K->631508K(50331648K), 1.7160000 secs] 631530K->631508K(52219136K), [CMS Perm : 83967K->83967K(83968K)], 1.7161230 secs] [Times: user=1.71 sys=0.00, real=1.72 secs]

22829.187: [Full GC22829.187: [CMS: 631508K->631509K(50331648K), 1.7205260 secs] 631529K->631509K(52219136K), [CMS Perm : 83967K->83965K(83968K)], 1.7206500 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22830.909: [Full GC22830.909: [CMS: 631509K->631509K(50331648K), 1.7196440 secs] 638232K->631509K(52219136K), [CMS Perm : 83967K->83967K(83968K)], 1.7197680 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22832.630: [Full GC22832.630: [CMS: 631509K->631509K(50331648K), 1.7194840 secs] 633609K->631509K(52219136K), [CMS Perm : 83967K->83967K(83968K)], 1.7196040 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22834.351: [Full GC22834.351: [CMS: 631509K->631510K(50331648K), 1.7182790 secs] 634135K->631510K(52219136K), [CMS Perm : 83967K->83967K(83968K)], 1.7184020 secs] [Times: user=1.71 sys=0.00, real=1.71 secs]

22836.071: [Full GC22836.071: [CMS: 631510K->631510K(50331648K), 1.7149560 secs] 635687K->631510K(52219136K), [CMS Perm : 83967K->83967K(83968K) 83967K->83967K(83968K)], 1.7150900 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22837.788: [Full GC22837.788: [CMS: 631510K->631510K(50331648K), 1.7154750 secs] 642311K->631510K(52219136K) 642311K->631510K(52219136K), [CMS Perm : 83967K->83967K(83968K) 83967K->83967K(83968K)], 1.7156330 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22839.505: [Full GC22839.505: [CMS: 631510K->631518K(50331648K), 1.7155330 secs] 634781K->631518K(52219136K) 634781K->631518K(52219136K), [CMS Perm : 83967K->83967K(83968K)] 83967K->83967K(83968K)], 1.7156770 secs] [Times: user=1.72 sys=0.00, real=1.72 secs]

22841.222: [Full GC22841.222: [CMS: 631518K->631518K(50331648K) 631518K->631518K(50331648K), 1.7165380 secs] 639957K->631518K(52219136K), [CMS Perm : 83967K->83967K(83968K) 83967K->83967K(83968K)], 1.7166860 secs] [Times: user=1.72 sys=0.00, real=1.71 secs]


It seems you have allocated a very big heap of 50G which you might not need.

do you see any errors when you lower the heap size ?

Can you give us some more info on your realtime node config ?

Do you have multiple datasources being ingested in the same realtime node ?

what are the no. of rows that you are ingesting per minute and avg size of each row ?

Thanks Nishant!

We indeed have multiple datasources being ingested in the same realtime node.

But I solve this to add a historycal node.

I think which cause realtime node at a high cpu load and gc rate is the historycal node stopped hands off the segments.