I have realtime nodes running on dedicated ec2 instances inside docker containers and am using datadog to monitor. datadog shows that memory keeps increasing on realtime nodes and never recovers. sometimes it plateaus but it never goes down. I noticed that in my basePersitsDirectory folders from persitsed segments are not getting deleted (there are segments folders from 6 days back). I see logs on realtime nodes that it is removing index.zip so hand off looks like its performing normally and data is in s3. could those lingering folders be an issue with memory usage? are realtime nodes even using these folders still after handoff?
Im ingesting with kafka so I have enough realtimes to balance each partition in my topic. Each realtime alos has 13GB in heap.
What else could be a cause for memory?