I have a cluster of realtime nodes each on their own ec2 instances with about 16GBs of mem each. I am allocating 13GBs to JVM heap.
Over time, usable memory decreases but never increases until it gets to about 1-2% of usable memory. then the node hangs around, allocating and deallocating until finally it runs out.
Is this normal behavior for realtime nodes to use memory like this? At the moment, it takes a little over a week for a node to run out.
This is runtime.props it that helps:
# Enable Real monitoring