Historical node - No space left on device

Hello everyone!

We’re using druid version 0.6.160. We have following problems on production: from time to time, broker reports following error:

HTTP/1.1 500 Server Error
Date: Wed, 15 Apr 2015 14:28:13 GMT
Content-Type: text/plain
Connection: close
Server: Jetty(9.2.3.v20140905)

{“error”:“java.io.IOException: No space left on device”}
After some digging, it looks like one of our historical nodes (currently there are ~6 of them) had a near 250GB of segments in segments cache, which exhausted our disk capabilities.
One day after this event i’ve executed following commands to investigate why historical is overloaded:

root@ip-10-0-100-163:/var/lib/druid/data# du --max-depth=0 -h /var/lib/druid/data
198G	/var/lib/druid/data
 

Historical node config looks like this:

Hi Jarek,

you might be hitting the issue fixed in https://github.com/druid-io/druid/pull/855 which can lead to leaking files in segment cache.

Can you try upgrading to latest 0.7 branch ?