historical cache size limit and query unused fragments

Hi,

we have a druid cluster composed by one realtime, one historical, one broker and one coordinator.

We started from druid version 0.8.2-snapshot and pulled this: https://github.com/druid-io/druid/pull/1719 (we use cloudfiles as deep storage)

Cluster seems to works fine, but we have some questions:

  • in historical node we have:

druid.segmentCache.locations=[{“path”: “/tmp/druid/indexCache”, “maxSize”: 42949672960 }]

``

druid.server.maxSize=42949672960

``

The max size is 40GB but, used space grows beyond this limit. Directory /tmp/druid/indexCache/$datasource has reached 64GB. Why? What we wrong?

  • through coordinator consolle we set following rules:

loadByPeriod P15D

``

dropForever

``

So, historical node contains only last 15 days fragments. When we try to perform a query that involves fragments older than 15 days no data is returned by broker. Should historical node downloads from deep storage needed fragments or we misunderstood something and we can perform query only on last 15 days due to our rules configuration?

Thanks in advance,

Fx

Inline.

Hi,

we have a druid cluster composed by one realtime, one historical, one broker and one coordinator.

We started from druid version 0.8.2-snapshot and pulled this: https://github.com/druid-io/druid/pull/1719 (we use cloudfiles as deep storage)

Cluster seems to works fine, but we have some questions:

  • in historical node we have:

druid.segmentCache.locations=[{“path”: “/tmp/druid/indexCache”, “maxSize”: 42949672960 }]

``

druid.server.maxSize=42949672960

``

The max size is 40GB but, used space grows beyond this limit. Directory /tmp/druid/indexCache/$datasource has reached 64GB. Why? What we wrong?

This is odd, and I’m guessing certain segments have not yet been cleaned up. Can you include a SS of the directory?

  • through coordinator consolle we set following rules:

loadByPeriod P15D

``

dropForever

``

So, historical node contains only last 15 days fragments. When we try to perform a query that involves fragments older than 15 days no data is returned by broker. Should historical node downloads from deep storage needed fragments or we misunderstood something and we can perform query only on last 15 days due to our rules configuration?

Segments are never downloaded on demand from deep storage. We’d like to support this in the future. With this configuration, you’ll only be able to query last 15 days of data.

Hi,

do you perhaps now how the others ingesting billion of events a day handles queries involving time ranges older than the retention time?

Hi,
thank you for your response.

Here: http://pasted.co/3ea8c62c a find of my indexCache/$datasource directory. I see some discontinuity, for example segments for day 2015-12-08 jump from 14:00 - 15:00 to 17:00 - 18:00. Maybe is this the cause?

Here: http://pasted.co/faedfd39 a du of the dir.

thanks in advance,

Fx

Reading through the code, I think what is happening is that the CloudFilesDataSegmentPusher when it creates the segment is storing the compressed size of the segment instead of the uncompressed size, which is what S3DataSegmentPusher and others do.

Hi Fangjin,

in this case, I think the Azure Extension has the same issue:

So, if I’ve understood correctly, druid computes the cache size by summing up the segment size from the segment’s metadata instead of the actual bytes written in the disk. I’m going to fix the bug soon.

thanks,

Maurizio

Thanks Maurizio. For S3 and HDFS, the size stored in the metadata is the uncompressed size on disk.

Just opened the PR to fix the cloudfiles extension bug, I guess we should open an issue to keep track of the Azure bug too.
Please find the PR at the following url: https://github.com/druid-io/druid/pull/2191 .

Maurizio

Thanks Maurizio!