druid.broker.cache.cacheBulkMergeLimit and caches

Two questions about druid.broker.cache.cacheBulkMergeLimit:

  1. A terminology question: A “segment” is sharded into “partitions” when data gets indexed. Does the limit refer to number of segments, or the number of partitions? I’m guessing the latter, but want to be sure.

  2. When testing this limit, enabling a broker-level cache seemed to disable the cache on the historicals altogether. Shouldn’t both of them be active when the limit is being used?
    (Running Druid 0.9.2)

Thanks,

  • Tomas

Good questions, answered inline below:

Two questions about druid.broker.cache.cacheBulkMergeLimit:

  1. A terminology question: A “segment” is sharded into “partitions” when data gets indexed. Does the limit refer to number of segments, or the number of partitions? I’m guessing the latter, but want to be sure.

Your guess is correct. The nomenclature internally is not as consistent as it should be, but each “data blog which has a unique ID” (partition in your example) gets a cache entry.

  1. When testing this limit, enabling a broker-level cache seemed to disable the cache on the historicals altogether. Shouldn’t both of them be active when the limit is being used?
    (Running Druid 0.9.2)

It should behave the same way as a cache miss at the broker level, aka: pass the work to the historicals. If the historical is configured to use cache then it should do its own checking against the cache when the query reaches it. If the historical is not configured to use cache then it will not.

We have the bulk merge limit enabled at the broker level and see cache hits at the historical.

Thanks for the reply, that clears things up.

It should not matter if the Broker cache type is local or memcached, right?

  • Tomas

And another follow-up: I do see historical cache hits now — but don’t see the historical cache getting populated with new entries in this configuration.
(The hits correspond only to queries issued before the broker cache was turned on.) Is this the intended behavior? (Druid 0.9.2)