performance historical segment cache

query performance varies very much on different intervals (topN, granularity=all)

  1. interval of 2 days ago to current: > 1 seconds

  2. interval of 2 years ago to current: about 10 seconds

It is apparently that the historical nodes are loading segments: from deep-storage to disk / from disk to memory repeatedly.

other info:

the data retention period is 2-years.

no query cache allowed, for data consistency.

Question:

What is the general guide to tune historical nodes so that to gain performance?

correction:

  1. interval of 2 days ago to current: < 1 seconds

segments are never loaded from deep storage during a query execution.
Historical nodes downloads the segments and memory maps them and serves queries over memory mapped files.

In your case, it seems there are lots of disk page faults, which are causing slowness.

Few things that will help with the performance -

  • switch to SSD if you are not already using them, they have much better performance.

  • Adding more memory to historical nodes will help to reduce disk reads as more data can be served from memory without hitting disk.

Nishant,
Thank you so much for the response. I have following info, could you please guide me for improvement:

On one historical node:

  1. memory size 377G

  2. Java VM parameters: -Xmx16g -Xms16g -XX:NewSize=6g -XX:MaxNewSize=6g -XX:MaxDirectMemorySize=48g

  3. Metrics (average)

---- segment/max 2,083,782,746,535

---- segment/used 136,421,218,624

---- query/cpu/time 463,720

---- query/bytes 4,179

---- query/wait/time 828

---- segment/scan/pending 611

---- segment/count 321

---- query/time 133

---- query/segmentAndCache/time 10

---- query/segment/time 10

---- segment/usedPercent 0.1

  1. runtime.properties:

druid.segmentCache.dropSegmentDelayMillis=180000

druid.segmentCache.locations=[{“path”: “/somedir”, “maxSize”: 1099511627776}]

druid.server.maxSize=1099511627776

Nishant, please help with some advices. Appreciate!

Why aren’t you using a cache? Druid is smart enough to cache only what is immutable.