lazyLoadOnStart not working?

I was looking forward to upgrading to 0.19 because I read about druid.segmentCache.lazyLoadOnStart. I have roughly 140,000 segments for some reason (currently on a single node). It takes an hour or two to start the historical service. I set the druid.segmentCache.lazyLoadOnStart to true and fired it up. It’s still taking forever. I see a flood of messages like:

2020-08-22T10:23:57,694 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment cache file [87629/137779][var/druid/segment-cache/info_dir/Event_1_2020-06-27T19:00:00.000Z_2020-06-27T20:00:00.000Z_2020-06-27T18:53:12.884Z_118].

Is this expected? Or am I missing something here?

Thank you!

Tony Schwartz

That would indicate that the Historical is definitely loading your segments before they query… maybe check that druid.segmentCache.lazyLoadOnStart is properly being set in the historical’s runtime properties? https://druid.apache.org/docs/latest/configuration/index.html#historical

We’ve also blocked on this issue in some systems and found that memory mapping operation is induced by initializing segment metadata (like “index.drd”) even with lazy loading is configured. So I’ve changed it not to use memory mapped io for reading meta data and attained what we want (150K segments in 5 minute).

Wish this to be help.

2020년 8월 22일 토요일 오후 7시 25분 5초 UTC+9에 tonysc…@gmail.com님이 작성: