Query/time does not improve on historicals after enabling caching

Hey everyone,
I have enabled caffeine cache on one of the historicals on my druid cluster (8GB size).
Hitrate seems to be somewhere around ~65-70% .
However to my surprise , none of the metrics such as “query/time” , “query/cpu/time” seems to have improved for that historical since the time i have enabled the caching.
Infact “query/segment/time” has gone up by many factors .
Also “query/node/ttfb” of broker for that particular historical has not improved also.

Can anyone give me an insight on what can be done here ?