I was using jvisualvm to monitor thread behaviour on historicals and brokers in a Druid test cluster. Now, for a historical node, I can make sense of the threads that I see. If I specify in the Druid runtime.properties config file that I wish to have 31 processing threads then I end up seeing those threads. If I say that I want 50 http threads, then I can also see exactly 50 of them in the thread-view.
On the broker however, the threads that I see don’t seem to relate to my config settings at all.
Below for instance is the broker config I set up for the purpose of seeing how this config would translate to threads shown in jvisualvm:
broker config (stupid setup - it’s just for testing)
Below is a screenshot from the jvisualvm thread overview. As can be seen, I’ choosing to see all threads.
On the historical, the processing threads are named “processing-XX”, but on the broker I don’t see any processing threads. I declared 70 of them so they should show up prominently.
I DO see the 10 http threads showing up as “qtp*”. Interestingly one of them is an acceptor and four of them are selectors. This is on a 2xl node. The historical I monitor is an 8xl node and always ends up having 4 selectors and 4 acceptors. Don’t know if this is an automatic setting.
I also wonder why regardless of which changes I make to the broker config, I seem to end up with 15 Netty threads and they are always in state “running”.
Even though the cache setting enables segment merging on the historicals, the broker still needs to merge the results it gets back from the historicals. As I have two historicals, I would imagine that the broker needs at least one processing thread for the merging. Where else would the merging happen?
Can anybody shed some light?