seeing 502 errors calling druid broker

our client which makes calls to druid-broker is very frequently reporting 502 response codes.
We are using druid 0.8.1.

I can guarantee there is nothing wrong with the query itself.

How do we go about debugging/fixing this.

Can high request rate to broker cause this?
Our server sends about 500 requests in a matter of about 2-3 seconds.

If we send the same query (the one that causes 502) to a standalone broker, it never causes 502.

Hi Prashant, do you see any exceptions from the broker logs? This should not happen as a result of high traffic. Are you putting hte broker behind any kind of load balancer or accessing the IP directly?

the broker is indeed behind a load balancer.

While i do not see any errors in the log, doing a search for ‘502’ comes up with this:

As you can see, some events have ‘timeout’ listed in them but others dont.

Not sure what to make of this.

I don’t think Druid will emit 502s on its own, so that probably means they’re returned by the LB because the brokers are overloaded and not responding. You could try adding more brokers or tuning your existing brokers better (possibly give them more http threads- but also keep an eye on memory use if you do this).

so what does the ‘502’ output in the broker logs means?

I believe those are milliseconds on the log timestamps.