Query metadata is very very slow and timeout eventually. Help!

We’re trying to use Caravel on top of Druid and during the refresh process, it’s taking a very very long time. And eventually it will timeout. I tried it a couple of times and I tracked the issue down to this metadata query to Druid.

{

“queryType”:“segmentMetadata”,

“dataSource”:“sparrow-firehose-web”,

“intervals”:[“2016-09-06T22:09:55.891000+00:00/2016-09-13T22:09:55.891000+00:00”]

}

I’m not sure if it’s normal and how to debug or improve this?

I did some digging and when I query this endpoint which I think it’s getting the metadata

/druid/coordinator/v1/metadata/datasources/{dataSourceName}

``

If I use CURL and output to a file, it’s about 69MB. I’m not sure if it’s normal at all?

You can cut down how heavy this query is by only asking for the “analysisTypes” you want. By default a lot of them are enabled, and they can be expensive. Also, if you don’t need details of every segment, you can turn on “merge” and possibly also “lenientAggregatorMerge”.

Check the Druid docs for details on all of these options.

Thanks, I’ll check out the doc if anything I might have to submit a PR for Caravel. And is my 69MB size sounds normal? Or is there an optimization I can do?

Segment metadata queries return per-segment information about dims and other stuff, thus the size of the query result is linear to the number of segment covered by the query intervals.

You might be able to merge segments by having less partitions or/and increase the segment granularity but that can have negative impact if the number of rows per physical segment is more than 5M rows.

Hope that helps.

If you set “merge” to true then you should get a result substantially smaller than 69MB, since you’ll get a single metadata object instead of an object per segment.

Hi

socket connections are treated like files and they use file descriptor, which is a limited resource you need to increase the limit for open files.

do ‘ulimit -a’ to find out how many open file handles per process is allowed.