Druid segments missing from the query

Apache Druid Q&A Google Groups
#1

Hi,

We are using Druid 0.10.1. We had segments from two days missing from the query. The missing segments are shown from the coordinator console. The segments ‘used’ flags are set to True in the metadata db store. The segments were also pulled from deep storage (S3) to the local segmentCache on historical nodes. But the data from the segments didn’t return from Druid query. And the ‘segmentMetadata’ query returned empty for the segments. We didn’t find any meaningful logs related to the issue. Does anyone know what’s going on here?

Best,

Jing

query_missing_data.png

segments_shown_from_console.png

#2

Anyone?

A little more background of this issue:

  1. We were using Kafka to ingest the data to Druid

  2. Then we reindexed those segments using native ‘index’ type of ‘ingestSegment’

  3. Then we found incorrect data of the segments, it’s not caused by ingestion but our application logic issue. So we dropped those segments

  4. Then we re-ingest those segments using Kafka after the segments were dropped

  5. The we saw missing segments from query after those segments were re-ingested

Could it be caused by segment version conflict?

Best,

Jing

#3

Would like to check if anyone knows what’s going on here? Thanks.

#4

Hey Jing,

Not sure how helpful this is but if the coordinator see these segments and the historicals has loaded them, maybe something weird’s happening with the broker’s cache of cluster state? You could try modifying the query to specify the exact interval covered by the segment and issue a query directly at an historical to see if this is the case.

If that query returns the data, restarting the brokers and monitoring their logs for issues would be a good way of figuring out what’s going on.

Best regards,

Dylan

#5

Thanks Dylan.

Yeah the weird thing is the historical node didn’t return the segments, the query was sent to historical node directly:

{

“queryType”:“segmentMetadata”,

“dataSource”:“xxxxx”,

“intervals”:[“2019-12-07/2019-12-09”]

}

and it returned empty results.

So the coordinator listed the segments, but the historical node didn’t return the segments.

Best,

Jing