Segment check process

Hi!

We believed that some of our Druid segments were corrupt so we made a small bash process to dynamically check all segments within a node/source. The process is using Druid’s core segment-dump tool.

You can find it attached, and it may help you a little bit when troubleshooting errored segments.

We could identify a wrong segment and re-cache it. Feel free to change it as desired.

Hope it helps someone.

Regards,

segmentFailure.sh (1008 Bytes)

Sweet, that looks cool.

Although, fwiw, this won’t detect all kinds of corruption. To do that I’d compare the checksum of segment *.smoosh files against what’s in the index.zip on deep storage. The index.zip has a checksum on it (crc32 I think) so when you unpack that it’ll also verify the deep storage copy.