Permissions of segment files


We’ve seen that our graphs populated by a (single-machine) druid database contained gaps even though the source values were stable according to all other sources. Previously this had indicated that the historical’s maxSize parameter was set too low (segments not being served). However that was not the case here, It turned out that some segments (happening irregularly but frequent enough that it was noticeable, less than 5% of segments) were not able to be loaded by the historical due to an incorrect owner and permissions.
An example output of ls -lh on /opt/druid-data/ looks like this (bold for emphasis):

drwxr-xr-x 3 druid druid 4096 Jul 3 04:10 2020-07-02T19:00:00.000Z_2020-07-02T20:00:00.000Z
drwxr-xr-x 3 druid druid 4096 Jul 3 05:21 2020-07-02T20:00:00.000Z_2020-07-02T21:00:00.000Z
drwxr-xr-x 3 druid druid 4096 Jul 3 06:21 2020-07-02T21:00:00.000Z_2020-07-02T22:00:00.000Z
drwx------ 3 root root 4096 Jul 3 08:00 2020-07-02T22:00:00.000Z_2020-07-02T23:00:00.000Z
drwxr-xr-x 3 druid druid 4096 Jul 3 12:03 2020-07-03T02:00:00.000Z_2020-07-03T03:00:00.000Z
drwxr-xr-x 3 druid druid 4096 Jul 3 12:03 2020-07-03T03:00:00.000Z_2020-07-03T04:00:00.000Z

Somehow the directories and everything below is owned by root and has reduced permissions. The tasklogs for these segments show no exceptions or errors, just like we didn’t find any traces of this strange behavior in the other component logs.
Out of dozens to hundreds of setups, only two have shown this behavior.

Does anybody here have an idea on what might cause this and how we can resolve it? For now we’re using a cronjob to reset the permissions/owner periodically but that is clearly not ideal.
We’re using druid version 0.13.0-incubating and use kafka-indexing tasks (running 2 datasources which both experience this behavior independently) with one task per datasource.

If more details are needed, please let me know I will be happy to provide configuration details, etc.

Oh interesting! Could this be a process running in the root context somewhere???