Druid remove segments automatically

Hi all,
I have hundreds of historical segments serving on one druid cluster, running at version 0.6.51, but I found these segments are removed automatically once again.

Segment info:

timestamp from 2014-01-01 to yesterday, granularity by day, so there are about 365+ segments

each size is 200M + to 300M

Cluster info:

total capacity is about 150GB, two datasources are held on default-tier, we don’t have other tiers configured. I capacity is large enough to hold all data.

By tracing in code, I found segments are removed by the following codes in coordinator, like some timeline overshadow:

// Delete segments that are old
// Unservice old partitions if we've had enough time to make sure we aren't flapping with old data
if (params.hasDeletionWaitTimeElapsed()) {
  Map<String, VersionedIntervalTimeline<String, DataSegment>> timelines = Maps.newHashMap();

  for (MinMaxPriorityQueue<ServerHolder> serverHolders : cluster.getSortedServersByTier()) {
    for (ServerHolder serverHolder : serverHolders) {
      DruidServer server = serverHolder.getServer();

      for (DruidDataSource dataSource : server.getDataSources()) {
        VersionedIntervalTimeline<String, DataSegment> timeline = timelines.get(dataSource.getName());
        if (timeline == null) {
          timeline = new VersionedIntervalTimeline<String, DataSegment>(Comparators.comparable());
          timelines.put(dataSource.getName(), timeline);
        }

        for (DataSegment segment : dataSource.getSegments()) {
          timeline.add(
              segment.getInterval(), segment.getVersion(), segment.getShardSpec().createChunk(segment)
          );
        }
      }
    }
  }

  for (VersionedIntervalTimeline<String, DataSegment> timeline : timelines.values()) {
    for (TimelineObjectHolder<String, DataSegment> holder : timeline.findOvershadowed()) {
      for (DataSegment dataSegment : holder.getObject().payloads()) {
        coordinator.removeSegment(dataSegment);
        stats.addToGlobalStat("overShadowedCount", 1);
      }
    }
  }
}

Does anyone run through the same issue ?

Thanks

YL

Hi Yanli, some questions inline.

Hi all,
I have hundreds of historical segments serving on one druid cluster, running at version 0.6.51, but I found these segments are removed automatically once again.

When you say segments are removed, do you mean they are removed automatically after reindexing or just removed suddenly?

Hi FJ,

they are removed several hours after indexed into Druid, I manually set dataSource enabled, then segments were removed again.

For loading rules, I tried both loadByInterval: 2014-01-01/2020-01-01(which works for another datasource), and loadByPeriod: P500Y

They are removed after new segments are indexed? As in, are you generating a new set of segments with a higher version number that is causing Druid to automatically invalidate these segments?

Each segment is indexed once, and one version is kept for each segment

Do you have the logs for the coordinator around the time the segments are dropped? This is an older version of Druid, but I don’t recall any bugs of this nature in that version.