Historical nodes reload data from deepstorage

Hi all,

Recently because of space constraint, i decreased the data retention for one of our table from 6 month to 3 month. Now after space problem resolution i turned it back to 6 months.

My understanding was that as the data is available in deep storage, historical nodes should reload the data, however that is not happening. I read on internet that i might need to enable the datasource. I tried calling /druid/coordinator/v1/datasources/{dataSourceName} and /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}” , however it is not doing anything.

I was following the instructions on http://druid.io/docs/latest/operations/rule-configuration.html. under heading “**Reloading Dropped Data”. **

Can someone please help in this issue.

Thanks

@Fangjin Yang most of reference points to you. Would really appreciate your help in this.

Hi Anuj,

Your understanding is correct - the data should be in deep storage and should be reloaded by the historical once the load rule is reverted back to 6 months and the datasource is re-enabled.

What response code did you get back from that API? Note that it is an HTTP POST to /druid/coordinator/v1/datasources/{dataSourceName} on the coordinator.

Could you post the output of HTTP GET /druid/coordinator/v1/rules ?

Thanks David for reply.

The POST call was giving out 200 status.

The rules API output for my data source is [

{
“period”: “P30D”,
“tieredReplicants”: {
“tgtTierTest”: 1
},
“type”: “loadByPeriod”
},
{
“type”: “dropForever”
}
]

strangely when i check metastore i can see it knows about all data, however it is not loading it.

http://localhost:4080/druid/coordinator/v1/metadata/datasources/ds-tgt-recommend-bcp/segments

Hey Anuj,

Two things I noticed with your load rule:

  • The period is 30 days and not 6 months as you described (but that should be fine for the list of segments you posted)

  • You’re using a non-default historical tier name (the default is _default_tier). This is fine, but you need to ensure that you set druid.server.tier=tgtTierTest in the historical’s runtime.properties file, otherwise the coordinator will not assign segments to that historical.

If you’re still seeing issues, it would be helpful if you could post a few minutes of your coordinator’s logs, which may contain hints about why it’s not assigning segments.