Segments not propagated from real-time to historical node even after two weeks

Hi guys,

Once the data are ingested, segments are being created at
druid.segmentCache.locations=/var/tmp/druid/cache (they’re available via real-time node than), but not being propagated to druid.storage.storageDirectory=/var/tmp/druid/segments (druid.storage.type=local), not even after two weeks which makes Plywood
unavailable to GET the data via http://druidbroker:8082/druid/v2/datasources/stats ({“dimensions”:,“metrics”:})

Please how to get segments created to /var/tmp/druid/segments too, thus being available via historical node?

Also, not related to Drui directly, but how to GET the data from Druid older then two weeks via mentioned URL (keep getting Error: Can not use GET route, probably data is in a real-time node or more than a two weeks old)?

If I can provide any logs please just let me know.

Any help or insight would be much appreciated, tnx!

Shinesun

Hi Shinesun,

I’m not sure about the segment propagation question, but for your second question about GET, you can specify a interval string parameter with the GET request:

http://druid.io/docs/latest/design/broker.html

Alternatively, if you’re using Druid 0.8.1 or later, you can increase the default interval period used by the GET route by setting druid.query.segmentMetadata.defaultHistory in the broker configuration:

http://druid.io/docs/latest/configuration/broker.html

  • Jon

Hi Shinesun,

I wonder if you are running with none rejection policy, which might be causing the hand-off to not work ?

fwiw, rejectionPolicy, segmentGranularity and windowPeriod controls when the segments are handed off to historical nodes.

More details on how it works can be found here http://druid.io/docs/latest/ingestion/realtime-ingestion.html

There’s a lot of suggestions here http://druid.io/docs/latest/ingestion/faq.html around debugging failed handoff.

Hey guys,

Sorry for not replying earlier and thanks for helping.

I’m not sure about the segment propagation question, but for your second question about GET, you can specify a interval string parameter with the GET request:
http://druid.io/docs/latest/design/broker.html
Alternatively, if you’re using Druid 0.8.1 or later, you can increase the default interval period used by the GET route by setting druid.query.segmentMetadata.defaultHistory in the broker configuration:
http://druid.io/docs/latest/configuration/broker.html

Although it’s OK when working with Druid directly, question was related to Plywood, and an asnwer is available here: https://groups.google.com/d/msg/imply-user-group/gqYw3yIkMec/h7nu7d-tDQAJ

I wonder if you are running with none rejection policy, which might be causing the hand-off to not work ?
fwiw, rejectionPolicy, segmentGranularity and windowPeriod controls when the segments are handed off to historical nodes.
More details on how it works can be found here http://druid.io/docs/latest/ingestion/realtime-ingestion.html

I’m using batch ingestion and not quite sure how, but after Druid cluster re-configuratation per this article at http://druid.io/docs/latest/configuration/production-cluster.html segments started to show as expected. Though events weren’t dropped off (they were available from cache), but just not getting segments created afterwards.

Thank once again y’all.

Regards,

Shinesun