Tranquility stop stream ingestion from time to times.

Hi druids,
I’ve installed a full druid stack (0.9.0), and I’m streaming events into it from kafka (0.8.2.2) topics via tranquility (0.8).

The events are indexed in “hours” segments.

it works pretty well, but some times (I am not able to describe a reproductible use case :’( )

All I can say is that I have:

  • 9 indexing slots ( 6 in uses max)

  • 3 topics ( up to 20 k event / s in one)

Sometime (?) I see some events are rejected (tranquility log), I can see that no indexing task exists , but slots are available. This issue is transient but annoying.

Data node are not replicated.

Any idea where I can start to look at ?

Thx

Hi Oliver,
the most common cause for events getting rejected is that the events may be outside the windowPeriod.

In this case tuning the windowPeriod will help.

Other than this, I would start with looking into tranquility/Indexer/Task logs for any exceptions/errors.

Sorry for the late update…
In fact, the host (data node) was running out of resources (no more place on device…), when zipping the data, then peons could not start from time to time …

Now (with bigger disks) it works perfectly well.

The information of the druid console (available space) was false, at least it is not the disk space ( df -h told the truth)

Thx

Information shown in druid console is based on “druid.server.maxSize”.
Make sure it is set lower than total available disk space.