I am using tranquility for realtime ingestion.
I am facing following issues:
Many a times it happens that my ingestion tasks for nth hour are scheduled at nth hour and mth minute ( as bad as scheduling tasks for 10th hour at 10:40) as a result of which I lose events for first m minutes.
Also some tasks are created in the 3rd to 5th minute of the next hour and task for the next hour is not created as a result of which all my events are dropped.
For example my 18th hour task is scheduled at 19:03 and it ingests till 19:10(10 minutes window) and drops all subsequent events and does not create any new task for the 19th and further hours because of which there is no ingestion
for a few hours.
Can someone explain me the reasons behind this…
Is this because of some resource issues( number of worker nodes)??