I am seeing the following error once in a while in the middleManager logs,
2017-01-26T00:57:56,336 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [14,948] millis because persists were pending.
Usually it appears in bursts.
Any ideas on why and when does this happen, and there are config changes that need to be looked at.
Running Druid 0.9.2
Let me know what config settings are relevant so I can post them to get a better idea of the setup.
This means that because it took more time to write to disk than expected, ingestion was paused and wasn’t reading the stream for 15 seconds. If this is only intermittent, it probably means you had a load spike that the system took 15 seconds to recover from. If you want to avoid this then you could shard out ingestion more, or use faster disks.
Thanks for the quick reply.
If I have to go the sharding route, would it mean lowering the segmentGranularity? Currently, I have it set to FIFTEEN_MINUTE.
I also lowered maxRowsInMemory in the meanwhile. Does maxRowsPerSegment also affect the ingestion/writing to disk throughput?
maxRowsInMemory and segmentGranularity affect how often data is written to disk but don’t have a big effect on how much is written overall. One other thing you can do is set “buildV9Directly” : true in your tuningConfig if you haven’t already. That usually helps indexing performance and can reduce this persist throttling. But if you do that, and still keep seeing this throttling, then more tasks or better disks should help.
This is late, but wanted to let you know that things did improve making changes you recommended and also other tweaks like taskDuration, number of workers and task counts. A combination of all these made good improvements.
I have a few more questions, but I will post another thread for them.