Ingestion was throttled on small dataset with KIS

Hi everyone,

I’m getting the following throttling warning from middlemanager logs while using KIS to ingest data from Kafka.

2019-03-28T15:05:43,747 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [86,730] millis because persists were pending.

This is weird. I have NFS set up as deep storage and I’ve measured its writing speed to be around 100MB/s. Hence it is weird to have warnings (and related hiccups to data access) as the actual segment size saved is much less than 1MB. One thread… where should I look to troubleshoot this?!

I don’t understand why this throttling would be needed… If I understand correctly that would mean, that writing the segments to disk took more time than the limit (15s?).

  1. Druid version and setup information

druid-incubating-v0.13

  1. Relevant logs on any Druid nodes, especially exceptions

2019-03-28T15:05:43,747 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [86,730] millis because persists were pending.

Other warning messages:

2019-03-28T15:04:19,142 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29

2019-03-28T15:04:21,723 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type.

2019-03-28T15:04:21,772 WARN [main] org.apache.druid.query.lookup.LookupReferencesManager - No lookups found for tier [__default], response [org.apache.druid.java.util.http.client.response.FullResponseHolder@bfe47a8]

  1. Configuration settings, including runtime.properties, system properties and JVM flags

I have multiple nodes and one instance of each druid role… middleManager shares location with historical and coordinator shares location with overlord. My metastore is Postgresql on another node… connection among nodes can handle 1Gb/s.

Thanks,

Sandor Kazi

Hey,

Did you ever figure this out? I’m having the exact same issue on the (nearly) exact same setup.

Thank you!

Kiefer