[issue] Loading segments from S3 slow - seems to happen sequentially

Our 15 node cluster is loading segments from S3 onto the historicals very slowly, even though we have opened all valves:

  • numLoadingThreads is set to 25
  • maxSegmentsToMove 200
  • replicationThrottleLimit we tried 300 and -1
  • load rate is at 2.5GB per minute per node
  • pulling of a single segment-shard (200MB) takes roughly 8 seconds.
  • unzipping of a single segment-shard takes roughly 6 seconds.

We see in the coordinator console that plenty of segments are slated for loading (sometimes 1000 segments and terabytes of data) but actually loading them is slow.

We investigated the problem and found out that there are indeed 25 threads loading segments per historical node BUT these threads seem to execute sequentially as can be seen from the attached log-file.

We don’t know why this is the case but looking at the source code, it seems that the jets3t S3 rest client is used as a singleton and itself uses a non-thread-safe apache commons http client, so maybe the code itself is not setup for concurrent operations. Just digging in the dark here though.

Did anyone have a similar problem? Anything we could try to find out what’s happening?

thanks
Sascha

historical3.log (11.2 KB)

As you can see, the 25 threads load segments sequentially in an instance with 32 CPU cores:

check your metrics coming off the node. I suspect you’re going to see about 78% cpu accounted for by IOWait.

Hi Charles,

We don’t see iowait at all on these historical nodes! They are r3.8xlarge and i2.8xlarge EC2 instances which use their local, ephemeral, SSD-based disks to store the data. So getting iowait while writing less than 50MB/s would be surprising anyway. The absence of iowait (and other noteable resource utilization) is what led us to dig further into this issue and to notice the mentioned sequential behavior. The problem is also present when the node isn’t queried at all, so we were able to eliminate queries in parallel to loading segments as reason as well.

Best regards,

Daniel

Thanks Daniel,

Can you file a ticket about this? I think numloadingthreads is handled wrong in ZKCoordinator

Hi Charles,

We now created an issue for that: https://github.com/druid-io/druid/issues/3202

Best regards,

Daniel

Hi, regarding segment loading performance, should this druid.segmentCache.announceIntervalMillisbe set to zero when adding a new historical node? For the segments to be loaded quicker?