Druid Segment Load

Hi

After running a Hadoop ingestion task, I find my Druid cluster slows down greatly and has trouble communicating with the historical nodes

  1. The number of nodes will drop to 2 in the UI though I have 3 historical nodes setup
  2. I will see there are 11 to 20 segments to load even though my ingestion task was run one segment’s (one day’s) worth of data partitioned to create 2 shards on average
  3. Queries will return an empty array as a result despite returning data prior to ingestion

I was wondering if I could get any clues as to why this slowdown occurs. The machines have 380 GB of RAM and I think I have allocated ample memory for the nodes:

Here’s the relevant logs after the showing the index task and segment insertion finished successfully

2017-06-14T15:11:28,462 INFO [task-runner-0-priority-0] io.druid.indexer.IndexGeneratorJob - Adding segment CompositeRevenue_2017-03-25T00:00:00.000Z_2017-03-26T00:00:00.000Z_2017-06-14T15:00:56.179Z to the list of published segments
2017-06-14T15:11:28,462 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[/tmp/druid-indexing/CompositeRevenue/2017-06-14T150056.179Z_cf441206296042ed8d1f38f1b9baba68]
2017-06-14T15:11:28,500 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_hadoop_CompositeRevenue_2017-06-14T15:00:56.134Z]: SegmentInsertAction{segments=[DataSegment{size=86812245, shardSpec=NoneShardSpec, metrics=[Views, AdvertiserValue, Leads, VideoFirstQuartile, VideoMidpoint, VideoThirdQuartile, VideoComplete, VideoMute, VideoUnmute, VideoPause, VideoResume, VideoFullscreen, VideoSkip, VideoCompanionViews, VideoCompanionClicks, ClickConversion, ViewConversion, AcceptedViewConversion, ExternalValue1, ExternalValue2, ExternalValue3, ExternalValue4, ExternalValue5, Clicks, OriginalBid, Bid, LearningCost, SupplyAuctionPrice, DemandAuctionPrice, CostAfterDiscrepencies, LicenseeClearingCost, AcuityMargin, AcuityTechFee, AcuityThirdPartyFees, CostToLicensee, ClientClearingCost, LicenseeMargin, LicenseeTechFee, LicenseeThirdPartyFees, CostToClient], dimensions=[CountryRegionKey, CampaignID, AcuityVerticalID, AgeID, BannerID, ChildrenID, EthnicityID, ExchangeID, Frequency, GenderID, HouseholdIncomeID, HouseholdSizeID, OperatingSystemID, PageQualityID, PlacementID, PubID, RaceID, SegmentID, SiteID, InventoryID, UserLanguage, UserQualityID, VerticalID, WebBrowserID, DealID, CountryCode, RegionCode, City, DeviceType, Interstitial, DeviceID, PixelKey, ExternalReportingID, ExternalReportingTypeID], version='2017-06-14T15:00:56.179Z', loadSpec={type=hdfs, path=hdfs://nj-db21.acuityads.org:8020/user/druid/data/CompositeRevenue/20170325T000000.000Z_20170326T000000.000Z/2017-06-14T15_00_56.179Z/0/index.zip}, interval=2017-03-25T00:00:00.000Z/2017-03-26T00:00:00.000Z, dataSource='CompositeRevenue', binaryVersion='9'}]}
2017-06-14T15:11:28,505 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_hadoop_CompositeRevenue_2017-06-14T15:00:56.134Z] to overlord[http://nj-db19.acuityads.org:8090/druid/indexer/v1/action]: SegmentInsertAction{segments=[DataSegment{size=86812245, shardSpec=NoneShardSpec, metrics=[Views, AdvertiserValue, Leads, VideoFirstQuartile, VideoMidpoint, VideoThirdQuartile, VideoComplete, VideoMute, VideoUnmute, VideoPause, VideoResume, VideoFullscreen, VideoSkip, VideoCompanionViews, VideoCompanionClicks, ClickConversion, ViewConversion, AcceptedViewConversion, ExternalValue1, ExternalValue2, ExternalValue3, ExternalValue4, ExternalValue5, Clicks, OriginalBid, Bid, LearningCost, SupplyAuctionPrice, DemandAuctionPrice, CostAfterDiscrepencies, LicenseeClearingCost, AcuityMargin, AcuityTechFee, AcuityThirdPartyFees, CostToLicensee, ClientClearingCost, LicenseeMargin, LicenseeTechFee, LicenseeThirdPartyFees, CostToClient], dimensions=[CountryRegionKey, CampaignID, AcuityVerticalID, AgeID, BannerID, ChildrenID, EthnicityID, ExchangeID, Frequency, GenderID, HouseholdIncomeID, HouseholdSizeID, OperatingSystemID, PageQualityID, PlacementID, PubID, RaceID, SegmentID, SiteID, InventoryID, UserLanguage, UserQualityID, VerticalID, WebBrowserID, DealID, CountryCode, RegionCode, City, DeviceType, Interstitial, DeviceID, PixelKey, ExternalReportingID, ExternalReportingTypeID], version='2017-06-14T15:00:56.179Z', loadSpec={type=hdfs, path=hdfs://nj-db21.acuityads.org:8020/user/druid/data/CompositeRevenue/20170325T000000.000Z_20170326T000000.000Z/2017-06-14T15_00_56.179Z/0/index.zip}, interval=2017-03-25T00:00:00.000Z/2017-03-26T00:00:00.000Z, dataSource='CompositeRevenue', binaryVersion='9'}]}
2017-06-14T15:11:28,506 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://nj-db19.acuityads.org:8090
2017-06-14T15:11:28,571 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_CompositeRevenue_2017-06-14T15:00:56.134Z] status changed to [SUCCESS].
2017-06-14T15:11:28,573 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_hadoop_CompositeRevenue_2017-06-14T15:00:56.134Z",
  "status" : "SUCCESS",
  "duration" : 628412
}