java.io.IOException: Filesystem WriteOperationHelper {bucket=<bucketname>} closed

Hi Team,

I configured the HDP platform with Druid and am trying to setup S3 as deepStore.

have tried to ingest the data using hadoop batch ingester. in the logs I can see that the map task is at 100% and reduce is at 100% but one of the reduce task keeps running which can be seen in the container log and show an exception like below. Breaking my head since 5 days. Appreciate anyones help here.

2019-03-21T20:34:27,857 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.common.config.NullValueHandlingConfig] from props[druid.generic.] as [io.druid.common.config.NullValueHandlingConfig@146dcfe6]
2019-03-21T20:34:27,873 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2019-03-21T20:34:28,004 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=null}]
2019-03-21T20:34:28,007 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.emitter.ambari.metrics.AmbariMetricsEmitterModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,009 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,009 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,010 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,026 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.s3.S3StorageDruidModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,027 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.firehose.s3.S3FirehoseDruidModule] for class [interface io.druid.initialization.DruidModule] from classpath extension
2019-03-21T20:34:28,043 WARN [main] io.druid.initialization.Log4jShutterDownerModule - Shutdown callback registry expected class [io.druid.common.config.Log4jShutdown] found [org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry]. Skipping shutdown registry
2019-03-21T20:34:28,548 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.security.Escalator] from props[druid.escalator.] as [io.druid.server.security.NoopEscalator@413bef78]
2019-03-21T20:34:28,556 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@1235151c]
2019-03-21T20:34:28,561 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@3332c7a5]
2019-03-21T20:34:28,611 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
2019-03-21T20:34:28,613 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2019-03-21T20:34:28,641 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@58d3f4be]
2019-03-21T20:34:28,649 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}]
2019-03-21T20:34:28,683 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@1c30cb85]
2019-03-21T20:34:28,687 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
2019-03-21T20:34:28,687 INFO [main] io.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: io.druid.java.util.emitter.core.NoopEmitter@19b07407
2019-03-21T20:34:28,687 INFO [main] io.druid.server.emitter.EmitterModule - Extra service dimensions: {version=0.12.1.3.1.0.0-78}
2019-03-21T20:34:28,688 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@30aec673]
2019-03-21T20:34:28,693 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.segment.writeout.SegmentWriteOutMediumFactory] from props[druid.peon.defaultSegmentWriteOutMediumFactory.] as [io.druid.segment.writeout.TmpFileSegmentWriteOutMediumFactory@1d3c112a]
2019-03-21T20:34:28,695 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()]
2019-03-21T20:34:28,697 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.size, ${base_path}.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]
2019-03-21T20:34:28,698 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()]
2019-03-21T20:34:28,775 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexer.HadoopKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.indexer.HadoopKerberosConfig@0]
2019-03-21T20:34:28,792 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.common.aws.AWSCredentialsConfig] from props[druid.s3.] as [io.druid.common.aws.AWSCredentialsConfig@48cb2d73]
2019-03-21T20:34:28,840 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.s3.S3DataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.s3.S3DataSegmentPusherConfig@7c7e73c5]
2019-03-21T20:34:28,841 INFO [main] io.druid.storage.s3.S3DataSegmentPusher - Configured S3 as deep storage
2019-03-21T20:34:30,766 INFO [main] io.druid.indexer.IndexGeneratorJob - 34,371 lines completed.
2019-03-21T20:34:30,771 INFO [main] io.druid.segment.IndexMergerV9 - Starting persist for interval[2018-05-10T00:00:00.000Z/2018-05-10T01:00:00.000Z], rows[34,371]
2019-03-21T20:34:30,864 INFO [main] io.druid.segment.IndexMergerV9 - Using SegmentWriteOutMediumFactory[TmpFileSegmentWriteOutMediumFactory]
2019-03-21T20:34:30,871 INFO [main] io.druid.segment.IndexMergerV9 - Completed version.bin in 5 millis.
2019-03-21T20:34:30,878 INFO [main] io.druid.segment.IndexMergerV9 - Completed factory.json in 7 millis
2019-03-21T20:34:30,909 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[ReportType] conversions with cardinality[1] in 27 millis.
2019-03-21T20:34:30,927 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DataSource] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:30,928 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[current_date] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:30,930 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandPartner] conversions with cardinality[26] in 2 millis.
2019-03-21T20:34:30,931 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandTagID] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:30,980 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandTag] conversions with cardinality[2,310] in 49 millis.
2019-03-21T20:34:30,983 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Country] conversions with cardinality[109] in 1 millis.
2019-03-21T20:34:30,983 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[TagIDs] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:30,984 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[StickerReferralID] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:30,985 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[ListID] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:31,002 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[SupplySourceID] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,004 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[SupplySource] conversions with cardinality[63] in 0 millis.
2019-03-21T20:34:31,004 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[PlayerWidthxHeight] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,005 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Devicecategory] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:31,005 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Key] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,006 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[values] conversions with cardinality[22] in 0 millis.
2019-03-21T20:34:31,007 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Transactiontypes] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:31,007 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DealIDs] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,008 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Dealnames] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,009 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Agency] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,009 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Brand] conversions with cardinality[1] in 0 millis.
2019-03-21T20:34:31,010 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DSP] conversions with cardinality[1] in 1 millis.
2019-03-21T20:34:31,010 INFO [main] io.druid.segment.IndexMergerV9 - Completed dim conversions in 129 millis.
2019-03-21T20:34:31,513 INFO [main] io.druid.segment.IndexMergerV9 - completed walk through of 34,371 rows in 483 millis.
2019-03-21T20:34:31,529 INFO [main] io.druid.segment.IndexMergerV9 - Completed time column in 16 millis.
2019-03-21T20:34:31,530 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[AdExchangeActiveViewmeasurableimpressions] in 0 millis.
2019-03-21T20:34:31,530 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[AdExchangeclicks] in 0 millis.
2019-03-21T20:34:31,530 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[AdExchangeimpressions] in 0 millis.
2019-03-21T20:34:31,533 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[AdExchangerevenue] in 2 millis.
2019-03-21T20:34:31,533 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[AdRequests] in 0 millis.
2019-03-21T20:34:31,533 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[BidRate] in 0 millis.
2019-03-21T20:34:31,534 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Bidranges] in 1 millis.
2019-03-21T20:34:31,534 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Bids] in 0 millis.
2019-03-21T20:34:31,535 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Clicks] in 0 millis.
2019-03-21T20:34:31,535 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Cost] in 0 millis.
2019-03-21T20:34:31,535 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Entries] in 0 millis.
2019-03-21T20:34:31,536 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Estimatedearningsfrombids] in 1 millis.
2019-03-21T20:34:31,538 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Impressions] in 2 millis.
2019-03-21T20:34:31,539 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Impressionswon] in 0 millis.
2019-03-21T20:34:31,540 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Inventorysizes] in 1 millis.
2019-03-21T20:34:31,540 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[MatchedRequests] in 0 millis.
2019-03-21T20:34:31,541 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Opportunities] in 1 millis.
2019-03-21T20:34:31,541 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[PageLands] in 0 millis.
2019-03-21T20:34:31,542 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[PageViews] in 1 millis.
2019-03-21T20:34:31,544 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Revenue] in 2 millis.
2019-03-21T20:34:31,544 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[TagRequests] in 0 millis.
2019-03-21T20:34:31,545 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[VASTAds] in 1 millis.
2019-03-21T20:34:31,545 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[VPAIDAds] in 0 millis.
2019-03-21T20:34:31,546 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[VPAIDAttempts] in 0 millis.
2019-03-21T20:34:31,546 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[VPAIDResponses] in 0 millis.
2019-03-21T20:34:31,546 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[ViewabilityMeasuredImpressions] in 0 millis.
2019-03-21T20:34:31,547 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[ViewableImpressions] in 1 millis.
2019-03-21T20:34:31,547 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Visits] in 0 millis.
2019-03-21T20:34:31,547 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Voters] in 0 millis.
2019-03-21T20:34:31,548 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Votes] in 1 millis.
2019-03-21T20:34:31,548 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[Wins] in 0 millis.
2019-03-21T20:34:31,548 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric column[count] in 0 millis.
2019-03-21T20:34:31,548 INFO [main] io.druid.segment.IndexMergerV9 - Completed metric columns in 19 millis.
2019-03-21T20:34:31,563 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[ReportType] inverted with cardinality[1] in 14 millis.
2019-03-21T20:34:31,574 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DataSource] inverted with cardinality[1] in 7 millis.
2019-03-21T20:34:31,578 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[current_date] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,583 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandPartner] inverted with cardinality[26] in 4 millis.
2019-03-21T20:34:31,587 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandTagID] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,661 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DemandTag] inverted with cardinality[2,310] in 33 millis.
2019-03-21T20:34:31,670 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Country] inverted with cardinality[109] in 8 millis.
2019-03-21T20:34:31,679 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[TagIDs] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,682 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[StickerReferralID] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,686 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[ListID] inverted with cardinality[1] in 2 millis.
2019-03-21T20:34:31,689 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[SupplySourceID] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,699 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[SupplySource] inverted with cardinality[63] in 9 millis.
2019-03-21T20:34:31,713 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[PlayerWidthxHeight] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,717 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Devicecategory] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,720 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Key] inverted with cardinality[1] in 2 millis.
2019-03-21T20:34:31,728 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[values] inverted with cardinality[22] in 7 millis.
2019-03-21T20:34:31,749 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Transactiontypes] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,752 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DealIDs] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,756 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Dealnames] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,759 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Agency] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,762 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[Brand] inverted with cardinality[1] in 2 millis.
2019-03-21T20:34:31,766 INFO [main] io.druid.segment.StringDimensionMergerV9 - Completed dim[DSP] inverted with cardinality[1] in 3 millis.
2019-03-21T20:34:31,770 INFO [main] io.druid.segment.IndexMergerV9 - Completed index.drd in 4 millis.
2019-03-21T20:34:31,782 INFO [main] io.druid.java.util.common.io.smoosh.FileSmoosher - Created smoosh file [/hadoop/yarn/local/usercache/druid/appcache/application_1553193839475_0006/container_e01_1553193839475_0006_01_000003/tmp/base836525711497550462flush/merged/00000.smoosh] of size [420419] bytes.
2019-03-21T20:34:32,798 INFO [main] io.druid.indexer.JobHelper - Creating new ZipEntry[version.bin]
2019-03-21T20:34:32,799 INFO [main] io.druid.indexer.JobHelper - Creating new ZipEntry[meta.smoosh]
2019-03-21T20:34:32,799 INFO [main] io.druid.indexer.JobHelper - Creating new ZipEntry[factory.json]
2019-03-21T20:34:32,810 INFO [main] io.druid.indexer.JobHelper - Creating new ZipEntry[00000.smoosh]
2019-03-21T20:34:33,076 INFO [main] io.druid.indexer.JobHelper - Zipped 421,974 bytes to [s3a://druiddeepstorev1/Druid_BaseKey/Hadoop_Demo/2018-05-10T00:00:00.000Z_2018-05-10T01:00:00.000Z/2019-03-21T20:33:58.217Z/0/index.zip.0]
2019-03-21T20:34:33,182 INFO [main] io.druid.indexer.JobHelper - Attempting rename from [s3a://druiddeepstorev1/Druid_BaseKey/Hadoop_Demo/2018-05-10T00:00:00.000Z_2018-05-10T01:00:00.000Z/2019-03-21T20:33:58.217Z/0/index.zip.0] to [s3a://druiddeepstorev1/Druid_BaseKey/Hadoop_Demo/2018-05-10T00:00:00.000Z_2018-05-10T01:00:00.000Z/2019-03-21T20:33:58.217Z/0/index.zip]
2019-03-21T20:34:33,557 INFO [main] io.druid.indexer.JobHelper - Exception in descriptor pusher retry loop
java.io.IOException: Filesystem WriteOperationHelper {bucket=druiddeepstorev1} closed
	at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.checkOpen(S3ABlockOutputStream.java:229) ~[hadoop-aws-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.flush(S3ABlockOutputStream.java:241) ~[hadoop-aws-3.1.1.3.1.0.0-78.jar:?]
	at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[?:1.8.0_112]
	at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[?:1.8.0_112]
	at io.druid.indexer.JobHelper$5.push(JobHelper.java:487) [druid-indexing-hadoop-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at com.sun.proxy.$Proxy91.push(Unknown Source) [?:?]
	at io.druid.indexer.JobHelper.writeSegmentDescriptor(JobHelper.java:499) [druid-indexing-hadoop-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at io.druid.indexer.JobHelper.serializeOutIndex(JobHelper.java:450) [druid-indexing-hadoop-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:751) [druid-indexing-hadoop-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:501) [druid-indexing-hadoop-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) [hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628) [hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) [hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) [hadoop-mapreduce-client-app-3.1.1.3.1.0.0-78.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
	at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [hadoop-common-3.1.1.3.1.0.0-78.jar:?]
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) [hadoop-mapreduce-client-app-3.1.1.3.1.0.0-78.jar:?]
2019-03-21T20:34:35,744 INFO [main] io.druid.indexer.JobHelper - Exception in descriptor pusher retry loop

Regards,
Chethan G Puttaswamy

Are your Hadoop jars matching up with HDP and Druid?

The hadoop-aws-3.1.1.3.1.0.0-78.jar is a different packaging and will differ from what HDP has and what’s in Druid Hadoop dependencies.

Rommel Garcia