Exception with Hadoop Batch Ingestion (AWS Lambda)

Hi,
We are facing following exception while running haddop batch ingestion task through lambda :

Where could be the issue? Is this issue while connecting to s3 or while reading files from s3 ?

2018-03-16 08:12:05.252+0000 INFO CAMP [task-runner-0-priority-0] io.druid.indexer.path.GranularityPathSpec Appending path

2018-03-16 08:12:05.299+0000 DEBUG CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Cluster Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider

2018-03-16 08:12:05.299+0000 DEBUG CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Cluster Cannot pick org.apache.hadoop.mapred.LocalClientProtocolProvider as the ClientProtocolProvider - returned null protocol

2018-03-16 08:12:05.299+0000 DEBUG CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Cluster Trying ClientProtocolProvider : org.apache.hadoop.mapred.YarnClientProtocolProvider

2018-03-16 08:12:05.338+0000 INFO CAMP [task-runner-0-priority-0] org.apache.hadoop.yarn.client.RMProxy Connecting to ResourceManager at ip-10-51-13-50.us-west-2.compute.internal/10.51.13.50:8032

2018-03-16 08:12:05.475+0000 DEBUG CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Cluster Picked org.apache.hadoop.mapred.YarnClientProtocolProvider as the ClientProtocolProvider

2018-03-16 08:12:05.524+0000 WARN CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

2018-03-16 08:12:05.540+0000 WARN CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader No job jar file set. User classes may not be found. See Job or Job#setJar(String).

2018-03-16 08:12:05.798+0000 INFO CAMP [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter Cleaning up the staging area /tmp/hadoop-yarn/staging/druid/.staging/job_1515060571688_0034

2018-03-16 08:12:05.811+0000 ERROR CAMP [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner Exception while running task[HadoopIndexTask{id=index_hadoop_eventstream_dsds_2018-03-16T08:11:55.777Z, type=index_hadoop, dataSource=eventstream_dsds}]

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_92]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_92]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_92]

at java.lang.Thread.run(Unknown Source) [?:1.8.0_92]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_92]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

… 7 more

Caused by: java.lang.RuntimeException: java.io.IOException: No input paths specified in job

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:207) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexer.JobHelper.runJobs(JobHelper.java:323) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_92]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

… 7 more

Caused by: java.io.IOException: No input paths specified in job

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:239) ~[?:?]

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387) ~[?:?]

at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) ~[?:?]

at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) ~[?:?]

at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) ~[?:?]

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_92]

at javax.security.auth.Subject.doAs(Unknown Source) ~[?:1.8.0_92]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) ~[?:?]

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]

at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:116) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexer.JobHelper.runJobs(JobHelper.java:323) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_92]

at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_92]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

… 7 more

2018-03-16 08:12:05.824+0000 INFO CAMP [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils Task [index_hadoop_eventstream_dsds_2018-03-16T08:11:55.777Z] status changed to [FAILED].

2018-03-16 08:12:05.826+0000 INFO CAMP [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle Task completed with status: {

“id” : “index_hadoop_eventstream_dsds_2018-03-16T08:11:55.777Z”,

“status” : “FAILED”,

“duration” : 6763

}

2018-03-16 08:12:05.832+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@5df6163a].

2018-03-16 08:12:05.833+0000 INFO CAMP [main] io.druid.server.coordination.AbstractDataSegmentAnnouncer Stopping class io.druid.server.coordination.BatchDataSegmentAnnouncer with config[io.druid.server.initialization.ZkPathsConfig@22e2266d]

2018-03-16 08:12:05.833+0000 INFO CAMP [main] io.druid.curator.announcement.Announcer unannouncing [/druid/announcements/10.51.13.12:8100]

2018-03-16 08:12:05.845+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@443cdaa4].

2018-03-16 08:12:05.846+0000 INFO CAMP [main] io.druid.curator.announcement.Announcer unannouncing [/druid/listeners/lookups/__default/10.51.13.12:8100]

2018-03-16 08:12:05.848+0000 INFO CAMP [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer Unannouncing start time on [/druid/listeners/lookups/__default/10.51.13.12:8100]

2018-03-16 08:12:05.848+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@1cd6b1bd].

2018-03-16 08:12:05.848+0000 INFO CAMP [main] io.druid.query.lookup.LookupReferencesManager Stopping lookup factory references manager

2018-03-16 08:12:05.851+0000 INFO CAMP [main] org.eclipse.jetty.server.ServerConnector Stopped ServerConnector@7ed49a7f{HTTP/1.1}{0.0.0.0:8100}

2018-03-16 08:12:05.853+0000 INFO CAMP [main] org.eclipse.jetty.server.handler.ContextHandler Stopped o.e.j.s.ServletContextHandler@4001d8c1{/,null,UNAVAILABLE}

2018-03-16 08:12:05.855+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.server.log.FileRequestLogger.stop()] on object[io.druid.server.log.FileRequestLogger@4ef2ab73].

2018-03-16 08:12:05.855+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@32bb0072].

2018-03-16 08:12:05.856+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@39832280].

2018-03-16 08:12:05.857+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@4997552e].

2018-03-16 08:12:05.859+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@13e5d243].

2018-03-16 08:12:05.860+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@3ba0ae41].

2018-03-16 08:12:05.860+0000 INFO CAMP [main] io.druid.curator.CuratorModule Stopping Curator

2018-03-16 08:12:05.860+0000 INFO CAMP [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting

2018-03-16 08:12:05.863+0000 INFO CAMP [main] org.apache.zookeeper.ZooKeeper Session: 0x3622d66aa3f000e closed

2018-03-16 08:12:05.863+0000 INFO CAMP [main-EventThread] org.apache.zookeeper.ClientCnxn EventThread shut down for session: 0x3622d66aa3f000e

2018-03-16 08:12:05.863+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@609e57da].

2018-03-16 08:12:05.879+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.server.namespace.NamespacedExtractionModule$NamespaceStaticConfiguration.stop()] on object[io.druid.server.namespace.NamespacedExtractionModule$NamespaceStaticConfiguration@33a55bd8].

2018-03-16 08:12:05.880+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@539a138b].

2018-03-16 08:12:05.880+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@12b5736c].

2018-03-16 08:12:05.880+0000 INFO CAMP [main] com.metamx.emitter.core.LoggingEmitter Close: started [false]

2018-03-16 08:12:05.880+0000 INFO CAMP [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@2f382a5e].

Heap

garbage-first heap total 960512K, used 422399K [0x0000000640000000, 0x0000000640101d50, 0x00000007c0000000)

region size 1024K, 409 young (418816K), 31 survivors (31744K)

Metaspace used 76053K, capacity 76978K, committed 77276K, reserved 1116160K

class space used 9727K, capacity 9971K, committed 9980K, reserved 1048576K

Thanks,

Pravesh Gupta