Hadoop Batch Ingestion from dataSource, mapReduce job failed

Hi all,

There are two same datasources in our system for different use. When one lost data, we wanted to retrieve data from other datasource.

Thus we use “Updating Existing Data”, that is reindexing with Hadoop Batch Ingestion.

Problems are:

  1. The task could find the segments in HDFS, but it FAILED. There were exceptions in mapReduce job and Kafka producer. How to solve it?

The main logs are:

2016-11-22T15:29:06,608 INFO [task-runner-0-priority-0] io.druid.indexer.path.DatasourcePathSpec - Found total [20] segments for [datasource_006] in interval [[2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z]]

2016-11-22T15:29:06,614 INFO [task-runner-0-priority-0] io.druid.indexer.HadoopDruidIndexerJob - No metadataStorageUpdaterJob set in the config. This is cool if you are running a hadoop index task, otherwise nothing will be uploaded to database.

2016-11-22T15:29:06,638 INFO [task-runner-0-priority-0] io.druid.indexer.path.DatasourcePathSpec - Found total [20] segments for [datasourcedatasource_006] in interval [[2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z]]

2016-11-22T15:29:07,045 WARN [task-runner-0-priority-0] org.apache.hadoop.hdfs.BlockReaderLocal - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

2016-11-22T15:29:07,084 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

2016-11-22T15:29:07,100 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set. User classes may not be found. See Job or Job#setJar(String).

2016-11-22T15:29:07,230 INFO [task-runner-0-priority-0] io.druid.indexer.hadoop.DatasourceInputFormat - segments to read [[{“segment”:{“dataSource”:“datasource_006”,“interval”:“2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z”,“version”:“2016-11-14T04:00:00.069Z”,“loadSpec”:{“type”:“hdfs”,“path”:"/user/monitor/segments/datasource_006/20161114T120000.000Z_20161114T130000.000Z/2016-11-14T04_00_00.069Z/0/index.zip"},“dimensions”:“db,table”,“metrics”:“count,bonus_sum”,“shardSpec”:{“type”:“numbered”,“partitionNum”:0,“partitions”:0},“binaryVersion”:9,“size”:2939210,“identifier”:“datasource_006_2016-11-14T12:00:00.000Z_2016-11-14T13:00:00.000Z_2016-11-14T04:00:00.069Z”},“interval”:“2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z”},……]]

2016-11-22T15:29:07,253 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,294 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,295 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,297 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,298 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,300 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,301 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,302 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,303 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,305 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,306 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,307 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,308 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,310 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,311 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,312 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,314 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,315 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,316 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,318 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.FileInputFormat - Total input paths to process : 1

2016-11-22T15:29:07,318 INFO [task-runner-0-priority-0] io.druid.indexer.hadoop.DatasourceInputFormat - Number of splits [20]

2016-11-22T15:29:07,451 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter - number of splits:20

2016-11-22T15:29:07,514 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1479461229117_0036

2016-11-22T15:29:07,619 INFO [task-runner-0-priority-0] org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.

2016-11-22T15:29:07,828 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1479461229117_0036

2016-11-22T15:29:07,856 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - The url to track the job: http://bigdata:8088/proxy/application_1479461229117_0036/

2016-11-22T15:29:07,856 INFO [task-runner-0-priority-0] io.druid.indexer.IndexGeneratorJob - Job datasource_001-index-generator-Optional.of([2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z]) submitted, status available at http://bigdata:8088/proxy/application_1479461229117_0036/

2016-11-22T15:29:07,857 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Running job: job_1479461229117_0036

2016-11-22T15:29:12,875 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_1479461229117_0036 running in uber mode : false

2016-11-22T15:29:12,876 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - map 0% reduce 0%

2016-11-22T15:29:12,886 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_1479461229117_0036 failed with state FAILED due to: Application application_1479461229117_0036 failed 2 times due to AM Container for appattempt_1479461229117_0036_000002 exited with exitCode: 1

For more detailed output, check application tracking page:http://bigdata:8088/cluster/app/application_1479461229117_0036Then, click on links to logs of each attempt.

Diagnostics: Exception from container-launch.

Container id: container_e1397_1479461229117_0036_02_000001

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)

at org.apache.hadoop.util.Shell.run(Shell.java:456)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)

at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:370)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Shell output: main : command provided 1

main : user is hadoop

main : requested yarn user is root

Container exited with a non-zero exit code 1

Failing this attempt. Failing the application.

2016-11-22T15:29:12,917 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 0

2016-11-22T15:29:12,939 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[/tmp/druid-indexing/datasource_001/2016-11-22T072859.983Z_e4da7831933b4050b759a2fd59a10e28]

2016-11-22T15:29:12,956 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_datasource_001_2016-11-22T07:28:59.982Z, type=index_hadoop, dataSource=datasource_001}]

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:208) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

… 7 more

Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.IndexGeneratorJob] failed!

at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:94) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:261) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

… 7 more

2016-11-22T15:29:12,962 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_datasource_001_2016-11-22T07:28:59.982Z] status changed to [FAILED].

2016-11-22T15:29:12,964 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “index_hadoop_datasource_001_2016-11-22T07:28:59.982Z”,

“status” : “FAILED”,

“duration” : 9213

}

  1. Why batch ingestion need Kafka ingestion?

The main logs are:

2016-11-22T15:29:06,096 INFO [task-runner-0-priority-0] org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values:

compression.type = none

metric.reporters =

metadata.max.age.ms = 300000

metadata.fetch.timeout.ms = 60000

reconnect.backoff.ms = 50

sasl.kerberos.ticket.renew.window.factor = 0.8

bootstrap.servers = [bigdata:9092]

retry.backoff.ms = 100

sasl.kerberos.kinit.cmd = /usr/bin/kinit

buffer.memory = 33554432

timeout.ms = 30000

key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

sasl.kerberos.service.name = null

sasl.kerberos.ticket.renew.jitter = 0.05

ssl.keystore.type = JKS

ssl.trustmanager.algorithm = PKIX

block.on.buffer.full = false

ssl.key.password = null

max.block.ms = 60000

sasl.kerberos.min.time.before.relogin = 60000

connections.max.idle.ms = 540000

ssl.truststore.password = null

max.in.flight.requests.per.connection = 5

metrics.num.samples = 2

client.id =

ssl.endpoint.identification.algorithm = null

ssl.protocol = TLS

request.timeout.ms = 30000

ssl.provider = null

ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

acks = 1

batch.size = 0

ssl.keystore.location = null

receive.buffer.bytes = 32768

ssl.cipher.suites = null

ssl.truststore.type = JKS

security.protocol = PLAINTEXT

retries = 0

max.request.size = 1048576

value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

ssl.truststore.location = null

ssl.keystore.password = null

ssl.keymanager.algorithm = SunX509

metrics.sample.window.ms = 30000

partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner

send.buffer.bytes = 131072

linger.ms = 0

2016-11-22T15:29:06,099 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.9.0.1

2016-11-22T15:29:06,099 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 23c69d62a0cabf06

2016-11-22T15:29:06,099 WARN [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Error registering AppInfo mbean

javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=producer-2

at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) ~[?:1.8.0_65]

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) ~[?:1.8.0_65]

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) ~[?:1.8.0_65]

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) ~[?:1.8.0_65]

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) ~[?:1.8.0_65]

at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[?:1.8.0_65]

at org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57) [kafka-clients-0.9.0.1.jar:?]

at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:314) [kafka-clients-0.9.0.1.jar:?]

at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:181) [kafka-clients-0.9.0.1.jar:?]

at org.apache.logging.log4j.core.appender.mom.kafka.DefaultKafkaProducerFactory.newKafkaProducer(DefaultKafkaProducerFactory.java:29) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.appender.mom.kafka.KafkaManager.startup(KafkaManager.java:86) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.appender.mom.kafka.KafkaAppender.start(KafkaAppender.java:98) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:225) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:492) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:562) [log4j-core-2.5.jar:2.5]

at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578) [log4j-core-2.5.jar:2.5]

at org.apache.log4j.LogManager.reconfigure(LogManager.java:100) [log4j-1.2-api-2.5.jar:2.5]

at org.apache.log4j.BasicConfigurator.configure(BasicConfigurator.java:28) [log4j-1.2-api-2.5.jar:2.5]

at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:50) [sigar-1.6.5.132.jar:?]

at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44) [sigar-1.6.5.132.jar:?]

at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60) [sigar-1.6.5.132.jar:?]

at org.hyperic.sigar.Sigar.(Sigar.java:108) [sigar-1.6.5.132.jar:?]

at com.metamx.metrics.SigarUtil.getSigar(SigarUtil.java:63) [server-metrics-0.2.8.jar:?]

at com.metamx.metrics.SysMonitor.(SysMonitor.java:49) [server-metrics-0.2.8.jar:?]

at io.druid.server.metrics.MetricsModule.getSysMonitor(MetricsModule.java:138) [druid-server-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.server.metrics.MetricsModule$$FastClassByGuice$$9b6589cd.invoke() [druid-server-0.9.2-rc2.jar:0.9.2-rc2]

at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) [druid-api-0.9.2-rc2.jar:0.9.2-rc2]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) [guice-4.1.0.jar:?]

at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:93) [druid-server-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.server.metrics.MetricsModule$$FastClassByGuice$$9b6589cd.invoke() [druid-server-0.9.2-rc2.jar:0.9.2-rc2]

at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) [druid-api-0.9.2-rc2.jar:0.9.2-rc2]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]

at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) [guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) [guice-4.1.0.jar:?]

at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) [guice-4.1.0.jar:?]

at com.google.inject.Guice.createInjector(Guice.java:99) [guice-4.1.0.jar:?]

at com.google.inject.Guice.createInjector(Guice.java:73) [guice-4.1.0.jar:?]

at com.google.inject.Guice.createInjector(Guice.java:62) [guice-4.1.0.jar:?]

at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:366) [druid-server-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexer.HadoopDruidIndexerConfig.(HadoopDruidIndexerConfig.java:99) [druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:277) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

2016-11-22T15:29:06,106 INFO [KafkaManager-CloseThread] org.apache.kafka.clients.producer.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

2016-11-22T15:29:06,119 INFO [task-runner-0-priority-0] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.SysMonitor@4e55d154]

2016-11-22T15:29:06,119 INFO [task-runner-0-priority-0] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@20c9fda4]

  1. If the data in HDFS is import, that is to say, the path is like “/user/root/import/datasource_006/20161111T200000.000Z_20161111T210000.000Z”.

The reindexing task would not find the segments. But if the path uses default path, like “/user/root/segments/datasource_006/20161114T200000.000Z_20161114T210000.000Z”, the reindexing task could find them.

Related logs are:

2016-11-22T15:21:28,735 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_datasource_001_2016-11-22T07:21:22.231Z, type=index_hadoop, dataSource=datasource_001}]

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

… 7 more

Caused by: com.metamx.common.ISE: No segments found for ingestion spec [DatasourceIngestionSpec{dataSource=‘datasource_006’, intervals=[2016-11-12T00:00:00.000Z/2016-11-12T01:00:00.000Z], segments=null, filter=null, granularity=DurationGranularity{length=3600000, origin=0}, dimensions=null, metrics=null, ignoreWhenNoSegments=false}]

at io.druid.indexer.path.DatasourcePathSpec.addInputPaths(DatasourcePathSpec.java:102) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexer.HadoopDruidIndexerConfig.addInputPaths(HadoopDruidIndexerConfig.java:389) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexer.JobHelper.ensurePaths(JobHelper.java:337) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:55) ~[druid-indexing-hadoop-0.9.2-rc2.jar:0.9.2-rc2]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]

… 7 more

2016-11-22T15:21:28,743 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_datasource_001_2016-11-22T07:21:22.231Z] status changed to [FAILED].

The task spec is as follows.

{

“type” : “index_hadoop”,

“spec” :{

“dataSchema”: {

“dataSource”: “datasource_001”,

“granularitySpec”: {

“queryGranularity”: “NONE”,

“segmentGranularity”: “HOUR”,

“type”: “uniform”,

“intervals”: [“2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z”]

},

“metricsSpec”: [

{

“name”: “count”,

“type”: “count”

},

{

“fieldName”: “bouns”,

“name”: “bonus_sum”,

“type”: “longSum”

}

],

“parser”: {

“parseSpec”: {

“dimensionsSpec”: {

“dimensionExclusions”: [

],

“dimensions”: [

“db”,

“table”

],

“spatialDimensions”: [

]

},

“format”: “json”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

}

},

“type”: “hadoopyString”

}

},

“ioConfig”: {

“type”: “hadoop”,

“inputSpec”: {

“type”: “dataSource”,

“ingestionSpec”: {

“dataSource”: “datasource_006”,

“intervals”: [“2016-11-14T12:00:00.000Z/2016-11-14T13:00:00.000Z”],

“granularity”: “HOUR”

},

“paths”:“hdfs://user/root/import/datasource_006/20161114T120000.000Z_20161114T130000.000Z”

}

},

“tuningConfig”: {

“type”: “hadoop”,

“jobProperties”: {

“mapreduce.jobtracker.staging.root.dir”: “/user/root/hadoop”,

“yarn.app.mapreduce.am.staging-dir”: “/user/root/hadoop/hadoop-yarn/staging”,

“mapreduce.map.memory.mb”: “2048”,

“mapreduce.reduce.memory.mb”: “2048”,

“mapreduce.job.user.name”: “root”,

“mapreduce.job.user.classpath.first”: “true”

}

}

},

“hadoopDependencyCoordinates”: [“org.apache.hadoop:hadoop-client:2.7.2”]

}

Thanks a lot,

Xinxin

After try everything in http://druid.io/docs/0.9.2-rc2/operations/other-hadoop.html, the problem was still there. Exitcode was either 1 or 143.
Finally we changed <hadoop.compile.version>2.3.0</hadoop.compile.version> to 2.7.2 in the pom.xml, and recompiled druid.

And in the task spec, we add the mapreduce.job.classloader = true .

Both deep storage and hadoop index task use hadoop2.7.2, and it works fine.

在 2016年11月22日星期二 UTC+8下午4:17:59,Xinxin Yi写道: