Getting "Failing over to rm" when ingesting data to HDFS on clustered environment

Hi,

We are trying to ingest data to HDFS on clustered environment, after 20 to 30 mins the task gets failed with below issue

2017-08-02T09:17:31,666 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider - Failing over to rm2

2017-08-02T09:17:59,495 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider - Failing over to rm1

we are using HDP version 2.4.2.0-258 and hadoop-client - 2.4.2.0-258

Any suggestions ?

Can someone please reply or provide any suggestions ?

Which Druid Version are you using ?

Can you add the logs from the Hadoop Job Tracker and the complete Log from middle manager ?

Hi Slim,

Thanks for the response.

PFB the updated hadoop and error details

Cluster details :

  1. Hadoop version - 2.7.1

  2. Jackson version - jackson-core-asl-1.9.13.jar, jackson-mapper-asl-1.9.13.jar

Druid details:

Version - 0.10.0 stable version

Tried following different combinations of hadoop client along with jackson jars but still getting the same error on druid:

  1. hadoop client - 2.3.0 along with jackson-core-asl-1.8.8.jar, jackson-mapper-asl-1.8.8.jar

  2. hadoop client - 2.3.0 along with jackson-core-asl-1.9.13.jar, jackson-mapper-asl-1.9.13.jar

  3. hadoop client - 2.7.3 along with jackson-core-asl-1.9.13.jar, jackson-mapper-asl-1.9.13.jar

Error Details :

2017-08-06T16:13:52,494 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job -  map 0% reduce 0%
2017-08-06T16:13:57,688 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1501939697200_0009_m_000000_0, Status : FAILED
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2017-08-06T16:14:02,739 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1501939697200_0009_m_000000_1, Status : FAILED
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2017-08-06T16:14:07,768 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1501939697200_0009_m_000000_2, Status : FAILED