Druid ingestion task failing randomly

Hi,

We are using druid 0.9.2.And while running batch ingestion task getting below exception

2017-08-17T13:29:20,084 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1501261194059_188051_m_000797_0, Status : FAILED

Error: java.lang.RuntimeException: native lz4 library not available

at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)

at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)

at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)

at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1609)

at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)

at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

Because of this tasks are failing randomly.

Not able to understand what can be the reason for the same.

Hi,

Does anyone have any idea about the same.

How are you installing druid?

We have cloned the git repo and build it for hadoop 2.7 version and then using the same for our use.
Let me know if this answers your question ?

Anuj, I have seen this error I think before and it usually means you need to ensure you are pointing at the proper library, or your ingestion task is setup properly. Can I see your ingestion spec?

Hi,
Attached is ingestion Spec.

I think all nodes on Hadoop cluster doesn’t have lz4 lib. thats why on those node MapRed is failing and druid doesn’t have these native lib in its hadoop dependency.

ingestionspec.json (2.52 KB)

Hi,

I turned of compressing in intermediate state and now we don’t have any exception.

But I think this is not the solution.