Failed to save new beam due to JsonMappingException

Hello,

We have storm topology which ingests data to the druid in real-time. When we submit topology to storm it works OK for a while and after that stops.

In log file we see following errors:

2016-03-02 17:56:33.463 c.m.t.b.ClusteredBeam [WARN] Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/ody_live_bi_event
{ }
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/ody_live_bi_event] timestamp[2016-03-02T17:40:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[stormjar.jar:?]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[stormjar.jar:?]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[stormjar.jar:?]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [stormjar.jar:?]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [stormjar.jar:?]
at com.twitter.util.Promise.runq(Promise.scala:350) [stormjar.jar:?]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [stormjar.jar:?]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [stormjar.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.ArrayIndexOutOfBoundsException) (through reference chain: com.fasterxml.jackson.module.scala.deser.MapBuilderWrapper[“beams”])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase.wrapAndThrow(ContainerDeserializerBase.java:88) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:507) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[stormjar.jar:?]
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[stormjar.jar:?]
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3702) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2807) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeamMeta$.fromBytes(ClusteredBeam.scala:467) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:153) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:149) ~[stormjar.jar:?]
at com.twitter.util.Try$.apply(Try.scala:13) ~[stormjar.jar:?]
… 6 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
2016-03-02 17:56:33.463 c.m.e.c.LoggingEmitter [INFO] Event [{“feed”:“alerts”,“timestamp”:“2016-03-02T17:56:33.463Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to create merged beam: druid:overlord/ody_live_bi_event”,“data”:{“exceptionType”:“java.lang.IllegalStateException”,“exceptionMessage”:“Failed to save new beam for identifier[druid:overlord/ody_live_bi_event] timestamp[2016-03-02T17:40:00.000Z]”,“exceptionStackTrace”:“java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/ody_live_bi_event] timestamp[2016-03-02T17:40:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.ArrayIndexOutOfBoundsException) (through reference chain: com.fasterxml.jackson.module.scala.deser.MapBuilderWrapper[“beams”])\n\tat com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210)\n\tat com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177)\n\tat com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase.wrapAndThrow(ContainerDeserializerBase.java:88)\n\tat com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:507)\n\tat com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341)\n\tat com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76)\n\tat com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39)\n\tat com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3702)\n\tat com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2807)\n\tat com.metamx.tranquility.beam.ClusteredBeamMeta$.fromBytes(ClusteredBeam.scala:467)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:153)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:149)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\t… 6 more\nCaused by: java.lang.ArrayIndexOutOfBoundsException\n”}}]
2016-03-02 17:56:33.463 c.m.t.t.Tranquilizer [WARN] Failed to send 3 messages.
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/ody_live_bi_event] timestamp[2016-03-02T17:40:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[stormjar.jar:?]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[stormjar.jar:?]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[stormjar.jar:?]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[stormjar.jar:?]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [stormjar.jar:?]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [stormjar.jar:?]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [stormjar.jar:?]
at com.twitter.util.Promise.runq(Promise.scala:350) [stormjar.jar:?]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [stormjar.jar:?]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [stormjar.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.ArrayIndexOutOfBoundsException) (through reference chain: com.fasterxml.jackson.module.scala.deser.MapBuilderWrapper[“beams”])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase.wrapAndThrow(ContainerDeserializerBase.java:88) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:507) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[stormjar.jar:?]
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[stormjar.jar:?]
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3702) ~[stormjar.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2807) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeamMeta$.fromBytes(ClusteredBeam.scala:467) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:153) ~[stormjar.jar:?]
at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:149) ~[stormjar.jar:?]
at com.twitter.util.Try$.apply(Try.scala:13) ~[stormjar.jar:?]
… 6 more
Caused by: java.lang.ArrayIndexOutOfBoundsException

We solved this issue by updating all Jackson libraries from version 2.6.0 to version 2.6.3