ClusteredBeam: Failed to update cluster state

All / Gian,

Sorry — I’m following up again because we still see this issue popup on occasion. (Tranquility side)

A JVM restart fixes things, but I’d like to get to the bottom of the cause and/or figure out how to catch the exception and handle this programmatically. (w/o restart)

Any thoughts/insights are appreciated.

2016-05-10T05:55:30.983865+00:00 my_hosts: { }

2016-05-10T05:55:30.983869+00:00 my_hosts: com.fasterxml.jackson.databind.JsonMappingException: 128 (through reference chain: com.fasterxml.jackson.module.scala.deser.MapBuilderWrapper[“beams”])

2016-05-10T05:55:30.983873+00:00 my_hosts: #011at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983877+00:00 my_hosts: #011at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983880+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase.wrapAndThrow(ContainerDeserializerBase.java:88) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983884+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:507) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983887+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983890+00:00 my_hosts: #011at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[jackson-module-scala_2.11-2.6.0.jar:?]

2016-05-10T05:55:30.983894+00:00 my_hosts: #011at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[jackson-module-scala_2.11-2.6.0.jar:?]

2016-05-10T05:55:30.983898+00:00 my_hosts: #011at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3702) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983956+00:00 my_hosts: #011at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2807) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.983963+00:00 my_hosts: #011at com.metamx.tranquility.beam.ClusteredBeamMeta$.fromBytes(ClusteredBeam.scala:467) ~[tranquility-core_2.11-0.7.4.jar:0.7.4]

2016-05-10T05:55:30.983970+00:00 my_hosts: #011at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:153) [tranquility-core_2.11-0.7.4.jar:0.7.4]

2016-05-10T05:55:30.983974+00:00 my_hosts: #011at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:149) [tranquility-core_2.11-0.7.4.jar:0.7.4]

2016-05-10T05:55:30.983982+00:00 my_hosts: #011at com.twitter.util.Try$.apply(Try.scala:13) [util-core_2.11-6.30.0.jar:6.30.0]

2016-05-10T05:55:30.983986+00:00 my_hosts: #011at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [util-core_2.11-6.30.0.jar:6.30.0]

2016-05-10T05:55:30.983994+00:00 my_hosts: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72]

2016-05-10T05:55:30.984002+00:00 my_hosts: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72]

2016-05-10T05:55:30.984007+00:00 my_hosts: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]

2016-05-10T05:55:30.984015+00:00 my_hosts: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]

2016-05-10T05:55:30.984034+00:00 my_hosts: #011at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]

2016-05-10T05:55:30.984089+00:00 my_hosts: Caused by: java.lang.ArrayIndexOutOfBoundsException: 128

2016-05-10T05:55:30.984099+00:00 my_hosts: #011at com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer.addName(ByteQuadsCanonicalizer.java:870) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984107+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2340) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984112+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:2224) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984119+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1831) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984123+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1786) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984131+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1743) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984138+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1678) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984143+00:00 my_hosts: #011at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1007) ~[jackson-core-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984151+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:483) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984158+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984191+00:00 my_hosts: #011at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[jackson-module-scala_2.11-2.6.0.jar:?]

2016-05-10T05:55:30.984196+00:00 my_hosts: #011at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[jackson-module-scala_2.11-2.6.0.jar:?]

2016-05-10T05:55:30.984199+00:00 my_hosts: #011at com.fasterxml.jackson.module.scala.deser.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializerModule.scala:44) ~[jackson-module-scala_2.11-2.6.0.jar:?]

2016-05-10T05:55:30.984222+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:222) ~[jackson-databind-2.6.0.jar:2.6.0]

2016-05-10T05:55:30.984228+00:00 my_hosts: #011at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:495) ~[jackson-databind-2.6.0.jar:2.6.0]

thanks in advance!

-brian

Hey Brian,

That looks like a Jackson bug, fixed in 2.6.2: https://github.com/FasterXML/jackson-core/issues/216

I’m not sure if it exists in 2.4.6 (the version Tranquillity is bundled with) – at least, nobody else has reported it in the wild. We have been refraining from upgrading for a while due to issues with running inside Spark Streaming. But if you aren’t constrained by that, try updating your Jackson to 2.6.2 and see if that helps.

Great, thanks Gian.

We are running within Spark Streaming (1.6.1). (but using tranquility-core directly)

We’ll give 2.4.6 a try.

If stable, we can guinea pig 2.6.2, and let you know how it works out.

-brian

Jackson 2.6.2 is good on its own, the only issue is that when we run inside Spark (or any framework in general), and that framework does not provide classloader isolation, we need to use a version that is “close” to what that framework uses or else there will be runtime compatibility problems. If you can run 2.6.0 (like your stack trace indicates) without Spark complaining then you should be able to run 2.6.2, so I would give that a shot.

Yep, understood. Thanks again.

Ironically, it looks like we are pulling in 2.6.0 via tranquility 0.7.4’s dependency on scala-util…

Our grade dependency tree…

tranquility-core_2.11:0.7.4

±-- com.metamx:scala-util_2.11:1.11.6

±-- com.fasterxml.jackson.core:jackson-core:2.6.0

The 2.6.0 dependency is still there in the most recent version of the util…

https://github.com/metamx/scala-util/blob/v1.12.2/build.sbt

I wonder if we can get scala-util upgraded? :wink:

-brian

You should be able to override the dependency in your own pom/sbt/gradle file and things should work, since code built against 2.6.0 should be able to run with 2.6.2.

Understood, that is what we did.

I was just trying to think of ways we could make this less likely to happen. (since it results in an unrecoverable error)

With a transitive dependency on 2.6.0 in the tree (via scale-util), that could likely get pulled in by build systems.

Maybe an explicit exclude of jackson on the scala-util dependency might clean it up?

-brian

Sent this PR to scala-util. https://github.com/metamx/scala-util/pull/21

Btw, I’m not sure how you ended up getting Jackson 2.6.0. Tranquility depends on 2.4.6 (see the pom: https://repo1.maven.org/maven2/io/druid/tranquility-core_2.11/0.8.0/tranquility-core_2.11-0.8.0.pom) and that should override the 2.6.0 dependency in scala-util, at least with maven. Maybe gradle resolves dependencies differently somehow?

Thanks (again). And yes…

From what I’ve seen, gradle resolves dependencies differently. It will override an explicit/root version dependency declaration, if a newer version exists in the dependency tree.

(quoting the docs below…)

Gradle offers the following conflict resolution strategies:
• Newest: The newest version of the dependency is used. This is Gradle’s default strategy, and is often an appropriate choice as long as versions are backwards-compatible.

https://docs.gradle.org/current/userguide/dependency_management.html

If you need to override that behavior, you do so by declaring a dependency with “force=true”.

https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.dsl.DependencyHandler.html

This is the exact scenario that caught us, we explicitly declared 2.4.6 as a root dependency, but that was overridden by 2.6.0 because gradle takes newest… confirming w/ output of gradle:

initially…

±-- com.fasterxml.jackson.core:jackson-core:2.4.6 -> 2.6.0

then, with force=true:

±-- com.fasterxml.jackson.core:jackson-core:2.4.6

(makes you miss maven =)

-brian

Ha, “as long as versions are backwards-compatible”. It would be nice if we lived in that world :slight_smile: