Hadoop indexing task fail with java.lang.IncompatibleClassChangeError (sometimes)

Hi,

I’m running Druid 0.9.2-rc2 on Cloudera CDH 5.8.2 using the ORC extension. The first part of the indexing job consists of 23 map tasks, some of which fail with the following error:

ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : com.google.common.util.concurrent.ExecutionError: java.lang.IncompatibleClassChangeError: Implementing class
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4827)
	at com.google.inject.internal.FailableCache.get(FailableCache.java:48)
	at com.google.inject.internal.MembersInjectorStore.get(MembersInjectorStore.java:68)
	at com.google.inject.internal.InjectorImpl.getMembersInjector(InjectorImpl.java:993)
	at com.google.inject.internal.InjectorImpl.getMembersInjector(InjectorImpl.java:1000)
	at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:986)
	at io.druid.initialization.Initialization$ModuleList.addModule(Initialization.java:397)
	at io.druid.initialization.Initialization$ModuleList.addModules(Initialization.java:416)
	at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:320)
	at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:99)
	at io.druid.indexer.HadoopDruidIndexerMapper.setup(HadoopDruidIndexerMapper.java:48)
	at io.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.setup(DetermineHashedPartitionsJob.java:224)
	at io.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.run(DetermineHashedPartitionsJob.java:282)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.IncompatibleClassChangeError: Implementing class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
	at com.google.inject.internal.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:93)
	at com.google.inject.internal.MembersInjectorStore.access$000(MembersInjectorStore.java:37)
	at com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
	at com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:42)
	at com.google.inject.internal.FailableCache$1.load(FailableCache.java:37)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
	... 23 more

I tracked this down and found that it fails when adding the module 'JettyServerModule', more specifically as can be seen in the exception during the injectMembers call. I suspected it might be a node specific problem, but I'm 99% sure they are all set up the same and I'm seeing
failed map tasks on nodes that have successful tasks as well.

Ingestion spec:
{
  "type": "index_hadoop",
  "spec": {
    "ioConfig": {
      "type": "hadoop",
      "inputSpec": {
        "type": "static",
        "inputFormat": "org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat",
        "paths": "/user/hive/warehouse/druid.db/adops/year=2016/month=10/day=01"
      }
    },
    "dataSchema": {
      "dataSource": "AdOpsVolume",
      "parser": {
        "type": "orc",
        "parseSpec": {
          "format": "timeAndDims",
          "timestampSpec": {
            "column": "ts",
            "format": "y-M-d H:m:s"
          },
          "dimensionsSpec": {
            "dimensions": [
              "feed",
              "country",
              "siteuuid",
              "sitename",
              "siteisapp",
              "sitetags",
              "deals"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        },
        "typeString": "struct<ts:string,feed:string,country:string,siteuuid:string,sitename:string,siteisapp:boolean,sitetags:array<string>,deals:array<string>,requests:bigint>"
      },
      "metricsSpec": [{
          "type": "longSum",
          "name": "requests",
          "fieldName": "requests"
      }],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "none",
        "intervals": ["2016-10-01/2016-10-02"]
      }
    },
    "tuningConfig": {
      "type": "hadoop",
      "partitionsSpec": {
        "type": "hashed",
        "targetPartitionSize": 5000000
      },
      "jobProperties": {
        "mapreduce.job.user.classpath.first": "true",
        "mapreduce.map.memory.mb": "2048",
        "mapreduce.reduce.memory.mb": "4092",
        "mapreduce.map.java.opts": "-server -Duser.timezone=UTC -Dfile.encoding=UTF-8",
        "mapreduce.reduce.java.opts": "-server -Duser.timezone=UTC -Dfile.encoding=UTF-8"
      },
      "leaveIntermediate": true
    }
  },
  "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.6.0-cdh5.8.2"]
}

So at this point I'm lost for causes and would be grateful for every hint!

Thanks,
Alex

Hi Alex, do these docs help?

http://druid.io/docs/0.9.2-rc2/operations/other-hadoop.html

Hi,

I’ve read these and just today found the issue. The problem was caused by a dependency coming through the ORC extension. It pulled in a jetty-all-7.6.0.jar which was messing with the dependency injection. I removed it from the classpath and the indexing job then ran without any failed tasks. So if you’re using this extension, be aware that the dependencies from it might cause issues.

Thanks,

Alex