Javascript is disabled error in Hadoop Indexing job on EMR (0.10.0-rc2)

Hello all,
I’m trying to set-up an EMR cluster to index files using a javascript parser - and I’m getting a “JavaScript is disabled” error message in the Hadoop task logs.

I have javascript enabled in common.runtime.properties and when I run the task using the local mapred executor in the middle managers it works.

I think this has something to do with config files not being pushed to the Hadoop mapred tasks, and my reasoning is that the peon says:

2017-04-17T21:14:12,232 INFO [main] io.druid.cli.CliPeon - * druid.javascript.enabled: true

In the druid log, but the map tasks have:

2017-04-17T21:14:55,224 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.js.JavaScriptConfig] from props[druid.javascript.] as [JavaScriptConfig{enabled=false}]

I can’t find how to make sure the runtime properties are sent to the hadoop task.

I’m using 0.10.0-rc2 with EMR 2.7.3

Has anyone encountered this? Does anyone know how to work around this?

Thanks!

Gal

Hey Gal,

The runtime properties aren’t automatically passed down. (Maybe they should be, but they aren’t)

If you add “-Dhadoop.druid.javascript.enabled” to your druid.indexer.runner.javaOpts, does that help?

Hi Gian,

Thats what I thought, as I couldn’t find the properties in HDFS and no JAR was being sent.

I tried with “-Dhadoop.druid.javascript.enable” but that didn’t seem to have any effect (no new configs for the job in the history server)

I tried also with “-Dhadoop.druid.javascript.enabled=true” instead to the druid.indexer.runner.javaOpts - and “druid.javascript.enabled” showed up in the JobHistory server as “true” - but still no dice, same problem (“JavaScript is disabled”)


It looks like the partitioning job doesn’t pick up on the property.

2017-04-18T00:15:18,346 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.js.JavaScriptConfig] from props[druid.javascript.] as [JavaScriptConfig{enabled=false}]

Any ideas?

Thanks!

Gal

Ah, yeah it’s “enabled” not “enable”.

Hmm. How about adding -Ddruid.javascript.enabled=true to mapreduce.map.java.opts and mapreduce.reduce.java.opts? You could do that through your Druid hadoop task’s jobProperties. Something like the following, although you might need to adjust Xms/Xms for your environment.

“jobProperties” : {

“mapreduce.map.java.opts” : “-server -Xms512m -Xmx512m -Ddruid.javascript.enabled=true -Duser.timezone=UTC -Dfile.encoding=UTF-8”,

“mapreduce.reduce.java.opts” : “-server -Xms2g -Xmx2g -Ddruid.javascript.enabled=true -Duser.timezone=UTC -Dfile.encoding=UTF-8”

}

That works!

Added to the jobProperties, and now it works. I’ll let you know once it completes if that solves it (and then we can update GitHub and maybe talk about adding to documentation).

Thanks so much for the quick response!

(Btw, I might have had a typo - I think you wrote “enabled”.)

I think it’d make sense for Druid to pass that down automatically.

Finished, and it works. Thanks!

I agree.

Will update the git issue.