Wrong FS in batch hadoop index

Hi guys,

I met the following error when trying to run the wikipedia hadoop batch index tutorial. Can someone shed some light please?

2015-07-16T10:31:57,438 INFO [task-runner-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1437042548166_0002_r_000000_0, Status : FAILED

Error: java.lang.IllegalArgumentException: Wrong FS: file://hdfs:/172.31.23.106:8020/druidStorage/wikipedia/wikipedia/2013-08-31T00:00:00.000Z_2013-09-01T00:00:00.000Z/2015-07-16T10:30:35.726Z/0, expected: file:///

at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)

at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:513)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:499)

at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:594)

at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.serializeOutIndex(IndexGeneratorJob.java:470)

at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:446)

at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:292)

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

I use a local hadoop cluster as deep storage.

Overload is triggered as

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/*:$HADOOP_CONF io.druid.cli.Main server overlord &

Druid common config

Deep storage (local filesystem for examples - don’t use this in production)

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs://172.31.23.106:8020/druidStorage

Query Cache (we use a simple 10mb heap-based local cache on the broker)

druid.cache.type=local

druid.cache.sizeInBytes=10000000

Indexing service discovery

druid.selectors.indexing.serviceName=overlord

Task config

“ioConfig”: {
“type”: “hadoop”,
“inputSpec”: {
“type”: “static”,
“paths”: “/tmp/wikipedia_data.json”
},
“metadataUpdateSpec”: {
“type”: “db”,
“connectURI”: “jdbc:mysql://172.31.23.106:3306/druid”,
“user”: “druid”,
“password”: “diurd”,
“segmentTable”: “druid_segments”
},
“segmentOutputPath”: “/tmp/segments”
},

Thanks,

Ying

What is your classpath?

also did you put druid-hdfs-storage extension on the classpath?

Also, I would generally configure
druid.storage.storageDirectory=/druidStorage

and hdfs namenode address should come from hadoop configuration on the classpath. (but it does not matter for your current problem)

– Himanshu

Thanks for the reply Himanshu.

the classpath I use is -classpath config/_common:config/overlord:lib/*:$HADOOP_CONF.
I didn’t take care of the extension as I assumed it’s automatically load from maven. Am I right?

And I am trying to run Hadoop index task against a distributed Hadoop cluster (co-exist with the Druid cluster), should I set
druid.indexer.runner.type = remote in overloard config

Also, is middle manager a must have in this case.

Thanks,
Ying

Hi,

Yes, in theory, putting extensions in just the config should work. But, depending upon the hadoop distribution/version, we have noticed many problems with putting druid-hdfs-storage in the extension list and having it loaded dynamically. In general, current recommendation is to just put druid-hdfs-storage and other hadoop dependencies on the classpath directly.

From the stacktrace, it seems that “RawLocalFileSystem” was used where I would expect to see “DistributedFileSystem” which suggests some problem with the hdfs-storage setup.

also, please try and replace $HADOOP_CONF by the output of command “hadoop classpath”, if hadoop has been setup correctly on the machine then that command should give your everything necessary hadoop related.

– Himanshu

It works now! Thank you Himanshu!

I included more extensions and also removed hdfs prefix in druid.storage.storageDirectory. Not sure which one makes the effect though.

druid.extensions.coordinates=[“io.druid.extensions:druid-examples”,“io.druid.extensions:druid-kafka-eight”,“io.druid.extensions:mysql-metadata-storage”,“io.druid.extensions:druid-hdfs-storage:0.7.3”,“org.apache.hadoop:hadoop-client:2.7.1”]

Zookeeper

druid.zk.service.host=172.31.23.106

Metadata Storage (mysql)

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://172.31.23.106:3306/druid

druid.metadata.storage.connector.user=druid

druid.metadata.storage.connector.password=diurd

Deep storage (local filesystem for examples - don’t use this in production)

druid.storage.type=hdfs

druid.storage.storageDirectory=/druidStorage

Query Cache (we use a simple 10mb heap-based local cache on the broker)

druid.cache.type=local

druid.cache.sizeInBytes=10000000

Indexing service discovery

druid.selectors.indexing.serviceName=overlord

Monitoring (disabled for examples, if you enable SysMonitor, make sure to include sigar jar in your cp)

druid.monitoring.monitors=[“com.metamx.metrics.SysMonitor”,“com.metamx.metrics.JvmMonitor”]

Metrics logging (disabled for examples - change this to logging or http in production)

druid.emitter=noop

Sorry I am having a further problem that historical nodes can’t load segments. The segment file /druidStorage/wikipedia/wikipedia/20130831T000000.000Z_20130901T000000.000Z/2015-07-17T23_33_56.498Z/0/index.zip is indeed there in hdfs.

2015-07-17T23:51:44,017 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_2013-08-31T00:00:00.000Z_2013-09-01T00:00:00.000Z_2015-07-17T23:33:56.498Z], segment=DataSegment{size=9852, shardSpec=NoneShardSpec, metrics=[count, added, deleted, delta], dimensions=[continent, robot, country, city, newPage, unpatrolled, namespace, anonymous, language, page, region, user], version=‘2015-07-17T23:33:56.498Z’, loadSpec={type=hdfs, path=/druidStorage/wikipedia/wikipedia/20130831T000000.000Z_20130901T000000.000Z/2015-07-17T23_33_56.498Z/0/index.zip}, interval=2013-08-31T00:00:00.000Z/2013-09-01T00:00:00.000Z, dataSource=‘wikipedia’, binaryVersion=‘9’}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_2013-08-31T00:00:00.000Z_2013-09-01T00:00:00.000Z_2015-07-17T23:33:56.498Z]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.3.jar:0.7.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

Caused by: io.druid.segment.loading.SegmentLoadingException: Error loading [/druidStorage/wikipedia/wikipedia/20130831T000000.000Z_20130901T000000.000Z/2015-07-17T23_33_56.498Z/0/index.zip]

at io.druid.storage.hdfs.HdfsDataSegmentPuller.getSegmentFiles(HdfsDataSegmentPuller.java:280) ~[?:?]

at io.druid.storage.hdfs.HdfsLoadSpec.loadSegment(HdfsLoadSpec.java:59) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]

… 18 more

Caused by: java.io.IOException: No FileSystem for scheme: hdfs

at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304) ~[?:?]

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311) ~[?:?]

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) ~[?:?]

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) ~[?:?]

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) ~[?:?]

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) ~[?:?]

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) ~[?:?]

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) ~[?:?]

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) ~[?:?]

at io.druid.storage.hdfs.HdfsDataSegmentPuller.getSegmentFiles(HdfsDataSegmentPuller.java:177) ~[?:?]

at io.druid.storage.hdfs.HdfsLoadSpec.loadSegment(HdfsLoadSpec.java:59) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]

… 18 more

Problem solved after I include all hadoop jars in the classpath.
Thanks!