Hadoop Reindexing task getting stucked


Our re-indexing task has been stuck on this line :

2016-09-26T11:01:13,946 INFO [GetFileInfo #0] com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase - GHFS version: 1.4.4-hadoop2

Nothing is happening in the log. Even the job is not submitted to hadoop . I am attaching the task log file. Druid Version :

Even no errors has been seen on any nodes.

What can be the issue.



loghadoop.txt.zip (32.8 KB)

We have figure out where the issue is occuring, But don’t know how to fix it. Our re-indexing job is getting stucked while fetching the splits from google storage. Dont know much about GCS file format :

**File : **DatasourceInputFormat.java in indexinghadoop directory.

Package : io.druid.indexer.hadoop

Job go Stucked on this line org.apache.hadoop.mapred.InputSplit split : fio.getSplits(conf, 1)

Iterable locations = Collections.emptyList();

for (WindowedDataSegment segment : segments) {

FileInputFormat.setInputPaths(conf, new Path(JobHelper.getURIFromSegment(segment.getSegment())));
for (org.apache.hadoop.mapred.InputSplit split : fio.getSplits(conf, 1)) {
locations = Iterables.concat(locations, Arrays.asList(split.getLocations()));

I don’t see anything in getSplits that could keep it
looping forever: https://hadoop.apache.org/docs/r2.7.1/api/src-html/org/apache/hadoop/mapred/FileInputFormat.html#line.312

line before uses JobHelper.getURIFromSegment which was that last bug I fixed: https://github.com/erikdubbelboer/druid/blob/google-extensions/indexing-hadoop/src/main/java/io/druid/indexer/JobHelper.java#L739

Anyone else have any suggestions on what could be happening? At the moment I don’t see how this could be related to the google-extension.

I ran into the same issue when reindexing segments now as well. I just pushed a fix here: https://github.com/druid-io/druid/pull/3788