batch indexing failing due to dependency issues!!

Hi,

I am trying to do batch indexing for my data. I started with the example. I have tried druid with realtime node setup and it works. Now I am trying to do batch indexing of data. I already ran pull-deps and pushed the files to the machine. The example given is the start guide is not running for me . It is trying to reach to remote maven repository for some decencies and my machine is not accessible to internet.
I am attaching the logs from the overlord node.
I see all the dependencies in the extensions-repo folder and all other nodes(realtime,historical, etc run fine)
I get the same error regardless of the type of task I am trying to run.

Did anyone got the similar error before?

Thanks
A

overlordtaskFail.txt (77.2 KB)

Hello,

In the Druid standard distribution there is a directory called extensions-repo full of .jar files. Is possible that, depending on which directory are you when you start the commands, the Druid processes try to create a new externsions-repo directory there and start downloading the dependencies from the Internet to fill it. You can specify in the settings where Druid should go to find the extensions-repo directory. Fixing an absolute path may help you.

Absolute path also does not help. The tasks submitted fail with the dependency error. Following is the error. My extensions-repo has this jar file. What am I missing!

Downloading artifact commons-codec:commons-codec:pom:1.4 from  (https://repo1.maven.org/maven2/, releases+snapshots)
Downloading: https://repo1.maven.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.pom

java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.Socket.connect(Socket.java:579)
        at com.squareup.okhttp.Connection.connect(Connection.java:100)
        at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:287)
        at com.squareup.okhttp.internal.http.HttpEngine.sendSocketRequest(HttpEngine.java:248)
        at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:197)
        at com.squareup.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:388)
        at com.squareup.okhttp.internal.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:339)
        at com.squareup.okhttp.internal.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:540)
        at com.squareup.okhttp.internal.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:148)
        at io.tesla.aether.okhttp.OkHttpAetherClient$ResponseAdapter.getStatusCode(OkHttpAetherClient.java:196)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.resumableGet(AetherRepositoryConnector.java:549)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:391)
        at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:232)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:320)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342)
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)
        at io.druid.initialization.Initialization.getClassLoaderForCoordinates(Initialization.java:253)
        at io.druid.indexing.common.task.HadoopTask.buildClassLoader(HadoopTask.java:96)
        at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:164)
        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:221)
        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:200)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Downloaded artifact commons-codec:commons-codec:pom:1.4 from  (https://repo1.maven.org/maven2/, releases+snapshots)
Downloading artifact commons-codec:commons-codec:pom:1.4 from  (https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local, releases+snapshots)
Downloading: https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local/commons-codec/commons-codec/1.4/commons-codec-1.4.pom

``

Any help on this please. I need to make it work.

From the documentation I thought its the version issue so I included the coordinates in the index spec. But it still gives me error. I am not sure why commons codec is giving the error.
“hadoopDependencyCoordinates”: [“org.apache.hadoop:hadoop-client:2.7.1”],

``

the entire file is same as examples/indexing/wikipedia_data.json exept the above line.

For starting the cluster I am giving the hadoop config path while running the overlord. Please point me in some direction.

I think your common configuration is incorrect. Make sure to set things to load extensions locally as this error says you are trying to d/l extensions remotely.

Honestly, you should probably try http://imply.io/docs/latest/quickstart for a much faster and better getting started experience.

I checked in this matter more. I made the internet available to see where it goes. It downloaded the dependencies that it needed from different maven repos, but at the end the task failed. In the logs I see the following exception:
Any idea what it is about

  • Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@42677203].
    2016-03-11T19:40:39,269 INFO [main] io.druid.cli.CliPeon - Finished peon task
    2016-03-11T19:40:39,276 INFO [Thread-54] io.druid.cli.CliPeon - Running shutdown hook
    2016-03-11T19:40:39,277 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@326f1dd8].
    2016-03-11T19:40:39,277 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop()] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@3bf6d2d9].
    2016-03-11T19:40:39,277 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@72fdad38].
    2016-03-11T19:40:39,278 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.client.ServerInventoryView.stop() throws java.io.IOException] on object[io.druid.client.BatchServerInventoryView@43d8842b].
    2016-03-11T19:40:39,279 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@210b4858].
    2016-03-11T19:40:39,279 INFO [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@68d970a5].
    2016-03-11T19:40:39,280 ERROR [Thread-54] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Exception when stopping method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@68d970a5]
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]
    at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:337) [java-util-0.27.4.jar:?]
    at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:261) [java-util-0.27.4.jar:?]
    at io.druid.cli.CliPeon$2.run(CliPeon.java:220) [druid-services-0.8.2.jar:0.8.2]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
    Caused by: java.lang.IllegalStateException: Already closed or has not been started
    at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[guava-16.0.1.jar:?]
    at org.apache.curator.x.discovery.details.ServiceCacheImpl.close(ServiceCacheImpl.java:91) ~[curator-x-discovery-2.8.0.jar:?]
    at org.apache.curator.x.discovery.details.ServiceProviderImpl.close(ServiceProviderImpl.java:78) ~[curator-x-discovery-2.8.0.jar:?]
    at io.druid.curator.discovery.ServerDiscoverySelector.stop(ServerDiscoverySelector.java:98) ~[druid-server-0.8.2.jar:0.8.2]
    … 8 more

``

Hi Fangjin,
I already setup the druid cluster with realtime streaming and it is working fine as of now with some 177 million rows. Can imply take over all of this setup easily otherwise it will be again more work to setup imply environment.
I need to know what will be best before we go into production. After that it will not be a viable option to resolve errors in this way.

Thnaks
Amey

Amey, IAP is designed to be interchanged with Druid and vice versa. In any case, your problems are because your dependencies are probably not set up correctly. In IAP or Druid 0.9.0, core extensions are packaged together with the distro so there’s no need to d/l them from a remote location.