Pull-Deps returning 404 and 501 errors

We are migrating our Druid cluster to Version 0.22.1 and when we try to get the dependency jars, we are seeing 404 and 501 errors.

I tried the suggested solution on this github issue: Pull-deps extremely slow and not working in docker build · Issue #12082 · apache/druid · GitHub, but that does not solve the problem, we are seeing more 404 and 501 issues with jitpack.io

Our updated command is :

    java -Dlog4j2.level=DEBUG \
      -cp "lib/*" \
      -Ddruid.extensions.directory="extensions" \
      -Ddruid.extensions.hadoopDependenciesDir="hadoop-dependencies" \
      -DignoreSnapshots=true \
      org.apache.druid.cli.Main tools pull-deps \
      --defaultVersion ${DRUID_VERSION} \
      --no-default-remote-repositories \
      -r https://repo.maven.apache.org/maven2/ \
      --no-default-hadoop \
      -c "org.apache.druid.extensions:druid-parquet-extensions:${DRUID_VERSION}"

I am pasting just a snippet of the logs:

Downloading: https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/org/apache/apache/21/apache-21.pom
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/org/apache/apache/21/apache-21.pom. Error code 404
        at io.tesla.aether.connector.AetherRepositoryConnector.handleResponseCode(AetherRepositoryConnector.java:298)
        at io.tesla.aether.connector.AetherRepositoryConnector.access$900(AetherRepositoryConnector.java:85)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.resumableGet(AetherRepositoryConnector.java:549)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:391)
        at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:232)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
        at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:122)
        at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:817)
        at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:669)
        at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:307)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:361)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:212)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342)
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:407)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:365)
        at org.apache.druid.cli.PullDependencies.run(PullDependencies.java:311)
        at org.apache.druid.cli.Main.main(Main.java:113)
Downloading: http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom. Error code 501, HTTPS Required
        at io.tesla.aether.connector.AetherRepositoryConnector.handleResponseCode(AetherRepositoryConnector.java:306)
        at io.tesla.aether.connector.AetherRepositoryConnector.access$900(AetherRepositoryConnector.java:85)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.resumableGet(AetherRepositoryConnector.java:549)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:391)
        at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:232)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
        at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:122)
        at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:817)
        at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:669)
        at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:307)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:361)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342)
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:407)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:365)
        at org.apache.druid.cli.PullDependencies.run(PullDependencies.java:311)
        at org.apache.druid.cli.Main.main(Main.java:113)
Downloading: https://jitpack.io/org/apache/apache/16/apache-16.pom
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://jitpack.io/org/apache/apache/16/apache-16.pom. Error code 404
        at io.tesla.aether.connector.AetherRepositoryConnector.handleResponseCode(AetherRepositoryConnector.java:298)
        at io.tesla.aether.connector.AetherRepositoryConnector.access$900(AetherRepositoryConnector.java:85)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.resumableGet(AetherRepositoryConnector.java:549)
        at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:391)
        at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:232)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
        at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
        at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:122)
        at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:817)
        at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:669)
        at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:307)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:361)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
        at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
        at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342)
        at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:407)
        at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:365)
        at org.apache.druid.cli.PullDependencies.run(PullDependencies.java:311)
        at org.apache.druid.cli.Main.main(Main.java:113)

anyone else seeing this issue, any suggestions on how to go about fixing this ?

Relates to Apache Druid 0.22.1

Maybe an unhelpful reply (!) as I don’t know this error… are you using pull-deps as from the latest, or from your older version?

@petermarshallio I am using “pull-deps” from the latest