Druid S3 Extension in Cluster mode setup on Ec2s

Hi all,

I’m trying to setup Druid cluster mode on Ec2 instances. I have Instance profiles attached to these instances and and while loading s3 extension, getting an error in coordinator-overload logs.

Can someone give some pointers around S3 and Kinesis config ?
should we just include the s3 bucket name or arn of the bucket ?

ERROR: 2020-03-02T21:30:56,086 ERROR [main] org.apache.druid.cli.CliCoordinator - Error when starting up. Failing.

com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error in custom provider, org.apache.druid.java.util.common.RE: server initialization exception

at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:164) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.initialization.jetty.JettyServerModule)

at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:164) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.initialization.jetty.JettyServerModule)

while locating org.eclipse.jetty.server.Server

1 error

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]

at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:141) ~[druid-core-0.17.0.jar:0.17.0]

at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) [druid-services-0.17.0.jar:0.17.0]

at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:59) [druid-services-0.17.0.jar:0.17.0]

at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.17.0.jar:0.17.0]

Caused by: org.apache.druid.java.util.common.RE: server initialization exception

at org.apache.druid.server.initialization.jetty.JettyServerModule.makeAndInitializeServer(JettyServerModule.java:397) ~[druid-server-0.17.0.jar:0.17.0]

at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:164) ~[druid-server-0.17.0.jar:0.17.0]

at org.apache.druid.server.initialization.jetty.JettyServerModule$$FastClassByGuice$$ff9924d8.invoke() ~[druid-server-0.17.0.jar:0.17.0]

at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]

… 5 more

Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Unknown provider[s3] of Key[type=org.apache.druid.tasklogs.TaskLogs, annotation=[none]], known options[[noop, file]]

at org.apache.druid.guice.PolyBind.createChoice(PolyBind.java:71) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.IndexingServiceTaskLogsModule)

while locating org.apache.druid.tasklogs.TaskLogs

while locating org.apache.druid.tasklogs.TaskLogKiller

for the 1st parameter of org.apache.druid.indexing.overlord.helpers.TaskLogAutoCleaner.<init>(TaskLogAutoCleaner.java:47)

while locating org.apache.druid.indexing.overlord.helpers.TaskLogAutoCleaner

while locating org.apache.druid.indexing.overlord.helpers.OverlordHelper annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=107, type=MULTIBINDER, keyType=)

at org.apache.druid.cli.CliOverlord$1.configureOverlordHelpers(CliOverlord.java:340) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliOverlord$1 -> com.google.inject.multibindings.Multibinder$RealMultibinder)

while locating java.util.Set<org.apache.druid.indexing.overlord.helpers.OverlordHelper>

for the 2nd parameter of org.apache.druid.indexing.overlord.helpers.OverlordHelperManager.<init>(OverlordHelperManager.java:48)

while locating org.apache.druid.indexing.overlord.helpers.OverlordHelperManager

for the 12th parameter of org.apache.druid.indexing.overlord.TaskMaster.<init>(TaskMaster.java:92)

at org.apache.druid.cli.CliOverlord$1.configure(CliOverlord.java:177) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliOverlord$1)

while locating org.apache.druid.indexing.overlord.TaskMaster

for the 1st parameter of org.apache.druid.cli.CoordinatorOverlordRedirectInfo.<init>(CoordinatorOverlordRedirectInfo.java:40)

while locating org.apache.druid.cli.CoordinatorOverlordRedirectInfo

at org.apache.druid.cli.CliCoordinator$1.configure(CliCoordinator.java:168) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliCoordinator$1)

while locating org.apache.druid.server.http.RedirectInfo

for the 1st parameter of org.apache.druid.server.http.RedirectFilter.<init>(RedirectFilter.java:48)

at org.apache.druid.cli.CliCoordinator$1.configure(CliCoordinator.java:166) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliCoordinator$1)

while locating org.apache.druid.server.http.RedirectFilter

1 error

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) ~[guice-4.1.0.jar:?]

at org.apache.druid.cli.CoordinatorJettyServerInitializer.initialize(CoordinatorJettyServerInitializer.java:117) ~[druid-services-0.17.0.jar:0.17.0]

at org.apache.druid.server.initialization.jetty.JettyServerModule.makeAndInitializeServer(JettyServerModule.java:394) ~[druid-server-0.17.0.jar:0.17.0]

at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:164) ~[druid-server-0.17.0.jar:0.17.0]

at org.apache.druid.server.initialization.jetty.JettyServerModule$$FastClassByGuice$$ff9924d8.invoke() ~[druid-server-0.17.0.jar:0.17.0]

at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]

… 5 more

Thanks,

Satya

Hi, could you provide your ingestion spec and the configuration file of your nodes ?
Especially, did you add the druid-s3-extensions extension in your druid.extensions.loadList parameter ?

For more insights, you can take a look, if not already done, at this page : https://druid.apache.org/docs/latest/development/extensions-core/s3.html#configuration

Guillaume

Thanks for the reply.
here is the configuration file for my nodes
common.runtime.properties:

First of all, you have this line set
druid.extensions.loadList=["mysql-metadata-storage

This means every other extension is not loaded, as s3 extension.

As stated in the comment some line above, you should either include all the extensions you need or don’t specify any (in this case, druid will load all extensions present in /extensions directory.

I don’t remember what is present by default in this case.

To be clearer, you should specify what extension you want to load.

Thank you. That was it. I explicitly mentioned the extensions I want to load and it works now. Appreciate the help.