Unable to use SQL Sever as a meta data storage

Hi

I am trying to use an existing SQL server as a metadata store, I installed the extension using pull-deps and also added sql server jdbc driver, but still druid is not detecting the extension, I have mentioned the error logs below.

2020-06-18T08:28:01,464 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final

2020-06-18T08:28:02,055 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [sqlserver-metadata-storage], jars: sqlserver-metadata-storage-0.12.3.jar, mssql-jdbc-8.2.2.jre8.jar

2020-06-18T08:28:02,063 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-hdfs-storage], jars: hadoop-mapreduce-client-core-2.8.5.jar, hadoop-yarn-api-2.8.5.jar, jackson-dataformat-cbor-2.10.2.jar, aws-java-sdk-core-1.11.199.jar, commons-configuration-1.6.jar, commons-logging-1.1.1.jar, jackson-core-2.10.2.jar, apacheds-i18n-2.0.0-M15.jar, httpcore-4.4.11.jar, hadoop-common-2.8.5.jar, jetty-sslengine-6.1.26.jar, druid-hdfs-storage-0.18.1.jar, hadoop-client-2.8.5.jar, curator-framework-4.3.0.jar, htrace-core4-4.0.1-incubating.jar, ion-java-1.0.2.jar, commons-digester-1.8.jar, jcip-annotations-1.0-1.jar, xmlenc-0.52.jar, hadoop-mapreduce-client-app-2.8.5.jar, json-smart-2.3.jar, hadoop-auth-2.8.5.jar, asm-7.1.jar, jackson-core-asl-1.9.13.jar, commons-codec-1.13.jar, hadoop-aws-2.8.5.jar, jsp-api-2.1.jar, hadoop-yarn-client-2.8.5.jar, api-util-1.0.3.jar, commons-collections-3.2.2.jar, api-asn1-api-1.0.0-M20.jar, apacheds-kerberos-codec-2.0.0-M15.jar, hadoop-yarn-server-common-2.8.5.jar, aws-java-sdk-kms-1.11.199.jar, hadoop-annotations-2.8.5.jar, hadoop-mapreduce-client-jobclient-2.8.5.jar, hadoop-hdfs-client-2.8.5.jar, accessors-smart-1.2.jar, httpclient-4.5.10.jar, gson-2.2.4.jar, aws-java-sdk-s3-1.11.199.jar, curator-recipes-4.3.0.jar, leveldbjni-all-1.8.jar, commons-net-3.6.jar, jackson-databind-2.10.2.jar, jackson-annotations-2.10.2.jar, jackson-mapper-asl-1.9.13.jar, hadoop-mapreduce-client-common-2.8.5.jar, jmespath-java-1.11.199.jar, joda-time-2.10.5.jar, hadoop-mapreduce-client-shuffle-2.8.5.jar, nimbus-jose-jwt-4.41.1.jar

2020-06-18T08:28:02,083 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.2.jar, zstd-jni-1.3.3-1.jar, druid-kafka-indexing-service-0.18.1.jar, kafka-clients-2.2.2.jar, lz4-java-1.6.0.jar

2020-06-18T08:28:02,085 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.18.1.jar

2020-06-18T08:28:02,143 INFO [main] org.apache.druid.cli.CliCoordinator - Coordinator is configured to act as Overlord as well (druid.coordinator.asOverlord.enabled = true).

2020-06-18T08:28:02,254 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [sqlserver-metadata-storage], jars: sqlserver-metadata-storage-0.12.3.jar, mssql-jdbc-8.2.2.jre8.jar

2020-06-18T08:28:02,255 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-hdfs-storage], jars: hadoop-mapreduce-client-core-2.8.5.jar, hadoop-yarn-api-2.8.5.jar, jackson-dataformat-cbor-2.10.2.jar, aws-java-sdk-core-1.11.199.jar, commons-configuration-1.6.jar, commons-logging-1.1.1.jar, jackson-core-2.10.2.jar, apacheds-i18n-2.0.0-M15.jar, httpcore-4.4.11.jar, hadoop-common-2.8.5.jar, jetty-sslengine-6.1.26.jar, druid-hdfs-storage-0.18.1.jar, hadoop-client-2.8.5.jar, curator-framework-4.3.0.jar, htrace-core4-4.0.1-incubating.jar, ion-java-1.0.2.jar, commons-digester-1.8.jar, jcip-annotations-1.0-1.jar, xmlenc-0.52.jar, hadoop-mapreduce-client-app-2.8.5.jar, json-smart-2.3.jar, hadoop-auth-2.8.5.jar, asm-7.1.jar, jackson-core-asl-1.9.13.jar, commons-codec-1.13.jar, hadoop-aws-2.8.5.jar, jsp-api-2.1.jar, hadoop-yarn-client-2.8.5.jar, api-util-1.0.3.jar, commons-collections-3.2.2.jar, api-asn1-api-1.0.0-M20.jar, apacheds-kerberos-codec-2.0.0-M15.jar, hadoop-yarn-server-common-2.8.5.jar, aws-java-sdk-kms-1.11.199.jar, hadoop-annotations-2.8.5.jar, hadoop-mapreduce-client-jobclient-2.8.5.jar, hadoop-hdfs-client-2.8.5.jar, accessors-smart-1.2.jar, httpclient-4.5.10.jar, gson-2.2.4.jar, aws-java-sdk-s3-1.11.199.jar, curator-recipes-4.3.0.jar, leveldbjni-all-1.8.jar, commons-net-3.6.jar, jackson-databind-2.10.2.jar, jackson-annotations-2.10.2.jar, jackson-mapper-asl-1.9.13.jar, hadoop-mapreduce-client-common-2.8.5.jar, jmespath-java-1.11.199.jar, joda-time-2.10.5.jar, hadoop-mapreduce-client-shuffle-2.8.5.jar, nimbus-jose-jwt-4.41.1.jar

2020-06-18T08:28:02,258 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.2.jar, zstd-jni-1.3.3-1.jar, druid-kafka-indexing-service-0.18.1.jar, kafka-clients-2.2.2.jar, lz4-java-1.6.0.jar

2020-06-18T08:28:02,261 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.18.1.jar

2020-06-18T08:28:03,178 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

2020-06-18T08:28:04,236 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode

2020-06-18T08:28:04,237 INFO [main] org.apache.curator.utils.Compatibility - Using emulated InjectSessionExpiration

Exception in thread “main” java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. Unknown provider[sqlserver] of Key[type=org.apache.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[derby]]

at org.apache.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:109) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.derby.DerbyMetadataStorageDruidModule)

while locating org.apache.druid.metadata.MetadataStorageConnector

for the 1st parameter of org.apache.druid.guice.JacksonConfigManagerModule.getConfigManager(JacksonConfigManagerModule.java:52)

at org.apache.druid.guice.JacksonConfigManagerModule.getConfigManager(JacksonConfigManagerModule.java:52) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.JacksonConfigManagerModule)

while locating org.apache.druid.common.config.ConfigManager

for the 1st parameter of org.apache.druid.common.config.JacksonConfigManager.(JacksonConfigManager.java:49)

at org.apache.druid.guice.JacksonConfigManagerModule.configure(JacksonConfigManagerModule.java:41) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.JacksonConfigManagerModule)

while locating org.apache.druid.common.config.JacksonConfigManager

for the 1st parameter of org.apache.druid.guice.JacksonConfigProvider.configure(JacksonConfigProvider.java:65)

at org.apache.druid.guice.JacksonConfigProvider.bind(JacksonConfigProvider.java:38) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliOverlord$1)

  1. Unknown provider[sqlserver] of Key[type=org.apache.druid.server.audit.AuditManagerProvider, annotation=[none]], known options[[derby]]

at org.apache.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:109) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.derby.DerbyMetadataStorageDruidModule)

while locating org.apache.druid.server.audit.AuditManagerProvider

at org.apache.druid.cli.CliCoordinator$1.configure(CliCoordinator.java:181) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliCoordinator$1)

while locating org.apache.druid.audit.AuditManager

for the 3rd parameter of org.apache.druid.common.config.JacksonConfigManager.(JacksonConfigManager.java:49)

at org.apache.druid.guice.JacksonConfigManagerModule.configure(JacksonConfigManagerModule.java:41) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.JacksonConfigManagerModule)

while locating org.apache.druid.common.config.JacksonConfigManager

for the 1st parameter of org.apache.druid.guice.JacksonConfigProvider.configure(JacksonConfigProvider.java:65)

at org.apache.druid.guice.JacksonConfigProvider.bind(JacksonConfigProvider.java:38) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliOverlord$1)

2 errors

at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:72)

at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)

at org.apache.druid.cli.Main.main(Main.java:113)

Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

Sorry I may be missing something here. As per documentation
here, only MySQL and Postgres are in the list of supported databases. Not sure whether Druid supports MS SQLserver for metadata storage

Thanks!

Manu

Have you done this already by any chance?

Manually download the Microsoft JDBC drivers from ( https://www.microsoft.com/en-ca/download/details.aspx?id=11774) and drop it into my druid/lib directory.

and does your config look similar to this:

druid.metadata.storage.type=sqlserver
druid.metadata.storage.connector.connectURI=jdbc:sqlserver://;databaseName=druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=

Hi guys

Yes I have downloaded the driver and yes my config looks like that, I was able to fix it by downloading the source code of sqlmetastore extension and I built the jar in my local and deployed, after that it started working but don’t know why