Druid Basic Auth with Mysql Metastore

Hi,

We had successfully implemented basic auth in druid while derby was our meta-store.

After meta-store migration to mysql, we are trying to setup basic auth, but it doesn’t work.

Common.runtime.properties

druid.auth.authenticatorChain=[“MyBasicAuthenticator”]

druid.auth.authenticator.MyBasicAuthenticator.type=basic

druid.auth.authenticator.MyBasicAuthenticator.initialAdminPassword=password1

druid.auth.authenticator.MyBasicAuthenticator.initialInternalClientPassword=password2

druid.auth.authenticator.MyBasicAuthenticator.authorizerName=MyBasicAuthorizer

druid.auth.authorizers=[“MyBasicAuthorizer”]

druid.auth.authorizer.MyBasicAuthorizer.type=basic

# Escalator
for the internal client that is used for the internal druid communication

druid.escalator.type=basic

druid.escalator.internalClientUsername=druid_system

druid.escalator.internalClientPassword=password2

druid.escalator.authorizerName=MyBasicAuthorizer

<

And we are trying to hit the authentication APIs, but they don’t work(no error also).

API: http://host:co.port/druid-ext/basic-security/authentication/db/{MyBasicAuthenticator}/users

While opening the unified console it asks for userid/password, and once we enter the initial id/password following error is thrown

HTML Error: org.apache.druid.java.util.common.IAE: No userMap is available for authenticator: [MyBasicAuthenticator]

Can someone help ?

Hi Soumya ,

I don’t think there should be an issue with mysql as metadb

Do you see any error in the druid service logs ?

Thanks,

Vaibhav

Hi Vaibhav,

Logs from coordinator logs

2019-11-08T09:04:39,503 ERROR [main] org.apache.druid.cli.CliCoordinator - Error when starting up. Failing.

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_211]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_211]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211]

at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:443) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating]

at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:339) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating]

at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:140) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating]

at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:106) [druid-services-0.15.1-incubating.jar:0.15.1-incubating]

at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:57) [druid-services-0.15.1-incubating.jar:0.15.1-incubating]

at org.apache.druid.cli.Main.main(Main.java:118) [druid-services-0.15.1-incubating.jar:0.15.1-incubating]

Caused by: java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Encountered shared text value reference, even though document header did not declared shared text value references may be included

at [Source: [B@18dcb8a7; line: -1, column: 6]

at org.apache.druid.security.basic.BasicAuthUtils.deserializeAuthenticatorUserMap(BasicAuthUtils.java:158) ~[?:?]

at org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.start(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:129) ~[?:?]

… 10 more

Caused by: com.fasterxml.jackson.core.JsonParseException: Encountered shared text value reference, even though document header did not declared shared text value references may be included

at [Source: [B@18dcb8a7; line: -1, column: 6]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581) ~[jackson-core-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533) ~[jackson-core-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.dataformat.smile.SmileParser._reportInvalidSharedStringValue(SmileParser.java:2842) ~[jackson-dataformat-smile-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.dataformat.smile.SmileParser._handleSharedString(SmileParser.java:714) ~[jackson-dataformat-smile-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.dataformat.smile.SmileParser.nextToken(SmileParser.java:683) ~[jackson-dataformat-smile-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3776) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3721) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2836) ~[jackson-databind-2.6.7.jar:2.6.7]

at org.apache.druid.security.basic.BasicAuthUtils.deserializeAuthenticatorUserMap(BasicAuthUtils.java:155) ~[?:?]

at org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.start(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:129) ~[?:?]

… 10 more

I get the same error from the ui trying to use postgresql as a metadata store with basic security enabled.

Hi,
Did you add the extension : mysql-metadata-storage ?

Also, did you create the corresponding user in your database ?

If I’m not wrong, yours should be named “druid_MyBasicAuthenticator” and have the password “password1”

You should also grant him all privileges on

*“druid_MyBasicAuthenticator.

My Druid has the same version as yours (0.15.1) and is running with MySQL, so we could achieve it :slight_smile:

Guillaume

Guillaume,

I was under the impression that druid would create these users? Is that a wrong assumption?

Maybe it creates it when using derby but not when using mysql/postgresql.

You can have this information here (for MySQL, see 2.) :
https://druid.apache.org/docs/latest/development/extensions-core/mysql.html

By the way, I forgot you have to create the database (which you grant the user to)

so in postgresql/mysql druid’s basic security is using the actual database users not a table in the database? so i would need to manually create the users in the database?

No, you only need to create the main (admin) user.
This user is mapped to druid by authenticator/authorizer.

Then, as this user is granted all permissions on the database, druid is capable of managing its own tables (so managing segments, creating new users)

When I add a new users, I just need some API calls

so the metadata connects to postgres with whats in the jdbc string and the authenticator/authorizer connects to postgres as the username named admin with the set password? i keep getting 401 errors in the logs and i set the admin user’s password to what i have defined in the common runtime properties

Could you please post your common config (for that part) and what you created in postgres ?

It is true that your authenticator will get added to the metadata base automatically after you have proper setup.

You need to add nothing yourself to mysql.

I believe the problem lies with the basic security setup.

I have not yet had a chance to look at your configuration, but how many nodes do you have? Also do you have a location set for your authentication cache?

yes this was due to a bad configuration on my part. also it looks like the coordinator/overlord process needed to start before the historical/middlemanager. i restarted my historical/middlemanager and now all the logs are looking good. thank you all for your help.

actually, i forgot i removed the auth configuration yesterday and deployed that this morning. i put the configuration back in and i am seeing some errors in the historical/middlemanager:

Error 401 User authentication failed username[druid_system].

Problem accessing /druid-ext/basic-security/authorization/db/BasicAuthorizer/cachedSerializedGroupMappingMap. Reason:

    User authentication failed username[druid_system].

Powered by Jetty:// 9.4.12.v20180830
at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchGroupMappingMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:423) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchGroupAndRoleMapFromCoordinator$6(CoordinatorPollingBasicAuthorizerCacheManager.java:360) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchGroupAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:358) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$1(CoordinatorPollingBasicAuthorizerCacheManager.java:155) ~[?:?]

i can see in my postgresql server the table druid_config has entries in it.