Druid 0.11.0 release

We’re excited to announce our latest release, Druid 0.11.0!

Druid 0.11.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds two major security features, TLS support and extension points for authentication and authorization.

Major new features include:

  • TLS (a.k.a. SSL) support

  • Extension points for authentication and authorization

  • Double columns support

  • cachingCost Balancer Strategy

  • jq expression support in JSON parser

  • Redis cache extension

  • GroupBy performance improvements

  • Various improvements to Druid SQL

You can download the release here: http://druid.io/downloads.html

The full release notes are here: https://github.com/druid-io/druid/releases/druid-0.11.0

Documentation for this release is at: http://druid.io/docs/0.11.0/

Thanks to everyone who contributed!

Hi,
When i using Druid 0.11.0 with kafka-emitter extension to emit Druid metrics, i found kafka-emitter does not work normally . I suspect this is a bug .

[root@gpsegment1 druid-0.11.0]# ls extensions/kafka-emitter/
kafka-clients-0.10.2.0.jar  kafka-emitter-0.11.0.jar  lz4-1.3.0.jar  slf4j-api-1.6.4.jar  snappy-java-1.1.2.6.jar

[root@gpsegment1 druid-0.11.0]# grep "kafka-emitter" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties
druid.extensions.loadList=["druid-hdfs-storage", "kafka-emitter", "druid-caffeine-cache", "druid-kafka-indexing-service", "mysql-metadata-storage", "druid-datasketches", "druid-histogram"]

[root@gpsegment1 druid-0.11.0]# grep "emitter.kafka" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties
druid.emitter=kafka
druid.emitter.logging.logLevel=info
druid.emitter.kafka.bootstrap.servers=192.168.1.115:9092
druid.emitter.kafka.metric.topic=druid-metrics-monitor
druid.emitter.kafka.alert.topic=druid-metrics-alert
druid.emitter.kafka.producer.config={"max.block.ms":10000}

``

But error occurs when i startup services . My kafka version is 0.10.2 .

2017-12-27T15:55:43,216 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@20a7953c]
2017-12-27T15:55:43,337 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
2017-12-27T15:55:43,340 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2017-12-27T15:55:43,416 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d]
2017-12-27T15:55:43,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.security.Escalator] from props[druid.escalator.] as [io.druid.server.security.NoopEscalator@78422efb]
2017-12-27T15:55:43,480 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=‘null’, authorizers=‘null’}]
Exception in thread “main” com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. Problem parsing object at prefix[druid.emitter.kafka]: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null.
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
    while locating com.google.common.base.Supplier<io.druid.emitter.kafka.KafkaEmitterConfig>
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:134) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
    while locating io.druid.emitter.kafka.KafkaEmitterConfig
    for the 1st parameter of io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56)
    at io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule)
    while locating com.metamx.emitter.core.Emitter annotated with @com.google.inject.name.Named(value=kafka)
    at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:118)
    at io.druid.server.emitter.EmitterModule.configure(EmitterModule.java:78) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.emitter.EmitterModule)
    1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:99)
    at com.google.inject.Guice.createInjector(Guice.java:73)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:401)
    at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:62)
    at io.druid.cli.ServerRunnable.run(ServerRunnable.java:49)
    at io.druid.cli.Main.main(Main.java:108)
    Caused by: java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)
    at io.druid.guice.JsonConfigurator.configurate(JsonConfigurator.java:103)
    at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:200)
    at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:80)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
    at io.druid.guice.SupplierProvider.get(SupplierProvider.java:52)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
    at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402)
    at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
    at io.druid.server.emitter.EmitterModule$EmitterProvider.findEmitter(EmitterModule.java:142)
    at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:120)
    at io.druid.server.emitter.EmitterModule$EmitterProvider$$FastClassByGuice$$622b285e.invoke()
    at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:54)
    at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:89)
    at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
    at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93)
    at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
    at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80)
    at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:223)
    at com.google.inject.internal.Initializer.injectAll(Initializer.java:132)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:174)
    … 8 more

``

Then i had to rollback to Druid 0.10.1, it works very well .

Is this a bug of Druid 0.11.0 ?

Best wishes !

在 2017年12月5日星期二 UTC+8下午3:25:20,Gian Merlino写道:

Hi there!!

I am upgrading to kafka 2.11_1.0.0 (lastest) and Druid 0.11.0. In the older version, I was using tranquility but now I want to use kafka-indexing-service. I have problems when Druid trigger the handoff:

"java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.FileSystem.rename(Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/Options$Rename;)V from class org.apache.hadoop.fs.HadoopFsWrapper"

If I install druid_0.10.0 with druid-hdfs-storage extensión (jars from 0.9.0 version) it runs fine but don’t with own version jars (0.10.0).

Looking for into google, I found related links:

https://github.com/druid-io/druid/pull/4363
https://github.com/druid-io/druid/pull/4348
https://github.com/druid-io/druid/pull/3650
https://github.com/druid-io/druid/pull/3787

I can see that the bug isn’t fixed yet, is true?

Thanks!

Hi Zhou,

Sorry, but yes this is a bug. It will be fixed in Druid 0.12.0 by this patch: http://github.com/druid-io/druid/pull/5201

Hi Alberto,

Could you try double checking the hadoop jars you have on your classpath?

I think this should work fine out of the box, so I suspect you have some “extra” jars somewhere.

Gian, You were right, the problem was the loaded wrong libs.
My old systemd file:
[Service]
Type=simple
User=druid
Group=druid
Environment=‘CLASSPATH=/opt/druid/lib/:/opt/druid/conf/_common:/opt/druid/conf/_common/hadoop_conf:/opt/druid/conf/overlord:/opt/druid/hadoop-dependencies/__hadoop-client/2.3.0/__’
Environment=‘JMX=-Dcom.sun.management.jmxremote.port=3003 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false’
I fixed the hadoop-client path to 2.7.3

Thanks!!

Alberto

Hi Gian,

I got it, Thank you very much !

Best wishes !

在 2018年1月9日星期二 UTC+8下午1:15:01,Gian Merlino写道: