Help! Cannot start realtime node.

Using druid 0.8.1.
When i try to start realtime node, I see this:

error

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134) ~[druid-api-0.3.9.jar:0.8.1]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.1.jar:0.8.1]

Caused by: java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id ‘kafka-0.8’ into a subtype of [simple type, class io.druid.

data.input.FirehoseFactory]

at [Source: /opt/druid/config/realtime/dripstatspec.json; line: 68, column: 9] (through reference chain: java.util.ArrayList[0])

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.guice.FireDepartmentsProvider.(FireDepartmentsProvider.java:52) ~[druid-server-0.8.1.jar:0.8.1]

at io.druid.guice.FireDepartmentsProvider$$FastClassByGuice$$229da177.newInstance() ~[guice-4.0-beta.jar:0.8.1]

at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49) ~[druid-api-0.3.9.jar:0.8.1]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:167) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:99) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:956) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3066) ~[jackson-databind-2.4.4.jar:2.4.4]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2122) ~[jackson-databind-2.4.4.jar:2.4.4]

at io.druid.guice.FireDepartmentsProvider.(FireDepartmentsProvider.java:43) ~[druid-server-0.8.1.jar:0.8.1]

at io.druid.guice.FireDepartmentsProvider$$FastClassByGuice$$229da177.newInstance() ~[guice-4.0-beta.jar:0.8.1]

at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49) ~[druid-api-0.3.9.jar:0.8.1]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

… 5 more

and then it shuts down.

Please help!

You need to include the kafka 8 module in your list of extensions.

kafka is included in the extensions:

druid.extensions.coordinates=[“io.druid.extensions:druid-s3-extensions”,“io.druid.extensions:druid-kafka-eight”,“io.druid.extensions:mysql-metadata-storage”]

this realtime node had been running for weeks.

We just had to terminate it cause it ran into a high cpu issue.

now whenever I try to start it, it dies with this error!

Did you make any changes here: /opt/druid/config/realtime/dripstatspec.json?

Druid is having trouble parsing your spec. Can you post the spec?

oops ok i see that there was an issue with the command line arguments i was passing.
I am now able to startup but its stuck here:

2015-12-14T22:04:24,168 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageTablesConfig] from props[druid.metadata.storage.tables.] as [io.druid.metadata.MetadataStorageTablesConfig@2e869098]

2015-12-14T22:04:24,174 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.metadata.MetadataStorageConnectorConfig] from props[druid.metadata.storage.connector.] as [DbConnectorConfig{createTables=true, connectURI='jd

bc:mysql://druidmysql.ctebvndt42nz.us-east-1.rds.amazonaws.com:3306/druid’, user=‘druiduser’, passwordProvider=io.druid.metadata.DefaultPasswordProvider}]

2015-12-14T22:04:24,222 INFO [main] io.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage

2015-12-14T22:04:24,227 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.FilteredServerViewProvider] from props[druid.announcer.] as [io.druid.client.FilteredBatchServerViewProvider@650ae78c]

2015-12-14T22:04:24,249 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2015-12-14T22:04:24,250 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

2015-12-14T22:04:24,255 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100396096] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]

2015-12-14T22:04:24,255 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]

2015-12-14T22:04:24,255 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]

2015-12-14T22:04:24,255 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]

2015-12-14T22:04:24,264 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.segment.data.BitmapSerdeFactory] from props[druid.processing.bitmap.] as [ConciseBitmapSerdeFactory{}]

2015-12-14T22:04:24,267 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@1a0d96a5]

2015-12-14T22:04:24,278 INFO [main] org.eclipse.jetty.util.log - Logging initialized @5174ms

2015-12-14T22:04:24,401 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.core.LoggingEmitter.start()] on object[com.metamx.emitter.core.LoggingEmitter@3bec5821].

2015-12-14T22:04:24,402 INFO [main] com.metamx.emitter.core.LoggingEmitter - Start: started [true]

2015-12-14T22:04:24,402 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@2e35

72e8].

2015-12-14T22:04:24,402 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@77c10a5f].

2015-12-14T22:04:24,403 INFO [main] io.druid.curator.CuratorModule - Starting Curator

2015-12-14T22:04:24,403 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

2015-12-14T22:04:24,438 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-12-14T22:04:24,438 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=realtime-group-3edv.c.root-isotope-91018.internal

2015-12-14T22:04:24,438 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_45

2015-12-14T22:04:24,438 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

2015-12-14T22:04:24,438 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre

Doesnt move past this.

Is this normal?

ok i can confirm things are back to normal now.
thanks again for your speedy responses.

Hmm, I wonder if the logging is misconfigured. Will ask someone to take a look.