Running standalone realtime node and indexing service

Hi,

My Druid cluster setup involves Realtime, Historical, Broker and Coordinator nodes and my Realtime node is ingesting data from Kafka using the Kafka Eight Firehose. I would like to configure some rules to drop older segments.

Would I be able to run the Indexing Service (Overlord, MiddleManager nodes) alongside a standalone Realtime node so as to just make use of the Indexing Service for submitting the kill task to remove older segments ? Please note that I don’t want to use the Indexing Service for ingestion, but just for cleaning up old segments. Ingestion is going to be handled by standalone Realtime nodes.

When I tried to run my Overlord node alongside Realtime nodes, I ran into the following error:

2016-09-09T01:36:48,332 ERROR [main] io.druid.cli.CliRealtime - Error when starting up. Failing.

com.google.inject.ProvisionException: Guice provision errors:

  1. Error injecting constructor, java.lang.NullPointerException

at io.druid.guice.FireDepartmentsProvider.(FireDepartmentsProvider.java:43)

while locating io.druid.guice.FireDepartmentsProvider

at io.druid.guice.RealtimeModule.configure(RealtimeModule.java:85)

while locating java.util.List<io.druid.segment.realtime.FireDepartment>

for parameter 0 at io.druid.segment.realtime.RealtimeManager.<init>(RealtimeManager.java:80)

while locating io.druid.segment.realtime.RealtimeManager

at io.druid.guice.RealtimeModule.configure(RealtimeModule.java:102)

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:106)

while locating io.druid.server.QueryResource

1 error

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:153) ~[druid-api-0.9.1.1.jar:0.9.1.1]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:91) [druid-services-0.9.1.1.jar:0.9.1.1]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40) [druid-services-0.9.1.1.jar:0.9.1.1]

at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1.jar:0.9.1.1]

Caused by: java.lang.NullPointerException

at java.io.FileInputStream.(FileInputStream.java:130) ~[?:1.8.0_91]

at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:708) ~[jackson-core-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2122) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.guice.FireDepartmentsProvider.(FireDepartmentsProvider.java:46) ~[druid-server-0.9.1.1.jar:0.9.1.1]

    ...

Any suggestions about resolving this error would also be great :slight_smile:

Thanks,
Saravana

Hi Saravana,

I’m trying to replace my Realtime nodes with the Indexing Service and had similar question. Were you able to run Realtime and Indexing nodes side by side without issue?

Hi,
you should be able to run realtime nodes & indexing service for kill Tasks together. FWIW, you will need to make sure that you only delete the data which is old, handed over to historical nodes not served by realtime node.

the NPE seems like a config issue, did you changed anything in realtime node config ? could you share the config for more details.