Possible reasons for "Failed to create merged beam: druid:overlord/pageviews"

prathamesh · January 24, 2019, 11:04am

Hi,

I have an application using tranquility core to send streaming data to druid. I have a single node druid setup as well as the druid instance setup using docker-druid image and can send data to both these instances successfully.

I have now setup a multi-node cluster. This is going to be a fully configure HA cluster with external zookeeper,metadata store and deep storage. After starting coordinator, overlord, historical and middle manager i accessed the coordinator console and was able to see that the setup has recognized all the nodes and indexing slots as well the historical node (as i can see this on the coordinator console)

However when i try to run my application to send data to this setup i get following error:

SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Jan 24, 2019 4:09:16 PM org.hibernate.validator.internal.util.Version

INFO: HV000001: Hibernate Validator 5.1.3.Final

Jan 24, 2019 4:09:19 PM com.twitter.finagle.Init$$anonfun$4 apply$mcV$sp

INFO: Finagle version 6.43.0 (rev=494d336279792deb44797dd57bb572ee747f84f0) built at 20170310-173428

Failed to send message: %sjava.lang.IllegalStateException: Failed to create merged beam: druid:overlord/pageviews

``

I have checked the zknode and overlord address is proper. I also see a zknode with datasource name under zknode:

ls /tranquility/beams/druid:overlord//pageviews

I also see mutex and data nodes under pageviews, but i still get “Failed to create merged beam: druid:overlord/pageviews” error. I didn’t see any errors in the logs for any of the 4 druid processes that are running. Neither is there any additional information in the application logs.

What are the possible reasons for this error? how can this be debugged further?

Thanks,

Prathamesh

prathamesh · January 25, 2019, 6:33am

Which node is responsible for running the tasks created by tranquility? I think it should be Middle Manager. can someone confirm?

Also, it appears that tranquility is able to find the overlord address since it is creating the entries in zookeeper under /tranquility. Where else could things go wrong leading to not being able to ingest the data using Beams?

Thanks,

Prathamesh

Nishant_Bangarwa · January 28, 2019, 12:11pm

Hi Prathmesh,
The choice of where the task is run depends on the runner being used druid.indexer.runner.type.

When it is set to local, the task will be run on the overlord itself.

when using remote, task is assigned to one of the available middlemanagers using the configured worker selection strategy.

See runner type and worker selection strategy here - http://druid.io/docs/latest/configuration/index.html#overlord

Hopefully it provides more info on the debugging.

prathamesh · January 30, 2019, 4:08am

Hi Nishant,

Thanks for the additional info.

The root cause for this error was that java.io.tmpdir was not writable location. However there was no indication in the logs of overlord. All i could see was failed task in my application and entries under zookeeper node **/tranquility/beams/druid:**overlord//data as shown below :

{“latestTime”:“1970-01-01T00:00:00.000Z”,“latestCloseTime”:“1970-01-01T00:00:00.000Z”,“beams”:{}}

It was only after enabling debug logging that i came to know the issue with directory.

Thanks,

Prathamesh