Is there any way to use multiple EMR Hadoop clusters for ingestion in different datasources?

Hi all,

Is there any way to use multiple EMR Hadoop clusters for ingestion in different datasources? The fact the Hadoop’s config files should be placed in the Druid config folder limits me to have only one EMR Hadoop per druid cluster. Does anybody know a way how to specify EMR/Hadoop per ingestion rather than per druid cluster?

Cheers,
Vadim.

Hi Vadim

I’m not sure about this, but looks like a nice useful feature request.

druid.storage.storageDirectory can accept multiple values, but ingestion spec seem to be using only the path instead of complete hdfs path. You could give a try and split them to multiple tiers.

Thanks & Rgds

Venkat

Vadim

Please also look up ->

https://groups.google.com/forum/#!topic/druid-user/Il9RFb6OUjE

Thanks & Rgds

Venkat