Knox proxy support in druid

Hi,

I want to use hdfs as deep storage for druid. My hdfs is behind a firewall and can only be accessed using apache knox proxy. Is there any way in druid to configure knox proxy? If yes, do you guys have any doc around it?

Thanks

Ankur

Hi Ankur,

You can setup druid behind your firewall and access druid nodes via KNOX gateway.

For configuration you will need to configure druid service endpoints in your KNOX topology file.

Refer to https://github.com/apache/knox/blob/master/gateway-release/home/conf/topologies/sandbox.xml#L136 for sample config.

Details about the mappings for druid http end points can be found here - https://drive.google.com/file/d/0BylGP2Nr5IM0bzJBN3Etbnh3Q28/view

Related JIRA - https://issues.apache.org/jira/browse/KNOX-758

Also, you might need to update your KNOX gateway if you are running on an older version.

Hi Nishant,

Thanks for the reply. The jira ticket talk about Indexer UI, Coordinator and broker but does it support data ingestion i.e. tranquility?

I am pushing my company to upgrade the knox version but I want to be sure before that.

Thanks

can you provide more details on how you are using tranquility and what sort of knox support you are looking for tranquility ?

Hey,

Thanks for your reply. Sorry I was not clear. Here is my topology.

kafka --> kafka consumer(massage data) --> tranquility server–> indexing service(task) ---->firewall (accessible through knox proxy)----> hdfs (deep storage).

AND

historical Node/read historical segments ---->firewall(accessible through knox proxy) --> hdfs

So my question is when a task is handing off a segment to deep storage can it go through knox proxy? Is that supported? Similarly, when historical node reads data, can it read through knox proxy?

Thanks