Google Pubsub Support

I have been trying to write a firehose extension for Google pubsub.
I am not quite sure If I need a realtime druid server or tranquility can poll the pubsub subscription and index the data (same as kafka).

Should I extend DistributionMain and add pubsub support?



Hi Rotem,

I’m afraid I don’t know the answer to your question, but I was wondering if you already put your changed on github somewhere so I can track them? We are running a Druid cluster on GCE and might be interested in this so we can replace Kafka with Google PubSub.


Hey Rotem,

Something along the lines of Tranquility Kafka (but for Google Pub/Sub) sounds good to me. If you want to just do it on your own, you can use Tranquility Core (like Tranquility Kafka does, under the hood) and write a program that reads from pubsub and writes to Druid. Or if you want to contribute it, you can create a tranquility-googlepubsub module similar to the tranquility-kafka module, and add a runner to DistributionMain.