We are pre-processing data with Spark and wanted to use Tranquility to ingest data into Druid. I can’t find any APIs to use Spark with Druid. What is the best way forward?
You can use tranquility’s direct API along with Spark Streaming’s foreachRDD to do this. The section “Design Patterns for using foreachRDD” on the Spark docs have some samples: http://spark.apache.org/docs/latest/streaming-programming-guide.html
Something like the last sample there should work well with tranquility. Tranquility beams can be used by multiple threads at a time, so you don’t need a ConnectionPool- just a singleton would suffice.