Hi, I have a C++ application that's receiving streaming data at a high rate to go into Druid. I'm trying to determine the fastest and most efficient way to insert based on this setup. It seems Kafka is used for streaming inserts, but I'd like to avoid setting up and maintaining Kafka if possible. It seems that LocalFirehose is a good option, but I would be writing data out to a file before loading, which seems inefficient. Does anyone know if there might be a better way for my situation?
How is the streaming data received by C++ app? If your app handles resiliency of your incoming data where it guarantees no duplicates are sent, it can resume pushing data to Druid where it stopped due to failures, and have data retention in place, then you can push it straight to Druid via Tranquility. If you C++ app don’t have the resiliency as defined prior, then having a queueing tool in the middle outweighs the work that will be done to set it up.
Hi Rommel, it's a userspace networking app that receives raw udp metrics. The are no duplicates sent, and that can be guaranteed. My first thought was tranquility would be slow due to http, but maybe that's not the case. Have you seen performance numbers for it? I'm looking at multiple Gbps streaming in, or millions of insertions per second.