I am evaluating Druid for our product in the network monitoring space. Wanted to check how is the support for storing and querying traceroute data in Apache Druid?
Welcome @Abhinav_Shroff! Sorry for the long delay in replying, but several of us were offsite.
Can you tell us a bit more about your use case? What’s the format of the traceroute data?
Thanks for your response.
I was evaluating Apache Druid to store the traceroute, flow data, PCAP files, and the metrics we derive based on the data. So that we can leverage the query engine to build dashboards.
The traceroute data is getting stored in JSON format and the flow data is in v9 format.
The JSON format should be no problem, but I’m not sure about the v9 format. I’ve taken the liberty of cross posting your question to the Apache Druid Slack workspace to invite more people into the conversation.
The blog was helpful, as it shows that Netflow v9 and IPFIX can be stored in Apache Druid, but just a quick clarification, so this is something that Apache Druid itself will support or it is something that only Imply version of Druid does?
Also, can we also store PCAP files(these are in binary format) in Druid?
If I understand the blog correctly, it is showing how to use nfacctd to publish data directly to kafka and then use Druid’s kafka ingestion capability to consume it. The data format in this case is JSON which Druid supports.
The PCAP binary file format is not directly supported, the supported input formats for ingestion are described here:
You can convert PCAP files into JSON or some other supported format. Here’s one that converts to JSON.
Thanks, @Sergio_Ferragut, for the clarification regarding the flow data blog.
Also, is there a way to store PCAP files through a custom format supported by Apache Druid?
Apache Druid uses its own internal format for ingested data called Segment files. Data can be ingested from any of the supported input formats that I described in the previous post. You can also extend the input format support by creating a custom extension by extending