Storing pre-calculated metrics data from Codahale with no aggration

Hi.

Our company currently uses the Codahale metrics libraries for our application metrics, e.g. timers (histogram and meter), gauges, etc. These metrics are retrieved on a regular basis and are then streamed to InfluxDB via a Kafka topic. The Codahale metrics are already pre-calculated, e.g. the percentiles, averages, min and max values, etc. so no further action is required other than to just store them in InfluxDB. Each row already contains all these values in individual columns, e.g. 50th percentile, 98th percentile, 1 minute rate, 5 minute rate, etc.

We are already using Druid for storing product usage information but are investigating whether or not we can also store this metrics data in Druid as well. This will mean that InfluxDB is no longer required in our solution.

The issue I have run into however is that Druid seems to be based on raw values and applies aggregation to them. For example rather than taking an already pre-calculated 50th percentile value the metrics section of the ingestion spec requires this to instead have a aggregation type of doubleSum, min, average etc. I am just wanting to store the values as they are ingested as no action (calculation or aggregation) is required for this data. I have upgraded to Druid 0.9.2 so that I can disable the rollup and I am just wanting each event with no aggregation. At query time we would just be pulling back that same raw data and plotting the timeseries for each value for that timestamp.

Is it possible to achieve what I’m looking for, or am I misunderstanding how this should be used? Should every metric value consumed be specified as being an average in the ingestion spec for cases where due to the query granularity there may be more than a single row returned within that interval? Is this not really the intended purpose of Druid and would be better suited for a different time series database, i.e. we remain using InfluxDB for this?

Thanks,

Simon

Assuming that you have a set of tags that uniquely identify each of the timeseries you are throwing into Influx, just set the data granularity to “none” and use the doubleSum aggregator for each of the metrics. This will result in no rollup during your ingestion because the dimensions and time stamps will be unique.

When you query, if you aren’t specifying enough filters/grouping by enough dimensions to look at individual “series”, then you will need to average the values to get anything even semi-useful out of it (include a count aggregator, do a sum aggregator and then divide them with a post aggregator).

If you want higher fidelity metrics, you would be better off emitting a metric per event. The way you are doing it, it is impossible to figure out the actual percentiles for your end user experience, because it is not “correct” to aggregate the percentiles as exposed by codahale metrics.

Fwiw, this use case for Druid is completely legit. We use Druid-on-Druid to watch our metrics for our Druid clusters. We use it by emitting individual events because that provides the best results, but there is also some codahale-style data that we deal with. Others also push codahale-style metrics into Druid. The power of Druid for these use cases is two fold: 1) its ability to aggregate the entire world and 2) not having limitations on cardinality that others systems do (Druid clusters regularly run in the billion+ range of cardinality for timeseries). So using a mechanism that doesn’t allow for “correct” aggregation over the whole means that you are missing out on some of the full value proposition.

–Eric

Hi Eric,

Thanks for your response. Some followup questions/clarifications in-line below.

Assuming that you have a set of tags that uniquely identify each of the timeseries you are throwing into Influx, just set the data granularity to “none” and use the doubleSum aggregator for each of the metrics. This will result in no rollup during your ingestion because the dimensions and time stamps will be unique.

Am I correct in assuming you’re referring to the value of the “queryGranularity” parameter in the ingestion spec? The answer to your above points is that yes our spec and data should have matched what you mentioned. Since the upgrade to Druid 0.9.2 we have set the “rollup” value in the spec to “false” so this should achieve the same outcome as well, i.e. single rows with no rollup.

When you query, if you aren’t specifying enough filters/grouping by enough dimensions to look at individual “series”, then you will need to average the values to get anything even semi-useful out of it (include a count aggregator, do a sum aggregator and then divide them with a post aggregator).

Each metric is named (and tagged where appropriate) to ensure that they are all uniquely identified. You are correct however in that averaging those values in the query will be required in the advent that we are wishing to look at responses times across servers, v.s. an individual server. The documentation states that the finest granularity Druid has for querying is by millisecond precision, so although potentially just an edge case, it is possible we could get multiple data points even if these are unique depending on where they fall.

If you want higher fidelity metrics, you would be better off emitting a metric per event. The way you are doing it, it is impossible to figure out the actual percentiles for your end user experience, because it is not “correct” to aggregate the percentiles as exposed by codahale metrics.

That was pretty much our issue. The percentiles for the end user experience are available as Codahale has already calculated them. Those values will be accurate, and not aggregated by Druid, in the ingestion process as they will not be rolled up. The issue would arise during query time if multiple rows are returned due to non-uniqueness and were attempted to be aggregated.

Fwiw, this use case for Druid is completely legit. We use Druid-on-Druid to watch our metrics for our Druid clusters. We use it by emitting individual events because that provides the best results, but there is also some codahale-style data that we deal with.

By “codahale-style” do you mean that they are actually generated by Codahale, or that they are just pre-calculated (e.g. rates and percentiles) in the same fashion as how Codahale does? Do you have an example of this? How are you dealing with these currently, using the same approach as what you have suggested above?

Others also push codahale-style metrics into Druid.

Do you have any examples of this? Am I correct in assuming that they would also be following a similar pattern as yourself?

The power of Druid for these use cases is two fold: 1) its ability to aggregate the entire world and 2) not having limitations on cardinality that others systems do (Druid clusters regularly run in the billion+ range of cardinality for timeseries). So using a mechanism that doesn’t allow for “correct” aggregation over the whole means that you are missing out on some of the full value proposition.

Absolutely agree with you there, hence wanting to get this correct the first time round.

Thanks once again for your input Eric, much appreciated.

Simon

Simon,

You’re right that disabling “rollup” is all you have to do to prevent Druid from aggregating at ingestion time. It will still potentially aggregate at query time, unless you use the “select” query (which doesn’t aggregate). So if you have “rollup” false and use the “select” query then you’re guaranteed that Druid will not do any aggregation at any point.

… which may or may not be what you want.

For aggregating rates, counters, etc, IMO you should definitely let Druid do aggregation. It doesn’t lose any fidelity and it’s fast.

For quantiles and histograms there is a more interesting tradeoff. Precomputing them before sending them to Druid (like you’re doing now) means you shouldn’t let Druid aggregate them, because you can’t just average two 98%iles and get a 98%ile of the combined set – that’s not how percentiles work. You could deal with this by turning off rollup in Druid and using the select query to pick out the records you want, without aggregation.

Or you could bypass Codahale metrics completely, send the raw telemetry to Druid, and let Druid do the aggregation. We have an approximate histogram sketch (http://druid.io/docs/latest/development/extensions-core/approximate-histograms.html) that can be used for computing percentiles in this case. If you’re using this you probably want to turn rollup on, to allow Druid to combine multiple raw data points into one Druid row. Each row will contain a sketch that can be aggregated with other rows’ sketches to get percentiles for any combination of dimensions you want, using an aggregating query like timeseries, topN, or groupBy. The storage footprint is a bit higher but you gain a lot of flexibility in what you can query for.