Updating a spec

I created a simple realtime spec to use with Kafka, and I want to add some more dimensions and metrics to it. However, I’d also like to preserve the data I collected with it. I haven’t seen any guides to doing this, but since the structure of the data is defined by the spec, I would assume that all I have to do is update the spec’s JSON.

Are there any risks to updating the spec’s dimension/metric fields? Should I expect to have to somehow backfill the data for either of these?

Druid segments, once finalized, are immutable and are pretty self contained. Segments know what dimensions and metrics they have. You can have different segments that have different schemas for the same table (datasource). That is to say, you should be able to change the schema of a segment. One caveat is if you change a dimension to a metric before finalization happens, you may get a error about merging segments. For realtime ingestion, Tranquility (https://github.com/druid-io/tranquility) provides more seamless schema migrations.

So does Tranquility automatically handle schema migrations if the spec is changed when using it for realtime ingestion? Or does it just make the migration process simpler?

What tranquility does is create limited lifetime tasks (usually these tasks are hourly). So for 1 hour of data, a table (datasource) has a certain schema. If you update the schema in Tranquility, the changes will be reflected in the next hour’s task.