Data Format with Schema Changes

My team and I are wondering how to do some data formatting while handling schema changes.
In our ingestion spec we have the schema pointed to a URL so that anytime a new schema gets posted there, Druid sees the changes. We have modified our dimensions spec a few times as we tried new things.

This is what we have done so far.

  1. Initial schema with 3 columns, and a spec file where the 3 columns were defined in the dimensions area. Worked as expected.
  2. Modified schema added a column so now we have 4 columns, and a dimension spec with 3 columns defined. Only got 3 columns ingested, expecting 4
  3. With the same 4 column schema, and a schema-less dimension, got all 4 columns ingested, expected 4. Worked as expected.
    So after our 3rd attempt, a schema-less dimension spec, we got the ingestion we where expecting after a schema change. What we figured out is that when you have a dimensions spec filled out those are the only columns you will see, hence we went with the schema-less dimension. However with the schema-less dimension you have no control over the column type. We would like to be able to have control over some column types while maintaining awareness of schema changes. Is this even capable?

Hope I explained that well enough, if not I will clarify anything

P.S. we aren’t anticipating a lot of schema changes but if it happens it would be nice if Druid will just handle the change and keep the proper data types.

Afternoon - as far as I know yes that’s a limitation - probably worth a gander at this:

Thanks for the response Peter.