Rollup errors when appending data using appendToExisting

Apache Druid Q&A Ingestion
,
#1

Apache Druid’s native batch ingestion lets you append data to existing segments using the appendToExisting flag in the ioConfig object in your ingestion spec. appendToExisting only works with forceGuranteedRollup set to false and dynamic partitioning. This is well documented, but if you make a mistake in your configuration, the rollup errors may not be intuitive, so this article lists the misconfiguration errors along with their cause to help you troubleshoot.

Rollup errors with appendToExisting

  • Error: “DynamicPartitionsSpec must be used for best-effort rollup” is caused by appendToExisting:true, forceGuaranteedRollup:false, partitioning=hashed.
  • Error: “DynamicPartitionsSpec cannot be used for perfect rollup” is caused by appendToExisting:true, forceGuaranteedRollup:true, partitioning=dynamic.
  • Error: “Perfect rollup cannot be guaranteed when appending to existing dataSources” is caused by appendToExisting:true, forceGuaranteedRollup:true, partitioning=hashed OR partitioning=single_dim.

Solution
The solution in any of thee cases is to use the correct configuration for appendToExisting: appendToExisting:true, forceGuaranteedRollup:false, partitioning=dynamic. Or otherwise set appendtoExisting:false.

For more information on appending data, see:

Relates to all Apache Druid versions that support appendToExisting

Thanks to Apache Druid committer Jihoon Son for the detailed description of Druid’s behavior!

2 Likes