We are doing batch ingestion on Druid and we are passing quite a lot of S3 files when ingesting, around 56 every time.
The job succeeds and some data gets ingested, but we’ve checked and not all of it does. It seems that all the data from the first few files do get ingested, but the data from the last files don’t.
We are wondering, is there a limit somewhere on the number of files that Druid ingests? If so, is there a way that we can change it?
I guess that the best way to do this is to create several jobs with a smaller number of files, or not?