We have recently started using Druid and have been experimenting with different configurations. One question I had was what is an optimum Task Duration.
Should the aim be to increase MM memory and have longer task duration as we observed if you keep segmentGranuarity smaller, the segments would pushed to deep storage.
As with so many things the answer is ‘it depends’. The default (1 hour) is a reasonable default. Increasing it somewhat can allow Druid to generate more consistently sized segments (which is good), especially if you are getting late data. On the other hand, one downside of a longer duration is that balancing (of tasks to MMs) happens less often. In most cases, something in the 1–4 hour range is the right choice for balancing the above considerations.