prioritization for IndexTasks?


Our ingestion setup, at least for the moment, is entirely batch-based. We get “eventually correct” updates from an upstream data warehousing service for (day) granularity sized batches of data which we turn into indexing tasks for the indexing service.

Is there any way provided for controlling the priority of these tasks? I don’t see anything in the documentation, but just want to double check that I’m not missing something.

Here is the use case I’m worried about. In the normal course of events we’ll mostly be getting periodic updates to today’s data plus occasional updates to older data trickling in. In that case I’m not too concerned about prioritization. But there will be some times when we’ll want to backfill large ranges of historical data, due to new dimensions becoming available and/or schema changes. If we just dump six months worth of “index this day” events into our system and create tasks for them all at once, it will saturate our indexers. I’d want to give priority to recent (new) data in this kind of situation rather than have it blocked waiting for six months of backfilling to finish.

Of course there are many ways we can handle this external to Druid and task creation - our own priority queue, or throttling the backfill notifications, spinning up additional indexers, etc. But is there any way to handle it from the Druid side?

Another lever that might help - can I control which indexers are eligible to take a task in the task spec somehow? E.g., we spin up an extra set of indexers to handle a backfill, and restrict the backfill tasks to go to those nodes, guaranteeing that one or two remain free for the new stuff?



Kind of:

Priority based task locking is something that’s on the docket for pushing over the finish line in 0.9.1

I think it will accomplish what you are looking for, though you may need some sort of decision maker on your side to determine what the appropriate priority for different tasks are.

To answer your second question, you can specify which middle-managers get which tasks based on a datasource basis by using fillCapacityWithAffinity worker select strategy , or any arbitrary logic using javascript worker select strategy (To be used VERY carefully as setting it wrongly can lead to tasks just waiting on indexing-service and not getting assigned to any worker at all.)
refer Worker Select Strategy here -

Thanks for the pointer.

I’m confused from reading through the pull request - does the task priority scheme implemented there affect execution priority as well as task locking? It seems to be focused on task locking preemption. In my use case, it’s actually execution priority that I want to control. In the backfill scenario the backfill tasks will generally be operating on different segments so they won’t be encountering lock contention anyway. What I want to avoid is having a fresh data ingestion task waiting to start execution queued up behind several dozen backfill tasks… I want to be able to put it at the head of the line.

Maybe my understanding of the task queue is off… is it the case that the max number of concurrently executing tasks is equal to the number of indexer nodes, or is that wrong?

Max number of concurrently executing tasks == # of middle manager nodes * slots per middle manager node

<resurrect old thread!>

We’re now attempting to ingest for a second data source, and we have ingestion tasks for both sources being created together. It appears that tasks are chosen from the Pending Tasks list for assignment to a worker based on the lexicographic order of the task id - is that correct? Because when we have a bunch of pending tasks for both data sources, the ones for the data source with the name that is lexicographically first all get assigned to workers before any of the ones for the other data source.

Presumably this happens before any of the logic associated with the Worker Select Strategy kicks in - the Worker Select Strategy picks which worker to use based on a task that has already been selected from the pending list?

I think we can probably work with this by playing with task ids, but it seems not very robust in cases where you’re prone to build up a backlog of pending tasks. Is there any other way to control the order in which tasks come off the pending list?

Hmm, I don’t think there is a way to control that behavior currently. Right now there are no particular guarantees about ordering.

Okay… so it does work as I outlined, then?

How do people normally deal with ingesting multiple data sources at once? Do you just try to have enough indexer capacity to avoid building backlogs of pending tasks?

Lexicographic order sounds believable :). I’m not 100% sure how it works though (there aren’t guarantees on ordering currently so whatever it does would be “accidental”).

Most people do try to avoid pending tasks, yeah, unless they really don’t care about what order things get done in. - should help with ordering pending task assignment by queue insertion time instead of lexicographic.