Can I run multiple indexing service(overlord) call for data ingestion simultaneously.

Hi,

When I insert the data with curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @example_index_task.json localhost:8090/druid/indexer/v1/task one after other manually. It works fine.

The druid is able to insert the data correctly.

However when I automated the above the with script to call druid indexer simultaneously with different data. It shows in logs data insertion success like below.

Task completed with status: {

“id” : “index_spinistagging_2015-11-17T12:38:33.475Z”,

“status” : “SUCCESS”,

“duration” : 937

}

However the same data is not available for querying when query using Broker 8082.

The configuration I am using is

data store : local file system (linux)

db : derby

indexing - druid indexer ( overlord process ).

Can you please suggest what going wrong ?

Shall I keep time gap before calling another indexing service.

Thanks in advance,

Aman

Hello,

What are the contents of your task json ? What parameter are you changing in the indexing task before submitting task multiple times with the script ? Remember that if you ingest data for same interval and same datasource multiple times the last one will overshadow the previous ones.

-Parag

Thanks for reply.

What are the contents of your task json ? Per day data from Production

What parameter are you changing in the indexing task before submitting task multiple times with the script ?

changing data name but same data source and interval I took long time interval.

I was taking same time interval for all the submission.

Please correct me if I am wrong here.

Ideally I should interval same for which the data is like if a data is for day the interval should be also that day right ?

Regards,

Aman

yes your interval should correspond to the data you are trying to index. If your data does not fall into the interval specified in your task json then the data will be ignored and not indexed.

Hi,

For push based ingestion it is much easier to use tranquility.

– Himanshu

Aman, please post teh task json

Hi Fangjin/Parag,

Thanks for your guys reply.

Actually I was trying with different dimension specs with same data source and time interval. So it was overriding it.

Thanks for the making it clear.

Regards,

Aman