Data is getting overridden even with same data source and different time stamp

Hi,

As suggested before I am using different time interval and same dataSource for data ingestion. But still it is overriding the previous ingested task data.

Configuration of druid : derby with localStorage

Below is the time interval I used for different ingestion tasks with same data source.

 "intervals" : [ "2015-03-31T00:00:00.000Z/2015-04-30T00:00:00.000Z" ]
 "intervals" : [ "2015-04-30T00:00:00.000Z/2015-05-30T00:00:00.000Z" ]

“intervals” : [ “2015-05-30T00:00:00.000Z/2015-06-29T00:00:00.000Z” ]

“intervals” : [ “2015-06-29T00:00:00.000Z/2015-07-29T00:00:00.000Z” ]

so on …

“intervals” : [ “2015-10-27T00:00:00.000Z/2015-11-26T00:00:00.000Z” ]

However when I fire time boundary query I can see data only of November that is last task which ran.

All previous data are missing while querying.

Time Boundary give below result instead of giving result from previous months:

[
{
“timestamp”: “2015-11-01T07:05:24.000Z”,
“result”: {
“minTime”: “2015-11-01T07:05:24.000Z”,
“maxTime”: “2015-11-30T10:15:26.000Z”
}
}
]

Thanks,
Aman

Hi,

Found data was not overridden but the corrdinator getting stuck somewhere.

I reduced number of dataSource then it discovered other ingested data.

Is there any limit for dataSource or we can have as many as we want.

Thanks,
Aman

Hi Aman, a common reason for things not loading is you need more capacity in your cluster. It should be fine to have thousands of datasources. In the millions, we aren’t sure, no one has ever gone that high, but maybe?

Coordinator logs will tell us why data was not being loaded.

Cool thanks FJ.