Reindexing interval smaller than segment granularity

Apache Druid Q&A Google Groups
#1

Hi,

Have some questions about updating and replacing historical data in druid.

In my system I have segmentGranualirty set to WEEK

  1. How to replace/reindex DAY interval data in my historical data?
  2. How to append DAY interval data to my historical data?

Now when I do reindex or delta task for day interval rest of the WEEK size segemnt became empty.

As for the delta task I found some not very optimal solution - merge with bigger dataSource interval covering full segment. But what about replace/reindex task?

P.S One more problem.

In my experiments I tried to run index task with such params to append data to my histoical data

“ioConfig” : {

“type” : “index”,

“firehose” : {

“type” : “local”,

“baseDir”: “/var/stats/”,

“filter” : “sample.json”

},
“appendToExisting” : true

},

“tuningConfig” : {

“type” : “index”,

“targetPartitionSize” : 5000000,

“maxRowsInMemory” : 75000,
“forceExtendableShardSpecs” : true

}

``

but got such error

2017-08-24T17:49:36,841 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - Cannot allocate segment for timestamp[2017-01-09T21:37:24.615Z], sequenceName[index_2017-01-09T00:00:00.000Z/2017-01-16T00:00:00.000Z_2017-08-24T17:49:36.760Z_0].
2017-08-24T17:49:36,861 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_myDataSource_2017-08-24T17:49:31.945Z, type=index, dataSource=myDataSource}]
io.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2017-01-09T21:37:24.615Z]
	at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:438) ~[druid-indexing-service-0.10.0-iap4.jar:0.10.0-iap4]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:207) ~[druid-indexing-service-0.10.0-iap4.jar:0.10.0-iap4]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.0-iap4.jar:0.10.0-iap4]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.0-iap4.jar:0.10.0-iap4]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

``

What am I doing wrong?

Thanks in advance :slight_smile: