Druid Datasource Availability Issues

I am in a situation where my Druid Running Tasks become stuck and I think it is related to some of my Druid Datasources not being fulling available.

I observe my Druid running tasks issues here:

Druid Indexer Coordinator Console - Running Task View

http://<OVERLORD_IP>:/console.html

http://localhost:8090/console.html

I observe my Druid datasource availability issues here:

Druid Console:

http://<COORDINATOR_IP>:/#/

http://localhost:8081/#/

On the Druid Console, when I move my mouse over the associated datasource’s red circle (Red Circle - ≤ 99% available) with name “datasource1”, it says “datasource1: 24% to load until available”. Other datasources with the same red circle issue will say things like “datasource2: 48% to load until available” and “datasource3: 3.6% to load until available”. These numbers never update or change, even after refreshing the page.

How do I resolve my Druid datasource availability issues?

So figured out the issue and associated fix.

Issue: Seems I was also getting the following exception in the logs:

Mar 01 10:59:08 druid01 druid-historical-start.sh[23091]: FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.common.ISE: Segment[datasource1_2016-02-29T00:00:00.000Z_2016-02-29T01:00:00.000Z_2016-02-29T01:16:10.733Z:1,973,859] too large for storage[/opt/druid-data/indexCache:1,250,893].\n\tat io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:116)\n\tat io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93)\n\tat io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:153)\n\tat io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:142)\n\t… 18 more\n",“segment”:{“dataSource”:“datasource1”,“interval”:“2016-02-29T00:00:00.000Z/2016-02-29T01:00:00.000Z”,“version”:“2016-02-29T01:16:10.733Z”,“loadSpec”:{“type”:“hdfs”,“path”:“hdfs://druid01:8020/druid-hdfs-storage/datasource1/20160229T000000.000Z_20160229T010000.000Z/2016-02-29T01_16_10.733Z/0/index.zip”},“dimensions”:“dim1,dim2,dim3”,“metrics”:“count,metric1,metric2”,“shardSpec”:{“type”:“linear”,“partitionNum”:0},“binaryVersion”:9,“size”:1973859,“identifier”:“datasource1_2016-02-29T00:00:00.000Z_2016-02-29T01:00:00.000Z_2016-02-29T01:16:10.733Z”}}}]

``

Fix: Updating the Druid Historical Node Property “druid.segmentCache.locations” to a larger value.

config/historical/runtime.properties:

From:

druid.segmentCache.locations=[{“path”: “/opt/druid-data/indexCache”, “maxSize”: 10000000000}]

``

To:

druid.segmentCache.locations=[{“path”: “/opt/druid-data/indexCache”, “maxSize”: 50000000000}]

``