Coordinator node not balancing segments amount history node

Hi,

I had a cluster setup and working( i thought so). There are few things I observed when it went down. I tried loading one day of data
from kafka streaming.

  1. disk of 1 of 2 historical node got full and
    it stopped responding. I had two nodes running with same configuration file. I queried other historical node but it had only 1 hour of data but
    it broker node and coordinator node were able to identify this node. Why load was not getting transferred to this node if other node was down?
  2. I restarted the historical node moving the indexCache to a larger disk and it reloaded the data properly. I am able to query it also. But in the coordinator node I am getting this warning :
    2016-02-03T00:55:40,654 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:25:00.000Z_2015-01-01T04:30:00.000Z_2015-01-01T04:25:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,654 DEBUG [pool-655220-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,654 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:20:00.000Z_2015-01-01T04:25:00.000Z_2015-01-01T04:20:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,655 DEBUG [pool-655221-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,655 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:15:00.000Z_2015-01-01T04:20:00.000Z_2015-01-01T04:15:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,655 DEBUG [pool-655222-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,655 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:10:00.000Z_2015-01-01T04:15:00.000Z_2015-01-01T04:10:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,655 DEBUG [pool-655223-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,655 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:05:00.000Z_2015-01-01T04:10:00.000Z_2015-01-01T04:05:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,656 DEBUG [pool-655224-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,656 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T04:00:00.000Z_2015-01-01T04:05:00.000Z_2015-01-01T04:00:00.000Z]! Expected Replicants[2]
    2016-02-03T00:55:40,656 DEBUG [pool-655225-thread-1] io.druid.server.coordinator.ServerHolder - Server[10.116.123.50:8083], MaxSize[10,000,000,000], CurrSize[17,307,674,149], QueueSize[0], SizeUsed[17,307,674,149], AvailableSize[-7,307,674,149]
    2016-02-03T00:55:40,656 WARN [Coordinator-Exec–0] io.druid.server.coordinator.rules.LoadRule - Not enough [_default_tier] servers or node capacity to assign segment[avping_2015-01-01T03:55:00.000Z_2015-01-01T04:00:00.000Z_2015-01-01T03:55:00.000Z]! Expected Replicants[2]

``

While
both nodes are running why it is not able to load segments in one history node which is up and running and also can be queried. this history node has data from 00.00 to 01.40 for the same day. Even after restarting this history node it loaded only this data. What am I doing wrong?

History node (with less data)
[ {
“timestamp” : “2015-01-01T00:00:00.000Z”,
“result” : {
“minTime” : “2015-01-01T00:00:00.000Z”,
“maxTime” : “2015-01-01T01:39:59.000Z”
}
} ]

``

History node 2(with more data)
[ {
“timestamp” : “2015-01-01T00:00:00.000Z”,
“result” : {
“minTime” : “2015-01-01T00:00:00.000Z”,
“maxTime” : “2015-01-01T09:39:59.000Z”
}
} ]

``

Let me know if I should post more details.

Resolved. The capacity of history node was reached as given in the historical.runtime.properties. Increased the size and restarted.

Thanks