Kafka indexing failing

Hi there,

I have a newly deployed instance of imply-2.4.9 standalone operational. Just recently I am not able to install segments via the kafka indexing service. Receiving from kafka used to work just fine.

I have a significant more events being sent to the kafka topic so not sure if that is the cause. I have a single dataSource. I updated my dimensionsSpec and POST’ed it to the supervisor. Clicked the reset link on the dataSource line. I restarted the druid processes via the druid quickstart.

Below are the coordinator logs - please provide ideas about other logs to check:

2018-04-01T20:09:36,738 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [19] available segments.

2018-04-01T20:09:36,738 INFO [Coordinator-Exec–0] io.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.

2018-04-01T20:09:36,739 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorBalancer - [_default_tier]: One or fewer servers found. Cannot balance.

2018-04-01T20:09:36,739 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - [_default_tier] : Assigned 0 segments among 1 servers

2018-04-01T20:09:36,739 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - [_default_tier] : Dropped 0 segments among 1 servers

2018-04-01T20:09:36,739 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2018-04-01T20:09:36,739 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[<hostname_removed>:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 1,901,800,611 bytes served.

2018-04-01T20:09:36,752 INFO [DatabaseSegmentManager-Exec–0] io.druid.metadata.SQLMetadataSegmentManager - Polled and found 19 segments in the database

2018-04-01T20:09:38,601 INFO [LookupCoordinatorManager–1] io.druid.server.lookup.cache.LookupCoordinatorManager - Not updating lookups because no data exists

2018-04-01T20:09:41,740 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting available segments.

2018-04-01T20:09:41,740 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [19] available segments.

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorBalancer - [_default_tier]: One or fewer servers found. Cannot balance.

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - [_default_tier] : Assigned 0 segments among 1 servers

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - [_default_tier] : Dropped 0 segments among 1 servers

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2018-04-01T20:09:41,741 INFO [Coordinator-Exec–0] io.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[<hostname_removed>:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 1,901,800,611 bytes served.

Cheers

What’s druid.server.maxSize on your single historical node? I wonder if the historical is out of space for segments.