So here’s a quick starter for 10 checklist of what to do if you have issues with Kafka ingestion!
Why not add your own suggestions! Please comment below…!
What’s in the supervisor log?
What’s in each of the task logs?
- Can you ping the bootstrap IP address from the servers?
- Can you ping the bootstrap by hostname?
- Can you telnet to the bootstrap by hostname : port?
- What about kafkacat -b server -L
- Can you kafkacat -vvv -C -b (bootstrap) -t (topic) -c 10?
- Is DNS VPC Peering enabled in both directions? (AWS Only)
- Do the machines have DNS names? - consider DHCP option set
- DHCP options sets - Amazon Virtual Private Cloud
- When using Azure Event Hub, check out azure-event-hubs-for-kafka/quickstart/kafkacat at master · Azure/azure-event-hubs-for-kafka · GitHub to configure kafkacat appropriately.