First Stop - Kafka connectivity checks

So here’s a quick starter for 10 checklist of what to do if you have issues with Kafka ingestion!

Why not add your own suggestions! Please comment below…!

General

What’s in the supervisor log?
What’s in each of the task logs?

Network

  1. Can you ping the bootstrap IP address from the servers?
  2. Can you ping the bootstrap by hostname?
  3. Can you telnet to the bootstrap by hostname : port?
  4. What about kafkacat -b server -L
  5. Can you kafkacat -vvv -C -b (bootstrap) -t (topic) -c 10?

AWS

  1. Is DNS VPC Peering enabled in both directions? (AWS Only)
  2. Do the machines have DNS names? - consider DHCP option set
  3. DHCP options sets - Amazon Virtual Private Cloud

Azure

  1. When using Azure Event Hub, check out azure-event-hubs-for-kafka/quickstart/kafkacat at master · Azure/azure-event-hubs-for-kafka · GitHub to configure kafkacat appropriately.

@Rich_Dobson / @Vaibhav / @Vijay_Narayanan1 / @Ben_Krug / @Hellmar_Becker can you think of anything else it’s useful to put in here? (Thanks @Mike_McLaughlin for the content in the first place…!!)