Trying to ingest a table created in K-SQL based on a kafka topic. When I check from K-SQL I can see the data. But while trying to extract the data to druid by creating datasource through data loader, it is not able to get the data. I can see topic listed in K-SQL as well. The data is coming in JSON format in the K-SQL table.
What is going wrong exactly? When you say “not able to get the data” what do you mean?
Assuming the KSQL query is writing to a normal Kafka topic (I think this is the case) you should be able to load the result into Druid.
In addition to Gian’s question, is it possible to verify by reading the data from your kafka topic using CLI tool Kafka-console-consumer?
If your broker knows about your topic, you should be able to get the data using Kafka-console-consumer.
As per this doc – https://docs.confluent.io/current/ksql/docs/index.html
“KSQL servers, clients, queries, and applications run outside of Kafka brokers, in separate JVM instances, or in separate clusters entirely.”
I am suspecting that your broker (which you mentioned in your druid kafka ingestion spec) might not be aware of the topic you are referring to.
One way to verify is using Kafka-console-consumer.
Thanks for the response
I created a table in K-SQL using Kafka topic to flatten out the data (the original Kafka topic had data coming within nested JSON)
The table has data in flattened format and value format is set as JSON
After creation, I can query and check the data in K-SQL and its showing correct data
I also checked topics from K-SQL and can see the corresponding Kafka topic created by K-SQL table
But when I try to use this topic inside druid to create the data source (using data loader) its not showing any data in the preview. Its just showing a message that no data returned by sampler
Any ideas on how to get this issue resolved?
I would go with Siva’s suggestion of running the kafka console consumer first to see if you can get the data in the consumer. The sampling error often indicates that the connection to kafka has not been made.