MsSQL to Druid

Hi,

Can I import data to druid from MSSQL Database on daily basis as my data storage is MSSQL but i want to use Druid for analytics

Yes you can.

Probably you need to export your data from MySQL to a CSV/TSV/JSON file and use Druid Native indexing methods to ingest them into Druid.

Thanks,
Vaibhav

Maybe you would like to go through below docs once you export your MySQL db tables as CSV/TSV/JSON or any other delimited format :

https://druid.apache.org/docs/latest/ingestion/data-formats.html
https://druid.apache.org/docs/latest/ingestion/native-batch.html

https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html

Thanks,

Vaibhav

But my project DB remains as SQL only . so converting to JSON daily and then importing isnt feasible. is there any other way to do this activity ?

One way is to stream MS SQL data through kafka and then use it as source for druid

https://medium.com/@adrianedbertluman/syncing-sql-server-database-using-kafka-part-2-running-kafka-connect-3ebc8234bfe

It seems you want to stream your MySQL data into Druid, In this case, you can use Streaming ingestion in druid (suck as Kafka indexing service ) where you will read your MySQL data [ Producer] through a KAFKA topic and druid can consume it through the same Kafka topic.
Something like below:

MySQL [Producer] —>KAFKA [ Topic] —>DRUID [Consumer]

https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html

Thanks,

Vaibhav

We have a similar problem. Our main database is different (prestodb) and we want to load one day’s worth of data in Druid everyday for analytics. Is it possible to setup a load rule “P1D” to load and keep only 1 days worth of data in Druid from main/backup database?

thanks Visakh and Vaibhav , I ll try to implement this. But is there any way in which if a transaction is made in sql that i can push to kafka at the same time.

kafka does live streaming itself. That is it streams data when transaction happens itself within the topic created
You can either use jdbc based connector to connect to table directly or use Debezium based connector if you need to process data using CDC service of MS SQL