My Bachelor degree thesis: Apache Hive and Apache Druid performance testing for MIND Foods HUB Data Lake

Hello guys!

First of all, thank you for the precious work that this wonderful community does every day to share and help people like me with Apache Druid.

Then, I want to share my Bachelor’s degree thesis work for the “Sicurezza dei Sistemi e delle Reti Informatiche” course, where I discuss the performance evaluation between Apache Hive and Apache Druid for the MIND Foods HUB Data Lake.

MIND Foods HUB is an interdisciplinary project to " implement a computational infrastructure to model, engineer and distribute data about plant phenotyping".

To store the phenotyping data to support various scientific analyses, MIND employs an Apache Hive instance, which sadly suffers from low maintainability and scarce aggregation performance.

My research goal was to test Apache Druid as a more performant alternative than Hive. For this purpose, I performance tested both platforms with Apache JMeter, following an approach similar to the one applied by Imply for the price-performance benchmark between Druid and Google BigQuery. Still, instead of relying on the Star Schema Benchmark, I preferred to design a workload representative of how the platform is used on the field, generating a dataset that sticks to the data model of the MIND Foods HUB table, using the actual SQL queries of the analysis processes.

If you are interested, here’s the GitHub repository with all the work that I did for the research:

https://github.com/SESARLab/mfh-dl-performace-testing

While here you can find the PDF of my thesis:

https://github.com/SESARLab/mfh-dl-performace-testing/blob/master/thesis.pdf

Tuesday will be the day of the discussion, so wish me luck, guys!

Thank you.

Gabriele

3 Likes

Best of luck @Gabriele_D_Arrigo! Let us know how it goes.

This topic was automatically closed after 3 minutes. New replies are no longer allowed.

Oh that’s awesome! Sending congratulations in advance. Thank you for sharing this with us all!