[druid-user] Doubts about "OneTime" Ingestion Task

Dear Druid devs,

I am a user of Apache Druid. I already did the “Droid Basics” course provided by Imply. I am using Druid within a research project for boosting and powering our AI algorithms. In the context of this project we are also developing a dynamic pipeline engine. We have an external client that needs to access ingestion specifications and change them if something happens in the pipeline. However, I had a doubt we are using Kafka for streaming ingestion and we are also ingesting data from minIO. In the latter case, as soon as I ingest data a new ingestion task is created. After the task has been successfully executed after a time it disappears from the UI and I really do not know how to retrieve the specification task by using the API. Could you please provide any suggestion/help about this?

Cheers,

Giovanni Di Orio

Hi Giovanni,

Here’s the task APIs doc.

If you’d like another resource to explore, my colleague @Sergio_Ferragut wrote a great blogpost on clustered deployment with k8s and min.io.

Best,

Mark

Hi Giovanni,
Have you configured logging storage? You can store the task logs in S3, for example.

Let us know how it goes,

Sergio

Thank you Mark,

I already passed through the task API methods and my problem is still there… I will try to store the task logs in S3 like @Sergi_Ferragut suggested.

Cheers,

Giovanni

A segunda-feira, 14 de março de 2022 à(s) 15:27:11 UTC, mark.h...@imply.io escreveu: