Monitoring Kafka indexing service data being loaded

Hi,

we’re testing Kafka indexing service.

I wonder whether it’s possible to monitor automatically

specific datasource whether it’s being loaded into the Druid.

E.g. For each resource in a form of

/druid/indexer/v1/supervisor/

is it possible to see how many jobs succeeded/failed

since the start?

Vlad

Hi Vlad,

To see the number of jobs that have succeeded and failed, you would use the overlord console (by default http://{OVERLORD_IP}:8090). You can adjust the druid.indexer.storage.recentlyFinishedThreshold parameter if you want to retain a longer history.

The Kafka indexing service has a status endpoint that’s helpful for seeing the current state of ingestion (number of tasks reading/publishing, current offsets, and remaining time) but doesn’t show whether tasks succeed or fail (since that’s handled above). This is available at the endpoint: GET http://{OVERLORD_IP}:8090/druid/indexer/v1/supervisor//status

Dne úterý 26. července 2016 19:00:25 UTC+2 David Lim napsal(a):

Hi Vlad,

To see the number of jobs that have succeeded and failed, you would use the overlord console (by default http://{OVERLORD_IP}:8090). You can adjust the druid.indexer.storage.recentlyFinishedThreshold parameter if you want to retain a longer history.

The Kafka indexing service has a status endpoint that’s helpful for seeing the current state of ingestion (number of tasks reading/publishing, current offsets, and remaining time) but doesn’t show whether tasks succeed or fail (since that’s handled above). This is available at the endpoint: GET http://{OVERLORD_IP}:8090/druid/indexer/v1/supervisor//status

Ok, thanks. I can see console.html calls druid/indexer/v1/completeTasks and runningTasks, which is usable for some kind of monitoring.