Any Java, Scala Example of Batch Ingestion with Tranquility?

Hello everyone,

I already downloaded the source code still didn’t understand how to write a code to move data from source to Druid I mean the history data not a realtime like a migration.

I read in the
package com.metamx.tranquility.example ScalaExample

``

is about the realtime one.

I add into
example.json

``

a intervals but it doesn’t work that way.

“granularitySpec”: {
“type”: “uniform”,
“segmentGranularity”: “DAY”,
“queryGranularity”: “NONE”,
** ****“intervals” : [ “2009-01-01/2013-09-01” ]**}

``

I read the document say use package com.metamx.tranquility.beam Beam Class but It’s a lot child class and seem not match what I want to do like Scala Example.

So I hope someone can give some example about that because it’s important process to migrate from another system to Druid.

Thanks.

Hi Chanh, these docs may be helpful to you:
https://imply.io/docs/latest/ingestion

The go into detail about how to load historical data. The Druid docs are based from the Imply docs.

Hi FJ,
Just confirm that you mean about ingest historical data right now this is the only way to do except the ingest from files?

Stream pull with the Kafka indexing service - Starting in IAP 1.3.0 / Druid 0.9.1, IAP and Druid include an experimental Druid extension offering exactly-once ingestion from Kafka, as well as the ability to ingest historical data.

Do we have any an API to ingest directly into Druid? Or we keep kafka as a interface to push data into Kafka.

Regards,

Chanh

Hi Chanh,

The standard way to ingest historical data into Druid is from files using the Hadoop indexing task (which you can use without a separate Hadoop cluster but it will take longer). The Kafka indexing service will also allow you to ingest historical data but you’ll need to set up one or more Kafka brokers to use it and then read your data into Kafka.

The other methods (all Tranquility-based methods and real-time nodes) are only designed for real-time data and won’t accept older events.

Thanks.
Chanh