Ingesting data using Tranquility not working

I’m trying out Tranquility for the first time and I’m not having much luck. I have reverted my setup to running everything locally on my machine. I downloaded the 0.7.1.1 binary, untarred it, and started up zk, mysql, overlord, coordinator, historical, and the broker with no changes to the configs.

Now I have a Java client using Tranquility 0.4.2 and have basically copied the first java example in the README of the Tranquility page. I have a class that sets up the druidService and saves that reference for later. Then I continually send an event to druidService.apply() and I’m using the Future to make sure it gets acknowledged. I don’t call close() on the service/curator until I shut down the app. The client code is here if you want to look at it: https://github.com/acesinc/json-data-generator/blob/master/src/main/java/net/acesinc/data/json/generator/log/TranquilityLogger.java

When I start clean (like the database and everything are clean), and I start my app, it will send events to Druid and I see druid start a Task. My app gets acknowledgement of sending events and keeps sending them. The events being sent have a timestamp of “now.” So it seems my events are being sent to druid, but I don’t see the Task doing anything in the Task log. And the datasource never shows up. I’ve put the task log, overlord log, and coordinator log here in this gist: https://gist.github.com/andrewserff/da349ed0abbba4f773eb

I feel like it is just a configuration issue in the Tranquility task builder, but I can’t figure out what is wrong.

Also, if I shutdown all the services and restart everything and try to start sending data again, it won’t work any more. The client sits there for a long time waiting for an ack, and druid never starts a task. However, it eventually does ask the sending of the data which is weird. And it will start letting it send more data, but each time, the ack takes like 2 min. And no task ever gets started. In order to get it to work again, I have to remove all the druid/tranquility entries in zookeeper…

Let me know what to try!

Thanks

Andrew

Hi Andrew,

A few questions:

Have you set UTC timezone on every service?

Do you have metrics emission turned on? If not, it would be good to turn that on and see what the logs say for the number of events accepted, rejected, etc.

Do you have the tranquility logs?

Your task json has more than one time zone (UTC and UTC-6) and that makes me wonder if this is a time zone related problem. The Druid task might be clipping your data if some time zone information got lost in translation between tranquility and Druid. Can you try running everything in the same time zone (UTC is preferred) and see if that helps?

Generally you can do this with -Duser.timezone=UTC on the java command line or by setting the environment variable TZ=UTC.