We are trying to use Druid for implementing a real time data pipeline.
While using the tranquility api … we have certain questions about dropping messages :-
This file has been truncated.
* Licensed to Metamarkets Group Inc. (Metamarkets) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. Metamarkets licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
In this since the sender is sending messages asynchronously to druid , What are the guarantees for the call :-
*FutureEventListener’s OnSuccess method . *
Does this mean the tranquility client successfully dispatched the message to Druid and then druids realtime server process will take care of actual ingestion ?
sender.flush() and sender.stop() could there be a window of time in between these 2 where on client side message can be lost ?
More so what does that sender.flush() actually achieve.
For a steady state operation( no fault taulerence scenarios) , could there be a scenario where the realtime node can drop a message successfully sent from tranquility client .
Thanks a bunch !!
Actually looked at the code a little closer .
#1 Guarantee is give by the twitter api so not an issue now .
#2 We still need help with this .
Will appreciate some help to get this off the ground .
sender.flush() will make sure all the events we have sent via calling sender.send() are processed and callbacks for either onSuccess or onFailure are called.
Since there will be no more events to be processed in above example between flush and stop. Its safe to call a stop after flush.