I have successfully managed to setup Druid with Tranquility and
Kafka in local mode in a single node. Everything works fine with 500
requests per second (on average).
Now I would like to change the Druid storage from local machine to
S3 bucket. Would that be possible while in quickstart mode? If this
is possible, then my problem is that Druid keeps storing the data on
the local machine as opposed to the S3 bucket although I have made
the required changes in file
What can be wrong on my side? How can I make sure that my settings
are not ignored?
Yes, it’s possible. Make sure you have added “druid-s3-extensions” to your loadList first.
I think I found the solution to my problem. First of all, I
confirm that I can write to S3 while in local mode. My problem was
that I couldn’t see any data in the S3 bucket - although I had set
everything up as well as there was data coming in to folder
So the reason was probably that the real-time nodes had a
relative big size through the
maxRowsInMemory variable in
the corresponding JSON file of data schemas.
Can you please confirm me if that was indeed the case? What is a
good value for
maxRowsInMemory provided that I get ~1K
events per second?
Got it. Publishing does only happen periodically, as a matter of design, about once every segmentGranularity. So you may not see data written to S3 for an hour, if you’re using “hour” segmentGranularity.