Can I use S3 while in quickstart mode?

Hi,

I have successfully managed to setup Druid with Tranquility and

Kafka in local mode in a single node. Everything works fine with 500
requests per second (on average).

Now I would like to change the Druid storage from local machine to

S3 bucket. Would that be possible while in quickstart mode? If this
is possible, then my problem is that Druid keeps storing the data on
the local machine as opposed to the S3 bucket although I have made
the required changes in file
./imply-2.0.0/conf-quickstart/druid/_common/common.runtime.properties?
What can be wrong on my side? How can I make sure that my settings
are not ignored?

Yes, it’s possible. Make sure you have added “druid-s3-extensions” to your loadList first.

Thank you.

I think I found the solution to my problem. First of all, I
confirm that I can write to S3 while in local mode. My problem was
that I couldn’t see any data in the S3 bucket - although I had set
everything up as well as there was data coming in to folder
./var/druid/task/.

So the reason was probably that the real-time nodes had a
relative big size through the maxRowsInMemory variable in
the corresponding JSON file of data schemas.

Can you please confirm me if that was indeed the case? What is a
good value for maxRowsInMemory provided that I get ~1K
events per second?

Got it. Publishing does only happen periodically, as a matter of design, about once every segmentGranularity. So you may not see data written to S3 for an hour, if you’re using “hour” segmentGranularity.