No data in S3

Hey all,

For some reason no data is getting pushed to S3. I have this working in another environment but when i moved my druid cluster to a more permanent environment my s3 bucket is still empty. Realtime nodes arent showing logs about segment at all either. This is my common.runtime.properties:

user.timezone=UTC

file.encoding=UTF-8

java.io.tmpdir=/data

# Extensions

druid.extensions.coordinates=[“io.druid.extensions:mysql-metadata-storage”, “io.druid.extensions:druid-s3-extensions:0.8.0”, “io.druid.extensions:druid-kafka-eight:0.8.0”]

# Zookeeper

druid.zk.service.host={zk}

druid.zk.paths.base={zk-paths}

druid.discovery.curator.path=/prod/discovery

# Request logging, monitoring, and metrics

druid.request.logging.type=emitter

druid.request.logging.feed=druid_requests

druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.emitter=noop

# Metadata storage

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI={db-connectURI}

druid.metadata.storage.connector.user={db-user}

druid.metadata.storage.connector.password={db-password}

# Deep storage

druid.storage.type=s3

druid.storage.bucket={bucket}

druid.s3.accessKey={access}

druid.s3.secretKey={secret}

The s3 extensions are getting pulled so I’m really not sure why theres still no data. Any help is appreciated.

Thanks,

any ideas?

this is also my specfile attached

my.spec (1.61 KB)

Hey Nicholas,

Do you know if that realtime node is actually ingesting data? If you don’t see any logs about it building and merging segments, it may not actually be ingesting anything. You could also add the RealtimeMetricsMonitor and set the emitter to “logging” in order to see specific metrics about how many events are being processed.

Another thing to try is that you currently have your rejectionPolicy set to “messageTime”. This can cause things to not be handed off properly if you don’t have a continuous stream of data. You may have better luck with “serverTime”.

ill change it to serverTime and and turn the realtimemetrics on and post back here.

Thanks,

Hey Gian,

So messageTime works perfectly at the moment. I do have a continuous stream of data coming in so that helps. I can also see a bunch of data in s3. Thanks

But when i use serverTime, everything gets thrown out. i now have my windowPeriod at 10m (default). can you elaborate more on this? The data coming in is at real time and the timestamps for that data is current time.

And what does “hand-off” really mean? is that just segment persistence?

Thanks for your help dude.

Hey Nicholas,

“handoff” means uploading a segment to deep storage, inserting a row for that segment in the metadata store, and waiting for a historical node to load that segment.

You shouldn’t be getting any thrownAway counts if all your timestamps are within your windowPeriod. If your data is being entirely thrown out then perhaps either the server clock is wrong, or something is off in how the times are being parsed (mismatched timezone maybe).