realtime node not pushing data to s3 but staying on realtime server

Hi,

My realtime node is not pushing data to s3. using druid 8.1. My s3 buckets are empty. But the data is remaining on the realtime server.

Here are my files. So realtime has access to mysql and s3. No issues with zookeeper. When I query the realtime node all is well but not doing what it supposed to do which is push data to s3 so I can query with historical node.

Thanks

runtime.properties

druid.host=<%=@ipaddress%>:<%=@druid_port%>

druid.service=realtime

druid.port=<%=@druid_port%>

druid.zk.service.host=<%=@zookeeper%>

druid.extensions.coordinates=[“io.druid.extensions:druid-kafka-eight:<%=@version%>”,“io.druid.extensions:mysql-metadata-storage:<%=@version%>”,“io.druid.extensions:druid-s3-extensions:<%=@version%>”]

These configs are only required for real hand off

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://<%=@mysql_host%>:3306/<%=@mysql_database%>

druid.metadata.storage.connector.user=<%=@mysql_username%>

druid.metadata.storage.connector.password=<%=@mysql_password%>

druid.processing.buffer.sizeBytes=100000000

druid.processing.numThreads=1

druid.monitoring.monitors=[“io.druid.segment.realtime.RealtimeMetricsMonitor”]

druid.s3.secretKey=<%=@AWS_SECRET_ACCESS_KEY%>

druid.s3.accessKey=<%=@AWS_ACCESS_KEY_ID%>

druid.storage.bucket=<%=@s3bucket%>

druid.storage.baseKey=<%=@s3basekey%>

druid.storage.disableAcl=true

druid.realtime.specFile=/var/realtime.spec

realtime.spec

[{

“dataSchema” : {

“dataSource” : “datasource-topic-impression”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column”: “utcdt”,

“format”: “iso”

},

“dimensionsSpec” : {

“dimensions”: ,

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec” : [{

“type” : “count”,

“name” : “impression”

}],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “minute”,

“queryGranularity” : “NONE”

}

},

“ioConfig” : {

“type” : “realtime”,

“firehose”: {

“type”: “kafka-0.8”,

“consumerProps”: {

“zookeeper.connect”: “<%=@zookeeper%>”,

“zookeeper.connection.timeout.ms” : “15000”,

“zookeeper.session.timeout.ms” : “15000”,

“zookeeper.sync.time.ms” : “5000”,

“group.id”: “topic-impression-<%=@environment%>”,

“fetch.message.max.bytes” : “1048586”,

“auto.offset.reset”: “largest”,

“auto.commit.enable”: “false”

},

“feed”: “topic-impression-<%=@environment%>”

},

“plumber”: {

“type”: “realtime”

}

},

“tuningConfig”: {

“shardSpec”: {

“type”: “linear”,

“partitionNum”: 1

},

“type” : “realtime”,

“maxRowsInMemory”: 500000,

“intermediatePersistPeriod”: “PT10m”,

“windowPeriod”: “PT10m”,

“basePersistDirectory”: “/tmp/realtime/basePersist”,

“rejectionPolicy”: {

“type”: “serverTime”

}

}

}]

Hi David,
Can you share the logs for more info ?
do you see any exceptions in the logs or any messages related to pushing of segments ?

e.g. “Found [%,d] sinks to persist and merge”

i didn’t see “druid.storage.type=s3” in the properties.

在 2015年10月12日星期一 UTC+8下午4:01:49,David Montgomery写道:

I think posting the common configurations will help diagnose the issue as well. Make sure to include the S3 extension and set the deep storage type of S3.

Hey,

I hit the same problem, changing druid.storage.type isn’t enough, one must also add io.druid.extensions:druid-s3-extensions extension.

The next problem is that when I change :

druid.extensions.coordinates=[“io.druid.extensions:druid-histogram”, “io.druid.extensions:druid-datasketches”]

to :

druid.extensions.coordinates=[“io.druid.extensions:druid-histogram”, “io.druid.extensions:druid-datasketches”, “io.druid.extensions:druid-s3-extensions”]

Then strangely all components fail due to :

org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.druid.extensions:druid-datasketches:jar:0.8.3-iap1

but If I leave it it like this :

druid.extensions.coordinates=[“io.druid.extensions:druid-histogram”, “io.druid.extensions:druid-datasketches”]

all components start up fine…

I’m using quick-start setup from imply/imply docker container, I’m just change properties