Failed to submit supervisor: Could not resolve type id 'kafka' into a subtype of [simple type, class org.apache.druid.indexing.overlord.supervisor.SupervisorSpec]: known type ids = [NoopSupervisorSpec, SupervisorSpec] at [Source: HttpInputOverHTTP@70

Hello,

I’m trying to run a sample supervisor task in druid router UI but came across this error. I tried to check the schema but couldn’t find anything useful. Any ideas on why this might have occurred?

Error:

Failed to submit supervisor: Could not resolve type id ‘kafka’ into a subtype of [simple type, class org.apache.druid.indexing.overlord.supervisor.SupervisorSpec]: known type ids = [NoopSupervisorSpec, SupervisorSpec] at [Source: HttpInputOverHTTP@706c5335[c=770,q=0,[0]=null,s=STREAM]; line: 1, column: 2]

Spec:

{
“type”: “kafka” ,
“dataSchema” : {
“dataSource”: “metrics-kafka” ,
“parser” : {
“type”: “string” ,
“parseSpec” : {
“format”: “json” ,
“timestampSpec” : {
“column”: “timestamp” ,
“format”: “auto”
},
“dimensionsSpec” : {
“dimensions” : ,
“dimensionExclusions” : [
“timestamp” ,
“value”
]
}
}
},
“metricsSpec” : [
{
“name”: “count” ,
“type”: “count”
},
{
“name”: “value_sum” ,
“fieldName”: “value” ,
“type”: “doubleSum”
},
{
“name”: “value_min” ,
“fieldName”: “value” ,
“type”: “doubleMin”
},
{
“name”: “value_max” ,
“fieldName”: “value” ,
“type”: “doubleMax”
}
],
“granularitySpec” : {
“type”: “uniform” ,
“segmentGranularity”: “HOUR” ,
“queryGranularity”: “NONE”
}
},
“tuningConfig” : {
“type”: “kafka” ,
“maxRowsPerSegment”: 5000000
},
“ioConfig” : {
“topic”: “metrics” ,
“consumerProperties” : {
“bootstrap.servers”: “localhost:9092”
},
“taskCount”: 1 ,
“replicas”: 1 ,
“taskDuration”: “PT1H”
}
}

Thanks!

Hi Naveen,

Please add “druid-kafka-indexing-service” extension in you common.runtime.properties file. That should resolve the issue.

Regards,

Viinay

I verified the props list.

druid.extensions.loadList=[“mysql-metadata-storage”, “druid-kafka-indexing-service”, “druid-kafka-eight”, “druid-distinctcount”]

Hi Naveen,
By any chance , are you using Quickstart to start your druid services but you are modifying common.runtime.properties of conf but not conf-quickstart?

This is common mistake people overlook when they run single node clusters.

Thank you.

–siva

Hey,

I have set up a multi-node cluster and i have custom common runtime props.

I have tried using coordinator and overlord together in a single service and i think that’s what is messing up but I’m not sure if that’s the root cause of this error.

Hi Naveen,
1 - Any chance to run coordinator and overlord separately to clarify the doubt?

2 - Is your task log showing druid-kafka-indexing-service as loaded?

3 - How are you submitting your kafka ingestion job? Can you give your curl command?

Thanks,

–siva