Hadoop merge segment, kafka index service throws [Cannot allocate new segment for dataSource]

druid 0.10.1

ubuntu 16.04

hadoop merge historical segment, segmentGranularity day to month

post kafka index service task throws

2018-01-31T07:31:08,321 WARN [qtp87784678-97] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Cannot allocate new segment for dataSource[svpn_users], interval[2018-01-01T00:00:00.000Z/2018-02-01T00:00:00.000Z], maxVersion[2018-01-31T07:31:08.285Z]: ShardSpec class[class io.druid.timeline.partition.NoneShardSpec] used by [svpn_users_2018-01-01T00:00:00.000Z_2018-02-01T00:00:00.000Z_2018-01-30T14:32:48.768Z].

After hadoop merge, the total number is smaller than the targetPartitionSize, and ShardSpec is set for.

NoneShardSpec, which causes the stream data to be unable to determine the partition number,

Therefore, SegmentIdentifier cannot be determined, and null is returned, thus causing.

The program Could not allocate segment for row with.Timestamp exception

i set “shardSpec”: {

“type”: “numbered”,

“partitionNum”: 0,

“partitions”: 0

}

but is not work.

this is my kis config file

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: “svpn_users”,

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “json”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

},

“dimensionsSpec”: {

“dimensions”: [

“user_id”,

“account”,

“client_ip”,

“ip_country”,

“ip_province”,

“ip_city”

]

}

}

},

“metricsSpec” : [

{ “type” : “count”, “name” : “views” },

{ “type” : “hyperUnique”, “name” : “user_total”, “fieldName” : “user_id” }

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “DAY”,

“queryGranularity”: “NONE”,

“rollup”: true

}

},

“ioConfig”: {

“topic”: “svpn_users”,

“consumerProperties”: {

“bootstrap.servers”: “node1:9092,node2:9092,node3:9092”

}

},

“tuningConfig”: {

“type”: “kafka”,

“forceExtendableShardSpecs”: true,

“forceGuaranteedRollup”: true,

“maxRowsPerSegment”: 5000000,

“shardSpec”: {

“type”: “numbered”,

“partitionNum”: 0,

“partitions”: 0

}

}

}

Try setting “forceExtendableShardSpecs” in your hadoop ingestion task spec file to create numbered shard spec instead of NoneShardSpec.
See http://druid.io/docs/latest/ingestion/batch-ingestion.html for more details.

in tuningConfig, i set up “forceExtendableShardSpecs”: true, but stil not work, task payload can’t see shardSpec

在 2018年2月1日星期四 UTC+8上午12:40:14,Nishant Bangarwa写道: