Issue in using insert-segment-to-db - Druid 0.13.0

Hi guys,

due to migration and other reasons, I lost my metadata storage and I tried to use the insert-segment-to-db tool to update the metadata store.

Before doing that I update the bucket name and key in the desciptor.json in S3 deepstorage (see below), as they’ve changed.

“loadSpec”: {

“type”: “s3_zip”,

“bucket”: “MY_NEW_BUCKET”,

“key”: “MY_NEW_FOLDER_PATH/MY_DATASOURCE/1980-01-06T00:00:00.000Z_1980-01-07T00:00:00.000Z/2018-12-15T00:17:24.100Z/0/index.zip”,

“S3Schema”: “s3n”

}

``

The issues is that I get an error that doesn’t let me rebuild the metadata storage:

2019-01-22T02:31:35,694 INFO [main] org.apache.druid.storage.s3.S3DataSegmentFinder - Updating loadSpec in descriptor.json at [MY_NEW_FOLDER_PATH/MY_DATASOURCE/1980-01-06T00:00:00.000Z_1980-01-07T00:00:00.000Z/2018-12-15T00:17:24.100Z/0/descriptor.json] with new path [S3Object [key=MY_NEW_FOLDER_PATH/MY_DATASOURCE/1980-01-06T00:00:00.000Z_1980-01-07T00:00:00.000Z/2018-12-15T00:17:24.100Z/0/descriptor.json,bucket=MY_NEW_BUCKET]]

Exception in thread “main” com.amazonaws.SdkClientException: Data read has a different length than the expected: dataLength=616; expectedLength=653; includeSkipped=false; in.getClass()=class com.amazonaws.internal.ReleasableInputStream; markedSupported=true; marked=0; resetSinceLastMarked=false; markCount=1; resetCount=0

at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)

at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:109)

at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)

at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:98)

at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)

at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)

at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)

at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:142)

at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160)

at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:158)

at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)

at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)

at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)

at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1235)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)

at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1720)

at org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.putObject(ServerSideEncryptingAmazonS3.java:109)

at org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.putObject(ServerSideEncryptingAmazonS3.java:104)

at org.apache.druid.storage.s3.S3DataSegmentFinder.findSegments(S3DataSegmentFinder.java:108)

at org.apache.druid.cli.InsertSegment.run(InsertSegment.java:96)

at org.apache.druid.cli.Main.main(Main.java:118)

``

Anyone knows what went wrong?

thank you!

UPDATE: I did execute the same insert-segment-to-db tool with Druid 0.12.3 and of course use io.druid instead of org.apache and the tool worked! So now the metadata storage is getting updated by the script!