Invalid Availability zone error when trying to autoscaling

Hello!

I am trying to configure Autoscaling for Middlemanagers, but I have problems when Overlord tries to provision a new EC2 instance. I put error log below:

2015-08-05T12:26:52,571 INFO [ScalingExec–0] io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy - Our target is 1 workers, and I’m okay with that (current = 0, min = 1, max = 10).

2015-08-05T12:26:52,682 ERROR [ScalingExec–0] io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler - Unable to provision any EC2 instances.

com.amazonaws.AmazonServiceException: Invalid availability zone: [eu-west-1b] (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 5a91beae-b385-4399-8cf3-d6eb87572036)

at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:4382) ~[druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler.provision(EC2AutoScaler.java:110) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.doProvision(SimpleResourceManagementStrategy.java:106) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at io.druid.indexing.overlord.autoscaling.ResourceManagementScheduler$1.run(ResourceManagementScheduler.java:83) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.metamx.common.concurrent.ScheduledExecutors$3.call(ScheduledExecutors.java:140) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.metamx.common.concurrent.ScheduledExecutors$3.call(ScheduledExecutors.java:136) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at com.metamx.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:173) [druid-services-0.7.1.1-selfcontained.jar:0.7.1.1]

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_03]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_03]

at java.lang.Thread.run(Unknown Source) [?:1.7.0_03]

This is the JSON that I am passing to Overlord with curl:

{

“selectStrategy”: {

"type": "equalDistribution"

},

“autoScaler”: {

"type": "ec2",

"minNumWorkers": 1,

"maxNumWorkers": 10,

"envConfig": {

  "availabilityZone": "eu-west-1b",

  "nodeData": {

    "amiId": "ami-fe5f0e89",

    "instanceType": "r3.4xlarge",

    "minInstances": 1,

    "maxInstances": 10,

    "securityGroupIds": [

      "sg-7eacb71b"

    ],

    "keyName": "arodriguez",

    "subnetId": "i-35905198",

    "associatePublicIpAddress": true

  },

  "userData": {

    "impl": "string",

    "data":   "XXXXXXX",

   "versionReplacementString": ":VERSION:",

   "version": 1

  }

}

}

}

In S3 configuration there are AWS credentials with sufficient permissions, but I think that there are some problems with regions.

I hope that you can help me. Thanks!

Looking through io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler#provision and io.druid.guice.AWSModule#getEc2Client it doesn’t look like the region or endpoint is set for the client.

So as per com.amazonaws.services.ec2.AmazonEC2Client#init it defaults to ec2.us-east-1.amazonaws.com

The API should be using the envConfig to select the availability zone. What version of Druid is this? I know in older versions, this problem existed for some non US availability zones.

Druid version is 0.7.7.1. We are sending availability zone to API, but I think that the problem is that Druid is trying to connect to US endpoint of Amazon and can’t found availability zone from EU. I have been looking for region configuration in druid doc and github but I haven’t found nothing.

Now I am trying with version 0.8 and I have same error. Please, could you indicate me how to change AWS API endpoint for my region (eu-west-1)? Thanks!

Hi,

This is a bug with the current autoscaling code in that the region should be passed in the envConfig and set in the AutoScaler. Do you mind submitting a PR with this fix?

http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-region-selection.html

Thanks,

FJ

Hey Fangjin,

Did anybody raised bug for this and is it covered in 0.9 release ?

Thanks

Chetan

Not that I’m aware of.

Hi I still cannot set the availability zone for ec2 client , were you able to fix it?