[elasticsearch] CDK seems not to handle multi-az domains correctly

[Jon-AtAWS]

Context: #10965

Short version: setting 3 AZs for an Amazon ES multi-AZ domain doesn't work with the default aws_vpc, which seems to deploy in 2 zones and I can't find a way to push to 3 zones.

Amazon ES domains can deploy into 1, 2, or 3 AZs, controlled by customer config. Obviously, the VPC must have that many AZs/subnets to support this configuration. See the referenced issue for my struggle with subnets and sending the correct data type. As part of that process, I expanded to 3 AZs. CDK diff did not report a problem, but cdk deploy failed when the subnet count was only 2.

                            zone_awareness=es.ZoneAwarenessConfig(enabled=True,
                                                                  availability_zone_count=3),

Yields this errror:

(.env) handler@laptop:~/code/cdk-vpc $ cdk deploy
jsii.errors.JavaScriptError:
  Error: When providing vpc options you need to provide a subnet for each AZ you are using
      at new Domain (/private/var/folders/4f/2b3kckld2mn59m48yyp5h_2hjr9cf9/T/jsii-kernel-lOc9Ud/node_modules/@aws-cdk/aws-elasticsearch/lib/domain.js:465:19)
      at /Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7934:49
      at Kernel._wrapSandboxCode (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8422:19)
      at Kernel._create (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7934:26)
      at Kernel.create (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7678:21)
      at KernelHost.processRequest (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7458:28)
      at KernelHost.run (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7396:14)
      at Immediate._onImmediate (/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7399:37)
      at processImmediate (internal/timers.js:461:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 9, in <module>
    CdkVpcStack(app, "cdk-vpc")
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_runtime.py", line 69, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/Users/handler/code/cdk-vpc/cdk_vpc/cdk_vpc_stack.py", line 26, in __init__
    domain = es.Domain(self, 'cdkd1',
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_runtime.py", line 69, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/aws_cdk/aws_elasticsearch/__init__.py", line 4478, in __init__
    jsii.create(Domain, self, [scope, id, props])
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 250, in create
    response = self.provider.create(
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 336, in create
    return self._process.send(request, CreateResponse)
  File "/Users/handler/code/cdk-vpc/.env/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 321, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: When providing vpc options you need to provide a subnet for each AZ you are using

So, I also added

        vpc = ec2.Vpc(self, 'cdkvpc', max_azs=3)

But get the same error.

Printing vpc.private_subnets, shows I have 2 (why not 3? I'm deploying in us-west-2). So I changed my node count and zone count to 2 and that worked.

As far as I can tell, there's no way to span 3 zones with a VPC, which mismatches the Amazon Elasticsearch Service best practice, 3-zone deployment. Or is there a less-obvious way to have the vpc in 3 zones?

Reproduction Steps

from aws_cdk import (
    aws_ec2 as ec2,
    aws_elasticsearch as es,
    core
)

class CdkVpcStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, 'cdkvpc', max_azs=3)

        es_sec_grp = ec2.SecurityGroup(self, 'ESSecGrpCDK', 
                                        vpc=vpc,
                                        allow_all_outbound=True,
                                        security_group_name='ESSecGrpCDK')
        es_sec_grp.add_ingress_rule(ec2.Peer.any_ipv4(), ec2.Port.tcp(80))
        es_sec_grp.add_ingress_rule(ec2.Peer.any_ipv4(), ec2.Port.tcp(443))

        domain = es.Domain(self, 'cdkd1', 
                            version=es.ElasticsearchVersion.V7_7,
                            domain_name='cdkd1',
                            capacity=es.CapacityConfig(data_node_instance_type='t3.small.elasticsearch',
                                                       data_nodes=3),
                            ebs=es.EbsOptions(enabled=True,
                                              volume_size=10,
                                              volume_type=ec2.EbsDeviceVolumeType.GP2),
                            vpc_options=es.VpcOptions(
                                security_groups=[es_sec_grp],
                                subnets=vpc.private_subnets,
                            ),
                            zone_awareness=es.ZoneAwarenessConfig(enabled=True,
                                                                  availability_zone_count=3),
                            enforce_https=True,
                            node_to_node_encryption=True,
                            encryption_at_rest={
                                "enabled": True
                            },
                            use_unsigned_basic_auth=True,
                            fine_grained_access_control={
                                "master_user_name": "admin",
                            },
                          )

What did you expect to happen?

The domain should deploy in 3 AZs, according to the config. Alternately, add a min_azs parameter to the vpc() construction.

What actually happened?

cdk deploy failed with insufficient subnets.

Environment

(.env) handler@laptop:/code/cdk-vpc $ aws --version
aws-cli/1.18.114 Python/3.8.3 Darwin/18.7.0 botocore/1.13.50
(.env) handler@laptop:/code/cdk-vpc $ cdk --version
1.68.0 (build a6a3f46)
(.env) handler@laptop:~/code/cdk-vpc $ node -v
v12.19.0

• OS :
  MacOS Mojave 10.14.6
  (.env) handler@laptop:~/code/cdk-vpc $ python --version
  Python 3.8.3

This is 🐛 Bug Report

[iliapolo]

Hi @Jon-AtAWS

Are you initializing the stack in a specific environment or in an agnostic way?

The documentation for the maxAzs property states:

Be aware that environment-agnostic stacks will be created with access to only 2 AZs, so to use more than 2 AZs, be sure to specify the account and region on your stack.

It's done to make sure that an environment agnostic vpc can actually be deployed to any environment.
This means that if you want to use max_azs=3, you need to initialize your stack in a specific region:

stack = CdkVpcStack(..., env=core.Environment(account='123456789', region='us-west-2'))

As an alternative, you can specify subnet configurations when creating the VPC, in which case you don't have to specify the environment on the stack. But you do have to configure maxAzs so that it doesn't conflict. From the docs as well:

To use "all AZs" available to your account, use a high number (such as 99).

const vpc = new ec2.Vpc(stack, 'Vpc', {
  maxAzs: 99,
  natGateways: 1,
  subnetConfiguration: [
    {
      subnetType: ec2.SubnetType.PRIVATE,
      name: 'Private1',
    },
    {
      subnetType: ec2.SubnetType.PRIVATE,
      name: 'Private2',
    },
    {
      subnetType: ec2.SubnetType.PRIVATE,
      name: 'Private3',
    },
  ],
});

[ambasta]

@iliapolo Not sure about OP, but I face this issue w/ region/account set to Aws.region/Aws.accountId respectively

[iliapolo]

@ambasta Aws.region and Aws.account are deploy time values, which means that at synth time, the stack is still treated as an environment agnostic stack, so it defaults to use just 2 AZs. You have to use concrete values to make the stack non-agnostic.

[ambasta]

@iliapolo Aight, short of hardcoding these variables, how do you ensure that the pipeline deploys an application to the account/region it is in?

[iliapolo]

@ambasta You don't need to hardcode in order to achieve that. You can just not define any environment on the stack and the cdk will use the environment defined in the aws credentials and deploy all stacks to that account and region.

You use Aws.region or Stack.of(this).region to get a reference to the region if you need it in one of your constructs.

The only reason to hardcode environment in stacks is if your stack is not environment agnostic, for example, if your VPC needs to be spread on 3 AZ's, its not environment agnostic since not all regions have 3 AZ's.

[ambasta]

Aight, that makes sense. Though I'd really argue default of 3 for HA instead of 2. Maybe add validation for stack failure if we deploy to a zone w/ fewer than 3 AZs

…

On Mon, Nov 23, 2020 at 11:56 AM Eli Polonsky ***@***.***> wrote: @ambasta <https://github.com/ambasta> You don't need to hardcode in order to achieve that. You can just not define any environment on the stack and the cdk will use the environment defined in the aws credentials and deploy all stacks to that account and region. You use Aws.region or Stack.of(this).region to get a reference to the region if you need it in one of your constructs. The only reason to hardcode environment in stacks is if your stack is not environment agnostic, for example, if your VPC needs to be spread on 3 AZ's, its not environment agnostic since not all regions have 3 AZ's. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11026 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAANGQNTPTRGIEEWUHRPP73SRJEXXANCNFSM4S2MFCUQ> .

-- ⍰⍰⍰

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[Jon-AtAWS]

I'm OK to close this issue. Though, I would argue that CDK can find out how many zones there are in a region and can default to min(3, number of zones in the region)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.