Allow specifying VPC peering connections and adding them into the route tables for the shoot nodes

[abh1kg]

How to categorize this issue?

/area networking
/kind enhancement
/priority 2
/platform aws

What would you like to be added:

When a shoot cluster is created on AWS by providing an existing VPC ID, the AWS provider creates and explicitly associates a route table for each subnet (as specified by the zone configurations) where the shoot nodes are attached. The routes for the tables include the "local" route for inter-container traffic and for public connectivity. However, if the VPC was already peered to another, it seems that the responsibility is with the user to manually add the route directed to the peering connection (i.e. the corresponding pcx-xxx) to these route tables.
It would be beneficial for end users if the shoot specification on AWS would allow VPC peering connection IDs to be specified and the provider adds routes for the peerings to the route tables for the shoot node subnets.

Why is this needed:

In the current situation, it seems like the user needs to be aware of the subnet tags and names generated by the AWS provider to be able to add the peer connection routes to the route tables. This is quite cumbersome for end users and also leaves users wondering if "messing around" with the routes would cause reconciliation issues later on. Having this feature inside the provider would help operational setup and overall end user experience.

[dkistner]

Hi @abh1kg,
thanks for the proposal.

I'm not an expert with vpc peering. Provider aws is using Terraform to create the basic network infrastructure for an aws shoot cluster. Could you provide an example how template would need to look like with the connection ids? https://github.com/gardener/gardener-extension-provider-aws/blob/master/pkg/controller/infrastructure/templates/main.tpl.tf

In this case provider-aws is not managing the vpc. What would happen if the peering is removed while the connection id routes are still in the route tables?

[abh1kg]

Hi @dkistner ,
I made a few changes to the template, primarily the section at: https://github.com/abh1kg/gardener-extension-provider-aws/blob/master/pkg/controller/infrastructure/templates/main.tpl.tf#L260-L275.
On second thought, I think we could support the peering connection routes to be added to the worker node route tables irrespective of whether Gardener creates the VPC or the user brings the VPC.
So, I expect an additional config section in the InfrastructureConfig resource for the AWS provider along the following lines:

apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
kind: InfrastructureConfig
enableECRAccess: true
networks:
  vpc: # specify either 'id' or 'cidr'
  # id: vpc-123456
    cidr: 10.250.0.0/16
    # optional list of peering connections (defaults to empty list if not specified)
    peering_connections:
    - name: connect_to_private_databases
      id: pcx-xxxx
      cidr_block: 10.16.0.0/16

If the peering is removed and the shoot YAML is not updated, I might be wrong but I don't think terraform would complain. On the AWS end, it would result in a black hole route, I would assume (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Route.html - search for blackhole).

[abh1kg]

@dkistner : Did you get a chance to look at this?

[dkistner]

Hi @abh1kg,

unfortunately I did not find the time to look deeper into this.

On the AWS end, it would result in a black hole route...

Would this block the shoot deletion/reconciliation?

[abh1kg]

I don't think so because this would be ignored in the terraform operation. But, we would need to verify this of course. Is there a way (may even a dirty method) I could use to run the extension provider for AWS locally?

[dkistner]

Yes, you just need a local k8s cluster and the crds here installed. Then you can run make start and apply your InfrastructureConfig to the local k8s cluster.