✨ Support of managing subnets on AWS Wavelength Zones by mtulio · Pull Request #4901 · kubernetes-sigs/cluster-api-provider-aws

mtulio · 2024-04-03T06:07:44Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR implements support of managed subnets and carrier gateway for AWS Wavelength zones. Feature request #4874 .

There API is changed to introduce the following fields:

VPCSpec.CarrierGatewayID: representation of Carrier Gateway resource ("internet gateway" for AWS Wavelength Zones)
SubnetSpec.ZoneType: representation of subnet's zone type
SubnetSpec.ParentZoneName: representation of subnet's parent zone name (an availability zone in the Region which the edge zone is tied)

The subnets in AWS Local Zones and Wavelength Zones are not eligible to create core components for the cluster, like NAT Gateway, Control Plane nodes, and Network Load Balancers, keeping compatibility with existing flow when edge subnets are added.

To create subnets in edge zones, the subnet must be added for each zone you want to create the subnet in NetworkSpec.Subnets. For example to create subnets in Wavelength Zone us-east-1-wl1-nyc-wlz-1, set:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster
metadata:
  name: aws-cluster-wavelength
spec:
  region: us-east-1
  networkSpec:
    vpc:
      cidrBlock: "10.0.0.0/20"
    subnets:
    # regular zones (availability zones. Eg us-east-1a, us-east-1b...)
    [...]
    # Subnets in Wavelength Zones of New York location (public and private)
    - availabilityZone: us-east-1-wl1-nyc-wlz-1
      cidrBlock: "10.0.128.0/25"
      id: "cluster-subnet-private-us-east-1-wl1-nyc-wlz-1"
      isPublic: false
    - availabilityZone: us-east-1-wl1-nyc-wlz-1
      cidrBlock: "10.0.128.128/25"
      id: "cluster-subnet-public-us-east-1-wl1-nyc-wlz-1"
      isPublic: true

This PR is a super set and includes it's dependencies, isolated on each PRs (and it is blocked by those):

Support of Local Zones: ✨ Introduce edge subnets to support AWS Local Zones #4882
Fix route table error handler: 🐛 fix/network/rtb: delete rtb handling err when failed to create routes #4899
Refeact route table/route creation: 🌱 refactor route discover/creation flow #4900
Fix subnet information in unmanaged when failed to tag subnet 🐛 fix: update network subnets prio tag failures #4917
Isolate nat gateway lookup 🌱 net/natgws: introduce test units for nat gateways #4923

Which issue(s) this PR fixes *

Ref #4874

Special notes for your reviewer:

Checklist:

Release note:

action required
Support deploying network requirements, subnets and carrier gateway, in AWS Wavelength Zones. This introduces new required IAM permissions. If you have an existing stack you will need to update it with clusterawsadm bootstrap iam create-cloudformation-stack

k8s-ci-robot · 2024-04-03T06:07:53Z

Hi @mtulio. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

api/v1beta2/network_types.go

pkg/cloud/services/network/routetables_test.go

pkg/cloud/services/network/subnets.go

rvanderp3 · 2024-04-08T17:27:17Z

/test pull-cluster-api-provider-aws-test

rvanderp3 · 2024-04-08T17:45:53Z

/test pull-cluster-api-provider-aws-build

rvanderp3 · 2024-04-08T17:46:05Z

/test pull-cluster-api-provider-aws-test

api/v1beta2/network_types.go

damdo · 2024-04-23T15:19:00Z

/test pull-cluster-api-provider-aws-test

damdo · 2024-04-23T15:27:31Z

@mtulio looks like units are breaking re: subnets.
Could you PTAL?

damdo · 2024-04-23T15:43:17Z

/tide refresh

mtulio · 2024-04-23T15:49:33Z

Converting to draft while waiting for merge && rebase from #4900 and #4882

@mtulio - both have merged now 🎉 Ping me when you have rebased and i will review. Thanks again for these changes.

Hey @richardcase and @damdo , this PR is ready for review. Triggering the e2e's.

cc @nrb

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

mtulio · 2024-04-23T16:52:37Z

pkg/cloud/services/network/carriergateways.go

FYI reviewers: The Carrier Gateway (CAGW) re-conciliator is very similar InternetGateway resource, adapting to CAGW's requirements.

pkg/cloud/services/network/carriergateways.go

pkg/cloud/services/network/carriergateways_test.go

pkg/cloud/services/network/carriergateways.go

mtulio · 2024-04-23T20:15:57Z

@richardcase , feedback addressed. 👍🏽

mtulio · 2024-04-23T23:13:05Z

e2e finished successfully in the first run after rebase. Triggering again after adding cloudformation permissions.

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

/assign @richardcase

mtulio · 2024-04-24T01:10:07Z

e2e job 1782910662712758272 failed, looks like timeout.

/test pull-cluster-api-provider-aws-e2e

richardcase · 2024-04-24T09:36:53Z

e2e job 1782910662712758272 failed, looks like timeout.

It passed on a re-run

mtulio · 2024-04-24T11:54:44Z

e2e job 1782910662712758272 failed, looks like timeout.

It passed on a re-run

🎉 Perfect! Thanks!

@richardcase Let me know if you are ok with the current changes.

My plan to fully address the #4874, aligned w/ we discussed in last community meeting, is to create individual e2e tests for Local Zones and Wavelength Zones (or mixed to save infra/runtime) in follow up PRs. cc @damdo @nrb

Create a dedicated document, "topic", with instructions to deploy network infrastructure (subnets, gateways and route tables) in "edge zones" - Local Zone and Wavelength Zone infrastructure.

This change introduce support of required network components to deploy subnets on AWS Wavelength Zones infrastructure. The NetworkSpec API handles the CarrierGatewayId on NetworkSpec with the unique identifier of Carrier Gateway resource attached to the VPC.

Subnets in AWS Wavelength Zone is a classified as a type of edge subnets, not used to create regular control plane resources, like nodes, NAT Gateways or API Load Balancers. The ZoneType is used to group the zones from regular and the edge zones. Regular zones are with type 'availability-zone', and the edge zones are types 'local-zone' and 'wavelength-zone'. The following statements are valid for edge subnets: - private subnets supports egress traffic only using NAT Gateway in the region. - public subnets in Wavelength must be attached to a route table with valid Carrier Gateway as a default route. - public subnets in Wavelength zones does not support map public ip on launch flag, instead, the runInstance must set the network interface flag to assign public ip from carrier gateway - IPv6 subnets is not supported in edge zones - subnet tags for load balancer are not set in edge subnets. Edge subnets should not be elected by CCM to create service load balancers. Use ALB ingress instead

✨ edge subnets/cagw: carrier gateway for public subnets in Wavelength Introduce Carrier Gateway resource reconciliator in the network service. Carrier Gateway is the gateway responsible to route ingress and egress traffic **in/out the Wavelength Zone**, located in the Carrier Infrastructure - communications service providers’ (CSP) 5G networks. Carrier Gateway is similar Internet Gatewat resource, responsible for the network border groups in the Region and Local Zones for public subnets. ✨ edge subnets/routes: supporting custom routes for Wavelength For private and public subnets in edge zones, the following changes is introduced according to each rule: General: - IPv6 subnets is not be supported in AWS Local Zones and Wavelength zone, consequently no ip6 routes will be created - nat gateways is not supported, default gateway's route for private subnets will use nat gateways from the zones in the Region (availability-zone's zone type) - one route table by zone's role by zone (standard flow) Private tables for Local Zones and Wavelength: - default route's gateways is assigned using nat gateway created in the region (availability-zones). Public tables for Wavelength zones: - default route's gateways is assigned using Carrier Gateway, resource introduced in the edge zone's feature. The changes in the standard flow (without edge subnets' support) was isolated in the PR kubernetes-sigs#4900

Add IAM policy on cloudformation templates for clusterawsadm to manipulate gateways in Wavelength zone: carrier gateway.

mtulio · 2024-04-24T18:43:38Z

Just rebased to get the most recent fixes from main - used in external projects to trigger e2e. cc @nrb @damdo

mtulio · 2024-04-25T04:22:17Z

@richardcase Let me know if you are ok with the current changes.

My plan to fully address the #4874, aligned w/ we discussed in last community meeting, is to create individual e2e tests for Local Zones and Wavelength Zones (or mixed to save infra/runtime) in follow up PRs. cc @damdo @nrb

Hi @richardcase - let me share the e2e tests we are running on OpenShift using this version and a few details about how the e2e for edge zones is built to hear from you if it makes sense to use a similar approach on CAPA.

e2e results on OpenShift using this PR/CAPA version:

Wavelength: Manual, Managed, Unmanaged (BYO VPC)
Local Zones: Managed, Unmanaged

e2e details - the "edge subnets" CI workflow is built on openshift using the flow similar to this:

pre
- one edge zone is randomly selected from DescribeAvailabilityZones(AllZones==true)
- the zone attribute OptInStatus is checked, and if the status is not-opted-in the zone's zone group is modified to opted-in
- the zone is ready to create subnets
if managed: the SubnetSpec will be created setting the public and private subnets for each regular zone, and the selected edge zone
if un-managed/BYO VPC: the required network components are created for unmanaged/BYO net (VPC, subnets, rtbs, gateways, etc), and the SubnetSpec will be created with subnet Ids, and respective zones
CAPA install
e2e run
Tear down:
- cluster destroyed
- if un-managed/BYO VPC, the cloudformation stacks will be deleted

Note 0: the edge zone is randomly selected to increase coverage and dynamically test new zones while AWS is adding it.
Note 1: the zone group isn't modified (opted out) to the old state as AWS requires to open a ticket to do it. There is no additional cost to keep those enabled or create subnets and carrier gateways.
Note 2: Considering "edge subnets" feature can include Local Zones (LZ) and Wavelength Zones (WL), we also created a single workflow randomly selecting one zone LZ and WL each to test creating a cluster and running e2e in a single flow, saving test time and infra costs.

cc @nrb @r4f4 @damdo

nrb · 2024-04-25T19:15:17Z

Thanks for the details on how OpenShift is testing this! It's good to know there's downstream testing of this feature, though of course we'd like to see similar tests added to this repo as well.

/lgtm

richardcase · 2024-04-26T13:39:36Z

@mtulio - just to let you know that i updated the "release note" section to mark it as action required.

richardcase · 2024-04-26T13:40:09Z

/test pull-cluster-api-provider-aws-e2e-eks

richardcase · 2024-04-26T13:40:33Z

From my side:

/approve

When the e2e passes then we can unhold to merge.

k8s-ci-robot · 2024-04-26T13:40:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nrb · 2024-04-26T16:11:48Z

/unhold

k8s-ci-robot requested review from luthermonson and richardcase April 3, 2024 06:07

k8s-ci-robot added needs-priority needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 3, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Apr 3, 2024

This was referenced Apr 3, 2024

CORS-3214: capi-aws - Wavelength Zones network provisioning openshift/installer#8174

Merged

🌱 refactor route discover/creation flow #4900

Merged

🐛 fix/network/rtb: delete rtb handling err when failed to create routes #4899

Merged

nrb reviewed Apr 4, 2024

View reviewed changes

api/v1beta2/network_types.go Outdated Show resolved Hide resolved

mtulio force-pushed the CORS-3214-wavelength-zones branch 2 times, most recently from d545d8d to edeba21 Compare April 4, 2024 19:30

mtulio force-pushed the CORS-3214-wavelength-zones branch 3 times, most recently from 939ef72 to 73f5db7 Compare April 8, 2024 14:53

mtulio commented Apr 8, 2024

View reviewed changes

pkg/cloud/services/network/routetables_test.go Outdated Show resolved Hide resolved

mtulio commented Apr 8, 2024

View reviewed changes

pkg/cloud/services/network/subnets.go Outdated Show resolved Hide resolved

mtulio force-pushed the CORS-3214-wavelength-zones branch from 73f5db7 to 2b0fe5b Compare April 8, 2024 17:39

mtulio commented Apr 23, 2024

View reviewed changes

api/v1beta2/network_types.go Outdated Show resolved Hide resolved

mtulio force-pushed the CORS-3214-wavelength-zones branch from 0d6c364 to a9ecbb4 Compare April 23, 2024 15:02

mtulio marked this pull request as ready for review April 23, 2024 15:03

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2024

mtulio force-pushed the CORS-3214-wavelength-zones branch from a9ecbb4 to 1b9c633 Compare April 23, 2024 15:37

mtulio commented Apr 23, 2024

View reviewed changes

richardcase reviewed Apr 23, 2024

View reviewed changes

pkg/cloud/services/network/carriergateways.go Outdated Show resolved Hide resolved

pkg/cloud/services/network/carriergateways_test.go Outdated Show resolved Hide resolved

pkg/cloud/services/network/carriergateways.go Outdated Show resolved Hide resolved

mtulio added 5 commits April 24, 2024 15:42

✨ edge subnets/docs: added guide subnets on Local and Wavelength zones

59f85a3

Create a dedicated document, "topic", with instructions to deploy network infrastructure (subnets, gateways and route tables) in "edge zones" - Local Zone and Wavelength Zone infrastructure.

✨ edge subnets/clusteradm/policy: add Carrier Gateway operations

2270604

Add IAM policy on cloudformation templates for clusterawsadm to manipulate gateways in Wavelength zone: carrier gateway.

mtulio mentioned this pull request Jun 15, 2024

Create network resources for AWS Local Zones and Wavelength Zones #4874

Closed

Conversation

mtulio commented Apr 3, 2024 • edited by richardcase Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rvanderp3 commented Apr 8, 2024

Uh oh!

rvanderp3 commented Apr 8, 2024

Uh oh!

rvanderp3 commented Apr 8, 2024

Uh oh!

Uh oh!

damdo commented Apr 23, 2024

Uh oh!

damdo commented Apr 23, 2024

Uh oh!

damdo commented Apr 23, 2024

Uh oh!

mtulio commented Apr 23, 2024

Uh oh!

mtulio Apr 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mtulio commented Apr 23, 2024

Uh oh!

mtulio commented Apr 23, 2024

Uh oh!

mtulio commented Apr 24, 2024

Uh oh!

richardcase commented Apr 24, 2024

Uh oh!

mtulio commented Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtulio commented Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtulio commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nrb commented Apr 25, 2024

Uh oh!

richardcase commented Apr 26, 2024

Uh oh!

richardcase commented Apr 26, 2024

Uh oh!

richardcase commented Apr 26, 2024

Uh oh!

k8s-ci-robot commented Apr 26, 2024

Uh oh!

nrb commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mtulio commented Apr 3, 2024 •

edited by richardcase

Loading

mtulio commented Apr 24, 2024 •

edited

Loading

mtulio commented Apr 24, 2024 •

edited

Loading

mtulio commented Apr 25, 2024 •

edited

Loading