feat(aws): always create AAAA alias records in route53

[rlees85]

Description

This change creates AAAA and A records for AWS Route53. It is heavily based on this pull request that seems to have sadly been abandoned: #3605

This is now based on the current master branch. I've tried to add some more thorough test cases so that when I let this loose in a Kubernetes cluster hopefully not too many bad things will happen.

Fixes: #3707
Fixes: #4509

All credits to @johngmyers for the vast majority of the leg work.

Checklist

• Unit tests updated
• End user documentation updated

Checklist to drop WIP

• Try this in a real cluster and see if it actually works
• Fully address concerns from maintainers in other pull request

[k8s-ci-robot]

Welcome @rlees85!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @rlees85. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rlees85]

I just let it rip in our development cluster. At first look, seems to work well. AAAA records created and nothing has broken, yet.

One observation, is that the TXT records that are created seem a bit odd:

We get: _extdns.aaaa-example.host.name for the AAAA record but the A record is translated somewhere, so we get _extdns.cname-example.host.name. This feels like carry-over from the caching thing or some intentional backwards compatibility? I don't know if we care about this, will need guidance here.

We also get: _extdns.example.host.name as normal.

If anyone has any good example test cases I can carry out I will do so.

[rlees85]

I found the concerns @mloiseleur raised:

• It should be documented in doc/tutorials/aws.md, since it works following this tutorial on Service with annotations

I think I've addressed this now.

• When using --managed-record-types="A,CNAME", it did NOT update records at all (Same behavior between this PR and v0.13.5). So it does not seem to be a working opt-out.

This works, we were using it wrong:

        - --managed-record-types=A
        - --managed-record-types=CNAME

• When changing the hostname, for instance from test2.example.com to test3.example.com, it did not delete the AAAA record:

I can confirm the last one is now working. It cleans up AAAA records when both changing the name of an ingress as well as when deleting it.

[ivankatliarchuk]

When you ready, please share all relevant manifest, kubectl and AWS commands. I would like to test it as well.

One concern here, not sure if covered, why would I need AAAA records alongside A records? This seems more like a flag to me, as why would I need AAAA records if I never asked for it, plus it's easy to remove a flag and make it default behaviour at some point?

[ivankatliarchuk]

/ok-to-test

[rlees85]

@ivankatliarchuk AAAA records are the same as A records, but for IPv6. So for example, if a load balancer in AWS has dualstack networking enabled (both IPv4 and IPv6) we should add both an A record with the IPv4 address and AAAA record for IPv6.

The previous pull request found that even with IPv4 only load balancers, adding AAAA records causes no harm, since simply no IPv6 address is returned. This makes the code much simpler.

There is not really much point in turning this off beyond not liking AAAA records being in the console when having a look there. I think there were concerns about the extra requests contributing to rate limiting, but a caching feature has gone in recently to mitigate that (I believe?). If a way to turn this off is a deal breaker, I will try to implement something.

With manifests sadly its all a bit rough right now.

• Compile the binary, with make
• Create this (terrible) Dockerfile next to the binary, in build:

FROM registry.k8s.io/external-dns/external-dns:v0.15.0
COPY "external-dns" "/ko-app/external-dns"
ENTRYPOINT [ "/ko-app/external-dns" ]

• Manually push the resulting image to a container registry. I've had to use a private one for now since its a company cluster.
• Manually edit the deployment for external DNS, to use the new image.

I will have a look to see if I can host it in a public repository somewhere for testing.

[ivankatliarchuk]

Just share manifests here. Normally, you don't need to create a container. You could do something like this https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/dev-guide.md#execute-code-without-building-binary, as long as external-dns is aware about the cluster context.

I got your point, sounds reasonable. What about dualstack-without-public-ipv4 ?

Probaly worth to validate cases when ALB switch is happening from ipv4 -> dualstack -> ipv4 where or not it works without issues

[ivankatliarchuk]

Am I understood correctly

Before

piVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ip-address-type: dualstack
  name: echoserver
spec:
  ingressClassName: alb
  rules:
  - host: echoserver.example.org
    http:
      paths:
      - path: /
        backend:
          service:
            name: echoserver
            port:
              number: 80
        pathType: Prefix

With this change, the proposed behaviour is not to rely on alb.ingress.kubernetes.io/ip-address-type: dualstack but all the records that have IPv6 to actually add them to DNS registry?

This statement is correct

So for example, if a load balancer in AWS has dualstack networking enabled (both IPv4 and IPv6) we should add both an A record with the IPv4 address and AAAA record for IPv6.

But now, on top of that, for example we have dozens of nodes with IPv6 that we do not want to expose over route53, suddenly will be exposed?

[rlees85]

But now, on top of that, for example we have dozens of nodes with IPv6 that we do not want to expose over route53, suddenly will be exposed?

I can't think of any case where you'd have a load balancer with dualstack enabled where you'd want to expose IPv4 but not IPv6. The simple solution here would be to make it not a dualstack load balancer, then it won't be exposed on IPv6.

We create an AAAA record for it (as long as ALIAS is enabled), but it just resolves to blank/empty, so nothing is exposed.

[rlees85]

You can exclude this by using the following flags:

        - --managed-record-types=A
        - --managed-record-types=CNAME

Tested working just now. I think that covers everything required to get this moving

[ivankatliarchuk]

Could you share e2e test results and manifests you were using? Example test cases #5085 (comment)

I believe in this case, we are looking for something like create NLB and validate different things, like change it from ipv4 to dualstack and back.

[ivankatliarchuk]

You can exclude this by using the following flags:

        - --managed-record-types=A
        - --managed-record-types=CNAME

Tested working just now. I think that covers everything required to get this moving

This most likely should go to documentation somewhere.

[ivankatliarchuk]

Any chance to rebase with master? I have make cover-html failing locally

[ivankatliarchuk]

The PR is quite big. I'll try to go over changes in sources over a weekend.

Could you review docs, there are some references to an annotation external-dns.alpha.kubernetes.io/dualstack. Just reference to annotations need to be removed, the actual manifest we better keep for , as it's still applies

Example

external-dns/docs/sources/gateway.md

Line 87 in 58ac76e

External DNS Controller uses the `external-dns.alpha.kubernetes.io/dualstack` annotation to determine this. If this annotation is

[ivankatliarchuk]

I see the same logic aka slices.Contains shell we consider to create a method in endpoints go?

something like

func (ep *Endpoint) IsSupportedRecordTypes(r ...string) bool

Just an idea

[rlees85]

Thanks for the review (so far). There is a lot to go at here, I will try and come back with some changes.
edit: I will also come back with some manifests and methods of testing that I've used.

[rlees85]

Starting with a Kubernetes cluster (EKS, if it matters) and the AWS load balancer controller installed, so that I can create load balancers for services and ingresses.

The a compiled version of the current head of this branch, as of this comment, is in a public Docker image located here: public.ecr.aws/arden-pub/rich-testing-edns:2

Steps:

• Create namespace: kubectl create namespace external-dns
• Prepare for Helm installation

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update

• Prepare Helm values (values.yaml)

domainFilters:
  - dev02.example.domain

extraArgs:
  - --aws-zone-type=public
  - --aws-zones-cache-duration=15m   # tried with and without
  - --min-event-sync-interval=10s    # tried with and without
  - --provider-cache-time=5m         # tried with and without

image:
  repository: public.ecr.aws/arden-pub/rich-testing-edns
  tag: "2"

interval: 2m
policy: sync
provider: aws

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<redacted>:role/<my-role-that-gives-external-dns-permission-to-route53>"

triggerLoopOnEvent: true
txtOwnerId: dev02-k8s
txtPrefix: _extdns.

• Install External DNS: helm -n external-dns upgrade --install --values "values.yaml" external-dns external-dns/external-dns

• Install a testing deployment: kubectl create -f deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.26
        ports:
        - containerPort: 80

• Install a testing service: kubectl create -f service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.dev02.example.domain
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer

• Check both A and AAAA records are created

time="2025-02-25T13:24:24Z" level=info msg="Desired change: CREATE _extdns.aaaa-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:24:24Z" level=info msg="Desired change: CREATE _extdns.cname-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:24:24Z" level=info msg="Desired change: CREATE _extdns.nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:24:24Z" level=info msg="Desired change: CREATE nginx.dev02.example.domain A" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:24:24Z" level=info msg="Desired change: CREATE nginx.dev02.example.domain AAAA" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:24:24Z" level=info msg="5 record(s) were successfully updated" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.

• Try to resolve the A record (should be successful)

$ q A nginx.dev02.example.domain.
nginx.dev02.example.domain. 1m A 192.168.133.154
nginx.dev02.example.domain. 1m A 192.168.134.198
nginx.dev02.example.domain. 1m A 192.168.136.104

• Try to resolve the AAAA record (should be blank, because although we created it, the balancer is not dual stack or IPv6)

$ q AAAA nginx.dev02.example.domain.

• Make our service "dualstack" by adding the following annotation: service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
• Wait for reconcile
• Repeat resolving the AAAA record (should now be successful)

$ q AAAA nginx.dev02.example.domain.
nginx.dev02.example.domain. 1m AAAA 2a05:d018:239:8410:5b19:5830:b3c8:3e26
nginx.dev02.example.domain. 1m AAAA 2a05:d018:239:8411:ce4f:40dc:20f5:81fb
nginx.dev02.example.domain. 1m AAAA 2a05:d018:239:8412:99b3:21b4:f03b:937a

• Delete the service: kubectl delete service nginx-service
• Check DNS records, including AAAA, are cleaned:

time="2025-02-25T13:38:05Z" level=info msg="Desired change: DELETE _extdns.aaaa-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:38:05Z" level=info msg="Desired change: DELETE _extdns.cname-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:38:05Z" level=info msg="Desired change: DELETE _extdns.nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:38:05Z" level=info msg="Desired change: DELETE nginx.dev02.example.domain A" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:38:05Z" level=info msg="Desired change: DELETE nginx.dev02.example.domain AAAA" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:38:05Z" level=info msg="5 record(s) were successfully updated" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.

• Edit the ExternalDNS deployment, add the following parameter: --exclude-record-types=AAAA
• Wait for the pod to restart
• Create the service again: kubectl create -f service.yaml. This time only A records should be created:

time="2025-02-25T13:50:41Z" level=info msg="Desired change: CREATE _extdns.cname-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:50:41Z" level=info msg="Desired change: CREATE _extdns.nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:50:41Z" level=info msg="Desired change: CREATE nginx.dev02.example.domain A" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:50:41Z" level=info msg="3 record(s) were successfully updated" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.

• Edit the ExternalDNS deployment, remove the following parameter: --exclude-record-types=AAAA
• Wait for the pod to restart
• The missing AAAA records should now be created:

time="2025-02-25T13:51:59Z" level=info msg="Desired change: CREATE _extdns.aaaa-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:51:59Z" level=info msg="Desired change: CREATE nginx.dev02.example.domain AAAA" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:51:59Z" level=info msg="2 record(s) were successfully updated" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.

• Edit the service, change the hostname to make sure records are moved properly

time="2025-02-25T13:54:41Z" level=info msg="Desired change: DELETE _extdns.aaaa-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: DELETE _extdns.cname-nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: DELETE _extdns.nginx.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: DELETE nginx.dev02.example.domain A" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: DELETE nginx.dev02.example.domain AAAA" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: CREATE _extdns.aaaa-apache.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: CREATE _extdns.apache.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: CREATE _extdns.cname-apache.dev02.example.domain TXT" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: CREATE apache.dev02.example.domain A" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="Desired change: CREATE apache.dev02.example.domain AAAA" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.
time="2025-02-25T13:54:41Z" level=info msg="10 record(s) were successfully updated" profile=default zoneID=/hostedzone/(redacted) zoneName=dev02.example.domain.

Currently the AWS load balancer controller seems not to support: dualstack-without-public-ipv4.
Infact it doesnt seem to support ALB at all, just NLB.

For this test I hand-cranked an application load balancer with this mode and again hand-cranked in
two Route53 records (one A and one AAAA, both alias) and tried to resolve them both:

• Try to resolve the A record (should be blank)

$ q A alb.dev02.example.domain.

• Try to resolve the AAAA record (should be successful)

$ q AAAA alb.dev02.example.domain.
alb.adev02.example.domain. 1m AAAA 2a05:d018:239:8400:460b:c2e4:6899:8978

Route53 seems to handle the fact that the backend can only support IPv4, IPv6 or both for you as long as you have both A and AAAA records created. Returning blank just means the other protocol gets used, which does return records.

[ivankatliarchuk]

/lgtm

cc @mloiseleur

[mloiseleur]

@project0 : You were interested by this feature, IIRC. Anything missing on this PR ? Do you want to review this PR before merge ?

[rlees85]

Definitely worth --exclude-record-types=AAAA going into the changlog with this change if/when this does release, just to make it nice and clear

[ivankatliarchuk]

/label tide/merge-method-squash

[mloiseleur]

Many thanks @rlees85 for taking the time to finish this one.
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivankatliarchuk, mloiseleur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mloiseleur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rlees85]

Now merged, if anyone happens to be using public.ecr.aws/arden-pub/rich-testing-edns:2 please move to something like gcr.io/k8s-staging-external-dns/external-dns:v20250304-v0.15.1-292-g1e8e5e06 instead (the former public registry isn't really mine and needs to be cleaned)

Many thanks