fix(cloudflare): improve handling of rate limiting errors

[Hackatosh]

What does it do ?

When a rate limiting error is encountered while using Cloudflare provider, external dns crashes. This PR fixes this behavior.

Motivation

Cloudflare library return a generic error when a rate limiting error is encountered (see #4876 (comment)). I added a condition to handle this type of error better and return a soft error.

An issue has been opened on Cloudflare side (cloudflare/cloudflare-go#4155) and a fix has been submitted (cloudflare/cloudflare-go#4156), but unfortunately there was no reaction from the library maintainers. That's why I try to fix the issue on external-dns side, even if it feels more like a "hack"

More

• [ X] Yes, this PR title follows Conventional Commits
• [X ] Yes, I added unit tests
• [ X] Yes, I updated end user documentation accordingly

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Hackatosh / name: Edouard Benauw (5f5e194, cce1dd3, 51ad1fe, 831ef61, 8a5c10b, cf6041e, 3f12436, 388a992)

[k8s-ci-robot]

Welcome @Hackatosh!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @Hackatosh. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vflaux]

I can't say whether this will be accepted, but regarding the code, I would suggest adding a new function isRateLimited() that checks both apiErr.ClientRateLimited() and the error message. Then, I would replace each call to apiErr.ClientRateLimited() with this new function.

[Starttoaster]

I agree with you about making a function for this duplicated code. But can you expand on why this wouldn't be accepted? This is a huge problem for multi-cluster multi-domain setups using external-dns, it's very likely you'll run into rate limit errors with the default polling interval. That means the only solution is to increase the polling interval with each new cluster or domain that external-dns manages records for, which I'd argue isn't a great solution or user experience.

With Cloudflare cold shouldering the fix on their end, it seems silly to not at least temporarily account for a known quirk of their API library here.

[ivankatliarchuk]

@Hackatosh any chance you sign EasyCLA so that we could review it?

[ivankatliarchuk]

This PR make sense, but not sure if it will be accepted as well

There was previous attempts #5233 to do something similar and similar issue #5225 (comment)

No need to close this; the idea itself makes sense. I'm very much on the fence about the order of resolution as well. Adding soft errors first feels like treating symptoms instead of the root cause.

External DSN currently abusing Cloudflare API, which is not correct. We should be good citizens and improve not just crash/no-crash, but actually start adding retry/backkoff etc to cloudflare provider.

Example

config, err = cloudflare.NewWithAPIToken(token, []cloudflare.Option{
			cloudflare.UsingRateLimit(rateLimit),
			cloudflare.UsingRetryPolicy(retry, minDelaySeconds, maxDelaySeconds), // 3 retries, min 1s delay, max 5s delay
			cloudflare.UserAgent(externaldns.UserAgent()),
		}...)

Something like that required as well. As at the moment, cloudflare API as it looks, uses some defaults

[Hackatosh]

@ivankatliarchuk I have signed the CLA, you can review the PR :)

We could improve retry/backoff policy if you want. What do you think would be appropriate ? Maybe we can make this configurable using environnement variables

[ivankatliarchuk]

/ok-to-test

[ivankatliarchuk]

I can see there were an attempt to fix #4437 and agree with the comment that the issue is with the upstream library.

As temp solution I think it should be acceptable to implement the fix.

Just follow the suggestion #5524 (comment) and make sure to add unit tests for all added lines.

We could improve retry/backoff policy if you want. What do you think would be appropriate ? Maybe we can make this configurable using environnement variables

If you have time and interest, you could try to improve retry/backoff in follow-up for sure

[ivankatliarchuk]

How do you know, the fix is actually solves the problem?

[Hackatosh]

I factorized error handling in convertCloudflareError and I added one unit test to test that the the condition I have added works correctly. The error I use for the test is identical to the one thrown by Cloudflare

To know if the fix works correctly, I am going to build external-dns with this fix and try it on a staging environnement where external-dns crashes everyday due to this issue, and see what happens. Is that OK for you ?

[vflaux]

Maybe add a comment as a remainder that this is a workaround for an issue with the cloudflare client ?
So we know that we can remove this when fixed ?

[Hackatosh]

I added a comment

[Hackatosh]

I tried the fix on our staging environnement and got logs like this instead of crash :

{"level":"error","msg":"Failed to do run once: could not fetch records from zone, soft error\nexceeded available rate limit retries (consecutive soft errors: 1)","time":"2025-06-16T17:09:15Z"}
{"level":"error","msg":"Failed to do run once: soft error\nexceeded available rate limit retries (consecutive soft errors: 2)","time":"2025-06-16T17:11:33Z"}
{"level":"info","msg":"All records are already up to date","time":"2025-06-16T17:14:45Z"}
{"level":"info","msg":"Reconciliation succeeded after 2 consecutive soft errors","time":"2025-06-16T17:14:45Z"}

For me it works correctly :D Is it ok for you to merge this PR ? 🙏

[ivankatliarchuk]

Hi @vflaux anything left on your side ?

[k8s-ci-robot]

@vflaux: changing LGTM is restricted to collaborators

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivankatliarchuk]

/lgtm

[vflaux]

lgtm

[k8s-ci-robot]

@vflaux: changing LGTM is restricted to collaborators

Details

In response to this:

lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Hackatosh]

Hello ! @ivankatliarchuk Does someone else need to approve this PR ? 🤔

[ivankatliarchuk]

cc @mloiseleur for final review

[mloiseleur]

Many thanks for your review @vflaux 👍
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mloiseleur, vflaux

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mloiseleur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Hackatosh]

Thank you for the merge ! :)

Do you know when this fix will be released in the official image ?

[mloiseleur]

You can use the staging image to use it before official release.
The release of 0.18 is planned for next week, see #5545

[Hackatosh]

Thank you very much !

Thank you for your reactivity on this PR, it was a nice experience to contribute :)