Enable concurrent workers for tagging controller

[shiv-amz]

/kind feature

What this PR does / why we need it:

• Tagging controller currently operates as a single-threaded process, leading to a backlog when processing nodes in larger clusters. PR introduces tagging-controller-concurrent-node-syncs option, allowing multiple parallel workers to pull from the work queue.
• Testing with 1,000 nodes at a concurrency level of 10, over 50% of the nodes are tagged in under 1 second. In contrast, with single concurrency, 90% of the nodes take between 1 to 10 seconds to be tagged.

A new command-line flag, `tagging-controller-concurrent-node-syncs`, allows more than one node to be processed at once.

[k8s-ci-robot]

Hi @shiv-amz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cartermckinnon]

/ok-to-test

[cartermckinnon]

/triage accepted
/kind feature

[shiv-amz]

/retest

[cartermckinnon]

CI account has hit a service quota:

   I0328 06:38:39.671578 87733 dump.go:53] At 2025-03-28 06:38:16 +0000 UTC - event for svc-proxy-terminating: {service-controller } SyncLoadBalancerFailed: Error syncing load balancer: failed to ensure load balancer: error authorizing security group ingress: "RulesPerSecurityGroupLimitExceeded: The maximum number of rules per security group has been reached.\n\tstatus code: 400, request id: 10e8ee61-8c6d-4f66-8845-0872eced910c" 

Seeing if we can get it resolved, if not we can override...

[cartermckinnon]

/retest

[k8s-ci-robot]

@shiv-amz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cloud-provider-aws-e2e-kubetest2	b586c99	link	false	/test pull-cloud-provider-aws-e2e-kubetest2

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[cartermckinnon]

Test flakes are not related to this change, going to continue looking into those out of band.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cartermckinnon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [cartermckinnon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment