Namespace during restore lack label during create call from backup namespace preventing create via validating webhooks

[birsanv]

What steps did you take and what happened:

Using Velero 1.14
We have an issue where a customer sets some labels on a namespace and uses validating webhooks to force the label being set when a namespace is created, manually or through a restore operation.

If you backup any resources created in this namespace, the actual namespace resource is not backed up. So when the restore is executed and the namespace doesn't exist, the restore operation creates a namespace using the resource metadata.namespace name. But since the namespace was not backed up , any other information is missing from the new ns resource and the validating webhooks fails the ns creation because is missing the required label.

We request that the namespace resource is also backed up with the resource so that it can be fully recreated on restore, if it doesn't exist.

Steps to reproduce:

Steps to Reproduce:

1. Create a namespace and set a label label1: value; define a validating webhooks that fails a ns creation if the ns doesn't have the label1

2. Create a resource under that ns

3. Back up the resource using a Velero backup

4. Delete the ns

5. Run the Restore operation to recreate the resource.

6. The ns is created by restore but with no label. And since the validating webhooks doesn't allow for the ns to be created, the restore fails

What did you expect to happen:
Step 6 should successfully recreate the backed up resource

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

https://issues.redhat.com/browse/OADP-6252

[sseago]

Note that the reproducing factor here is that the backup uses orLabelSelectors to select resources Since the namespace itself doesn't include those labels, the namespace isn't included in the backup. It's slightly confusing because there are two types of labels here:

1. namespace labels, required by the cluster's validating webhooks to exist -- so a simple kubectl create namespace will fail if the labels aren't present
2. workload labels, which the velero user adds to the backup spec to tell velero what resources to back up.

Because the backup's orLabelSelectors includes the workload labels, the namespaces which include those resources are not included in the backup.

In general, velero should always include a namespace in a backup if a resource is included in the backup that is inside this namespace, that way on restore, a workload that depends on specific namespace metadata (labels, annotations, etc.) will see the namespace as expected.

Proposed fix:
In the ItemBackupper, when backing up a namespaced resource, call backupItem on the resource bypassing labelSelector checks, but enforcing resource includes/excludes. In other words, we don't expect that the backup will have the workload label, so that's ok, but if the user has explicitly excluded namespaces from the backup, or if the user has set includeClusterResources to false, then we don't back up the namespace, otherwise we do.

[Lyndon-Li]

Does include the namespace into the backup work around this problem?

[sseago]

@Lyndon-Li I think that would work, however, the issue is:

1. user is backing up a single workload in a namespace using label selector, it's not reasonable to require the namespace to include every label for every workload in it
2. I think we either added (or were considering to add) code to include everything in a NS when using a label selector and the NS matched: @kaovilai I think you were working on this. Did this get implemented?

[blackpiglet]

#7409

Please check whether the PR can address this issue.

[kaovilai]

I think we either added (or were considering to add) code to include everything in a NS when using a label selector and the NS matched: @kaovilai I think you were working on this. Did this get implemented?

I moved it out to 1.18
#7492

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[github-actions]

This issue was closed because it has been stalled for 14 days with no activity.

[kaovilai]

https://issues.redhat.com/browse/OADP-6252