OSASINFRA-3675: Rework authentication in Manila CSI Driver Operator #373

dkokkino · 2025-03-26T11:04:17Z

Updated manila assets to mount secret and configmap to node/controller
Modified the secret generator to use different parameters in preparation for change to manila driver

openshift-ci-robot · 2025-03-26T11:04:21Z

@dkokkino: This pull request references OSASINFRA-3675 which is a valid jira issue.

In response to this:

Updated manila assets to mount secret and configmap to node/controller

Modified the secret generator to use different parameters in preparation for change to manila driver

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

stephenfin · 2025-03-27T11:42:31Z

pkg/openstack-manila/secret/secretsync.go

+	data["os-use-clouds"] = []byte(strconv.FormatBool(cloud.UseClouds))
+
+	if cloud.CloudsFile != "" {
+		data["os-clouds-file"] = []byte(cloud.CloudsFile)


This indicates that path that the clouds.yaml can be found in. We're using this value in the Manila CSI driver, thus the value of this should be static and correspond to the path you are mounting the clouds.yaml file to.

Looking at your manifests from an earlier commit, that file is currently /etc/kubernetes/secret/clouds.yaml (though I'm suggesting to change it). You should use the same static path here (with a comment explaining what the path corresponds to)

assets/overlays/openstack-manila/patches/node_add_driver.yaml

stephenfin · 2025-03-27T11:46:17Z

assets/overlays/openstack-manila/patches/node_add_driver.yaml

+            items:
+              - key: cloud.conf
+                path: cloud.conf
+        - name: secret-manilaplugin


I know you're aligning with Cinder again here, but would a more descriptive name be helpful? Say, cloud-credentials? Again, I'm planning on doing the same thing for Cinder.

stephenfin · 2025-03-27T11:47:08Z

assets/overlays/openstack-manila/patches/node_add_driver.yaml

+              mountPath: /etc/kubernetes/config
+              readOnly: true
+            - name: secret-manilaplugin
+              mountPath: /etc/kubernetes/secret


Purely a nice to have, but clouds.yaml are typically found in /etc/openstack. I know you're aligning with Cinder here - which is good - but I wonder if we could use that here?

Note: I am planning on changing this in Cinder too

stephenfin

Looks like you need to run make verify too

stephenfin · 2025-04-01T10:22:30Z

pkg/openstack-manila/secret/secretsync.go

-	}
+	data["os-cloud"] = []byte(util.CloudName)
+	data["os-use-clouds"] = []byte(strconv.FormatBool(true))
+	data["os-clouds-file"] = []byte("etc/openstack/clouds.yaml")


Suggested change

data["os-clouds-file"] = []byte("etc/openstack/clouds.yaml")

data["os-clouds-file"] = []byte("/etc/openstack/clouds.yaml")

You should also leave a comment here indicating what this path means (hint: you defined it in the assets)

stephenfin · 2025-04-01T10:24:24Z

pkg/openstack-manila/secret/secretsync.go

-		data["os-certAuthorityPath"] = []byte(cacertPath)
-	}
+	data["os-cloud"] = []byte(util.CloudName)
+	data["os-use-clouds"] = []byte(strconv.FormatBool(true))


We're never expecting to change this so it can be static.

Suggested change

data["os-use-clouds"] = []byte(strconv.FormatBool(true))

data["os-use-clouds"] = []byte("true")

pierreprinetti · 2025-04-24T08:45:18Z

/retest-required

assets/overlays/openstack-manila/base/config_secret.yaml

dkokkino · 2025-05-07T11:52:52Z

/retest-required

dkokkino · 2025-07-16T13:38:02Z

/retest

stephenfin · 2025-07-22T14:53:57Z

This is going to fail until we get the CPO changes from upstream to downstream. For that to happen, we need openshift/release#67348 to merge and openshift/cloud-provider-openstack#334 to be updated to sync from release-1.33 instead of release-1.32.

dkokkino · 2025-07-25T12:37:31Z

/retest

MiguelCarpio · 2025-08-06T09:09:59Z

/test e2e-openstack

- Adding clouds.yaml as a secret mount volume. This will be used for authentication by the manila driver to match how cinder does authentication.

- Goal is to change how the aunthentication for the manila driver occurs.Changes the fields generated in the secret csi-manila-secrets

- Previously the csi-manila-secrets secret in the openshift-manila-csi-driver namespace required secretsync.go to generate the secret dynamically. Since we are changing how the authentication occurs for the manila driver that process can be replaced with a static secret asset.

mandre · 2025-08-07T06:27:01Z

/test e2e-openstack

dkokkino · 2025-08-12T15:16:02Z

/retest

MiguelCarpio · 2025-08-13T14:05:26Z

/test e2e-openstack

dkokkino · 2025-08-14T15:24:24Z

/test e2e-openstack-manila-csi

dkokkino · 2025-08-18T08:09:41Z

/test e2e-openstack-manila-csi

dkokkino · 2025-08-18T11:15:48Z

/test e2e-openstack-manila-csi

MiguelCarpio · 2025-08-19T06:48:09Z

/test e2e-openstack-manila-csi

stephenfin · 2025-08-21T15:02:49Z

/approve
/lgtm
/hold

I've tested this locally with openshift/hypershift#6683 and HCP deploys just fine. @dkokkino please remove the hold once openshift/hypershift#6683 has merged

openshift-ci · 2025-08-21T15:03:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkokkino, stephenfin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [stephenfin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mandre · 2025-08-22T05:24:23Z

/retest

stephenfin · 2025-08-22T08:06:29Z

/retest

Looks like a slow node

dkokkino · 2025-08-22T10:14:38Z

/test hypershift-e2e-openstack-aws-csi-manila

dkokkino · 2025-08-22T11:57:22Z

/test hypershift-e2e-openstack-aws-csi-manila

dkokkino · 2025-08-22T14:23:40Z

/test hypershift-e2e-openstack-aws-csi-manila

stephenfin · 2025-09-16T10:50:40Z

I am still investigating the hypershift failures. Just leaving notes here for myself. Currently this is failing with:

E0822 16:03:47.510390       1 driver.go:331] [ID:593] GRPC error: rpc error: code = Unauthenticated desc = failed to create Manila v2 client: failed to authenticate: Get "https://38.102.83.157:13000/": tls: failed to verify certificate: x509: certificate signed by unknown authority

From the csi-driver container of the clusters-0168087f07fe593bfc16 / openstack-manila-csi-controllerplugin-75db69649-sm6vj pod. That pod defines its volumes like so:

apiVersion: v1
kind: Pod
...
spec:
  ...
  volumes:
    ...
    - name: cloud-credentials
      secret:
        defaultMode: 420
        items:
          - key: clouds.yaml
            path: clouds.yaml
        secretName: manila-cloud-credentials
    - configMap:
        defaultMode: 420
        items:
          - key: ca-bundle.pem
            path: ca-bundle.pem
        name: openstack-cloud-config
        optional: true
      name: cacert

I am attempting to fix this in openshift/hypershift#6777 but the approach I've taken is wrong (per that LLM-generated warning) so I'll need to take another punt at it.

stephenfin · 2025-09-16T11:49:50Z

Weird. I deployed this locally and things appear to be working as-is. My pod definitions are the same:

apiVersion: v1
kind: Pod
metadata:
  # ...
  name: openstack-manila-csi-controllerplugin-5994b65bf7-q2x8x
  namespace: clusters-stephenfin-hcp
  # ...
spec:
  # ...
  volumes:
    # ...
    - name: cloud-credentials
      secret:
        defaultMode: 420
        items:
          - key: clouds.yaml
            path: clouds.yaml
        secretName: manila-cloud-credentials
    - configMap:
        defaultMode: 420
        items:
          - key: ca-bundle.pem
            path: ca-bundle.pem
        name: openstack-cloud-config
        optional: true
    # ...

I see the credentials in the correct place with the correct key:

❯ oc get -n clusters-stephenfin-hcp secret manila-cloud-credentials -o yaml
apiVersion: v1
data:
  cloud.conf: <redacted>
  clouds.yaml: <redacted>
kind: Secret
metadata:
  annotations:
    hypershift.openshift.io/cluster: clusters/stephenfin-hcp
  creationTimestamp: "2025-09-16T11:00:30Z"
  name: manila-cloud-credentials
  namespace: clusters-stephenfin-hcp
  resourceVersion: "2080159"
  uid: 5ba57428-8af4-4b04-9dae-acb9f55c2bf2
type: Opaque

I don't see a ca-bundle.pem key in the config map, but that's expected since the cloud I'm on isn't using self-signed certs. And it was in the correct place in the

❯ oc get -n clusters-stephenfin-hcp cm openstack-cloud-config -o yaml
apiVersion: v1
data:
  cloud.conf: |
    <redacted>
kind: ConfigMap
metadata:
  creationTimestamp: "2025-09-16T11:04:40Z"
  name: openstack-cloud-config
  namespace: clusters-stephenfin-hcp
  ownerReferences:
  - apiVersion: hypershift.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: HostedControlPlane
    name: stephenfin-hcp
    uid: 1ab35480-3265-44e9-aed2-d890112f3018
  resourceVersion: "2081779"
  uid: 72ab196c-85af-49c0-b58a-560d9ee54752

So I'm not really sure what's going on currently 😕 ~~I wonder if the CA cert is old?~~ Nah, the cert hasn't changed...

dkokkino · 2025-10-02T12:41:02Z

/retest e2e-openstack

openshift-ci · 2025-10-02T12:41:07Z

@dkokkino: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test aws-efs-operator-e2e

/test aws-efs-single-zone-cross-account-operator-e2e

/test aws-efs-single-zone-operator-e2e

/test ci-index-aws-efs-csi-driver-operator-bundle

/test ci-index-smb-csi-driver-operator-bundle

/test e2e-aws-csi

/test e2e-aws-ovn-upgrade

/test e2e-azure

/test e2e-azure-csi

/test e2e-azure-file-csi

/test e2e-azure-file-nfs-csi

/test e2e-azure-ovn-upgrade

/test e2e-openstack-cinder-csi

/test e2e-openstack-manila-csi

/test hypershift-aws-e2e-external

/test hypershift-e2e-aks

/test hypershift-e2e-openstack-aws-csi-cinder

/test hypershift-e2e-openstack-aws-csi-manila

/test images

/test okd-scos-images

/test unit

/test verify

/test verify-deps

The following commands are available to trigger optional jobs:

/test aws-efs-cross-account-operator-e2e

/test aws-efs-operator-e2e-extended

/test aws-efs-single-zone-operator-e2e-extended

/test aws-efs-sts-cross-account-operator-e2e

/test e2e-aws-csi-extended

/test e2e-azure-csi-extended

/test e2e-azure-file-csi-extended

/test e2e-azure-manual-oidc

/test e2e-azurestack-csi

/test e2e-openstack

/test okd-scos-e2e-aws-ovn

/test security

/test smb-operator-e2e

/test smb-operator-e2e-extended

/test smb-win2019-operator-e2e

/test smb-win2022-operator-e2e

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-csi-operator-main-aws-efs-operator-e2e

pull-ci-openshift-csi-operator-main-ci-index-aws-efs-csi-driver-operator-bundle

pull-ci-openshift-csi-operator-main-ci-index-smb-csi-driver-operator-bundle

pull-ci-openshift-csi-operator-main-e2e-aws-csi

pull-ci-openshift-csi-operator-main-e2e-aws-ovn-upgrade

pull-ci-openshift-csi-operator-main-e2e-azure

pull-ci-openshift-csi-operator-main-e2e-azure-csi

pull-ci-openshift-csi-operator-main-e2e-azure-file-csi

pull-ci-openshift-csi-operator-main-e2e-azure-file-nfs-csi

pull-ci-openshift-csi-operator-main-e2e-azure-ovn-upgrade

pull-ci-openshift-csi-operator-main-e2e-openstack

pull-ci-openshift-csi-operator-main-e2e-openstack-manila-csi

pull-ci-openshift-csi-operator-main-hypershift-aws-e2e-external

pull-ci-openshift-csi-operator-main-hypershift-e2e-aks

pull-ci-openshift-csi-operator-main-hypershift-e2e-openstack-aws-csi-manila

pull-ci-openshift-csi-operator-main-images

pull-ci-openshift-csi-operator-main-okd-scos-e2e-aws-ovn

pull-ci-openshift-csi-operator-main-okd-scos-images

pull-ci-openshift-csi-operator-main-security

pull-ci-openshift-csi-operator-main-smb-operator-e2e

pull-ci-openshift-csi-operator-main-unit

pull-ci-openshift-csi-operator-main-verify

pull-ci-openshift-csi-operator-main-verify-deps

In response to this:

/retest e2e-openstack

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dkokkino · 2025-10-02T12:41:25Z

/test e2e-openstack

dkokkino · 2025-10-02T12:57:07Z

/test hypershift-e2e-openstack-aws-csi-manila

openshift-ci · 2025-10-02T15:56:12Z

@dkokkino: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hypershift-e2e-openstack-aws-csi-manila	`a62014f`	link	true	`/test hypershift-e2e-openstack-aws-csi-manila`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 26, 2025

dkokkino marked this pull request as draft March 26, 2025 11:04

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2025

openshift-ci bot requested review from EmilienM and tsmetana March 26, 2025 11:06

stephenfin suggested changes Mar 27, 2025

View reviewed changes

openshift-ci bot assigned stephenfin Mar 27, 2025

dkokkino force-pushed the OSASINFRA-3675 branch 3 times, most recently from 825f7b0 to f7a25a0 Compare March 28, 2025 16:45

dkokkino marked this pull request as ready for review March 28, 2025 16:47

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 28, 2025

openshift-ci bot requested review from jsafrane and RomanBednar March 28, 2025 16:48

stephenfin suggested changes Apr 1, 2025

View reviewed changes

dkokkino force-pushed the OSASINFRA-3675 branch 2 times, most recently from 2a4970a to 99ce2f5 Compare April 7, 2025 08:23

dkokkino force-pushed the OSASINFRA-3675 branch 4 times, most recently from 54a63e7 to c83cfb2 Compare April 10, 2025 14:50

mandre reviewed Apr 25, 2025

View reviewed changes

assets/overlays/openstack-manila/base/config_secret.yaml Show resolved Hide resolved

dkokkino force-pushed the OSASINFRA-3675 branch 2 times, most recently from 8dca79f to cd2eca5 Compare April 29, 2025 12:13

dkokkino added 3 commits August 6, 2025 14:38

Update manila assets for new authentication method

aa89352

- Adding clouds.yaml as a secret mount volume. This will be used for authentication by the manila driver to match how cinder does authentication.

Modify secret generator

faaa25b

- Goal is to change how the aunthentication for the manila driver occurs.Changes the fields generated in the secret csi-manila-secrets

mandre force-pushed the OSASINFRA-3675 branch from cd2eca5 to a62014f Compare August 6, 2025 12:39

stephenfin mentioned this pull request Aug 20, 2025

OSASINFRA-3675: Add credentials to openshift-manila-csi-driver namespace for Hypershift openshift/hypershift#6683

Merged

4 tasks

openshift-ci bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Aug 21, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2025

	data["os-clouds-file"] = []byte("etc/openstack/clouds.yaml")
	data["os-clouds-file"] = []byte("/etc/openstack/clouds.yaml")

	data["os-use-clouds"] = []byte(strconv.FormatBool(true))
	data["os-use-clouds"] = []byte("true")

OSASINFRA-3675: Rework authentication in Manila CSI Driver Operator #373

Are you sure you want to change the base?

OSASINFRA-3675: Rework authentication in Manila CSI Driver Operator #373

Uh oh!

Conversation

dkokkino commented Mar 26, 2025

Uh oh!

openshift-ci-robot commented Mar 26, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stephenfin Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stephenfin Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

stephenfin Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

stephenfin left a comment

Choose a reason for hiding this comment

Uh oh!

stephenfin Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

stephenfin Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

pierreprinetti commented Apr 24, 2025

Uh oh!

Uh oh!

dkokkino commented May 7, 2025

Uh oh!

dkokkino commented Jul 16, 2025

Uh oh!

stephenfin commented Jul 22, 2025

Uh oh!

dkokkino commented Jul 25, 2025

Uh oh!

MiguelCarpio commented Aug 6, 2025

Uh oh!

mandre commented Aug 7, 2025

Uh oh!

dkokkino commented Aug 12, 2025

Uh oh!

MiguelCarpio commented Aug 13, 2025

Uh oh!

dkokkino commented Aug 14, 2025

Uh oh!

dkokkino commented Aug 18, 2025

Uh oh!

dkokkino commented Aug 18, 2025

Uh oh!

MiguelCarpio commented Aug 19, 2025

Uh oh!

stephenfin commented Aug 21, 2025

Uh oh!

openshift-ci bot commented Aug 21, 2025

Uh oh!

mandre commented Aug 22, 2025

Uh oh!

stephenfin commented Aug 22, 2025

Uh oh!

dkokkino commented Aug 22, 2025

Uh oh!

dkokkino commented Aug 22, 2025

Uh oh!

dkokkino commented Aug 22, 2025

Uh oh!

stephenfin commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stephenfin commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkokkino commented Oct 2, 2025

Uh oh!

openshift-ci-robot commented Mar 26, 2025 •

edited by openshift-ci bot

Loading

stephenfin Mar 27, 2025 •

edited

Loading

stephenfin commented Sep 16, 2025 •

edited

Loading

stephenfin commented Sep 16, 2025 •

edited

Loading