Skip to content

OCPEDGE-2606: e2e: add fencing credentials update and validation tests for dual-replica etcd#31199

Open
fracappa wants to merge 1 commit into
openshift:mainfrom
fracappa:fca/update-fencing-credentials-test
Open

OCPEDGE-2606: e2e: add fencing credentials update and validation tests for dual-replica etcd#31199
fracappa wants to merge 1 commit into
openshift:mainfrom
fracappa:fca/update-fencing-credentials-test

Conversation

@fracappa
Copy link
Copy Markdown
Contributor

@fracappa fracappa commented May 20, 2026

Add a new e2e test that rotates BMC fencing credentials via the Redfish API, runs update-fencing-credentials-sh wiuth the new password, and verifies fencing still works with updated credentials. Introduces Redfish API helpers and fencing credentials discovery in the apis package.

Summary by CodeRabbit

  • Tests
    • Added an end-to-end test that updates fencing credentials, validates fencing behavior, triggers network disruption, and verifies node recovery and role stability; restores original credentials after testing.
    • Enhanced test utilities to locate fencing credentials, rotate BMC passwords via Redfish, and validate BMC credentials (including SSL handling and IPv6 formatting).

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented May 20, 2026

@fracappa: This pull request references OCPEDGE-2606 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Add a new e2e test that rotates BMC fencing credentials via the Redfish API, runs update-fencing-credentials-sh wiuth the new password, and verifies fencing still works with updated credentials. Introduces Redfish API helpers and fencing credentials discovery in the apis package.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

Walkthrough

This pull request adds infrastructure and an integration test for rotating BMC (baseboard management controller) fencing credentials during etcd cluster recovery scenarios. It introduces Redfish API helpers for password changes, utilities to locate credentials in Kubernetes secrets, and an orchestrated test that validates credential rotation, fencing tool compatibility, and cluster recovery.

Changes

Fencing Credential Rotation and Recovery Testing

Layer / File(s) Summary
Redfish API helpers for BMC operations
test/extended/edge_topologies/utils/apis/redfish.go
Parses redfish+ addresses to extract host/port/path with scheme-based default ports, discovers target BMC account by username, changes account password via Redfish AccountService PATCH request, and validates credentials work by executing fence_redfish --action status with IPv6 bracket formatting and optional SSL bypass.
Kubernetes fencing credentials lookup
test/extended/edge_topologies/utils/apis/baremetalhost.go
Adds FencingCredentials struct and FindFencingCredentialsByNodeName helper to list and extract address, username, password, and certificate verification fields from secrets in the fencing-credentials namespace by matching secret name prefix and node short name.
Fencing credential rotation and recovery test
test/extended/edge_topologies/tnf_recovery.go
New Ginkgo test case that locates current fencing credentials, parses Redfish details, rotates BMC password, validates credentials, runs the credential update script, checks Pacemaker status, triggers network disruption, confirms etcd cluster recovery without assuming initial leader role, and restores original credentials in cleanup.

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 12
✅ Passed checks (12 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding e2e tests for fencing credentials update and validation for dual-replica etcd.
Docstring Coverage ✅ Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed New test name is a static, literal string with no dynamic values, sprintf, string concatenation, node names, or other identifiers that change between test runs.
Test Structure And Quality ✅ Passed Test code meets all 5 requirements: single responsibility, proper cleanup via DeferCleanup, timeouts on cluster operations, meaningful assertion messages, and codebase pattern consistency.
Microshift Test Compatibility ✅ Passed Test wrapped with [apigroup:config.openshift.io] tag; MicroShift CI jobs auto-skip tests with unavailable API group tags. Infrastructure API usage is protected.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The new test is protected from SNO by topology check in BeforeEach: utils.SkipIfNotTopology(oc, v1.DualReplicaTopologyMode) ensures test only runs on 2-node clusters.
Topology-Aware Scheduling Compatibility ✅ Passed PR contains only test code and test utilities (no deployment manifests, operators, or controllers). No scheduling constraints are introduced that would affect topology compatibility.
Ote Binary Stdout Contract ✅ Passed All output writes occur within test blocks or use g.GinkgoWriter. No process-level stdout writes detected; no BeforeSuite/AfterSuite functions or problematic init code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Uses net.JoinHostPort() and IPv6 bracket notation for URLs. Curl calls are cluster-internal to BMC endpoints only. No hardcoded IPv4 addresses, IPv4 parsing, or external internet connectivity.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from jaypoulz and qJkee May 20, 2026 13:57
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 20, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fracappa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 20, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/edge_topologies/utils/apis/baremetalhost.go`:
- Around line 51-57: The FencingCredentials return builds values from
secret.Data without validating required keys, causing silent empty fields;
update the code around the return of &FencingCredentials (the secret variable
and FencingCredentials construction) to explicitly check that secret.Data
contains non-empty "address", "username", and "password" keys and return a clear
error identifying any missing/empty key(s) instead of returning empty strings;
treat "certificateVerification" as optional (or validate similarly if desired)
and ensure the function returns an error when required secret fields are absent
so callers get immediate, descriptive diagnostics.
- Line 50: The current selector uses strings.Contains(secret.Name, shortName)
which can falsely match names like "master-01" for "master-0"; change the
condition in the selection block (the if using strings.HasPrefix, secret.Name,
shortName, and fencingCredentialsPrefix) to require an exact name match (e.g.
construct expectedName := fencingCredentialsPrefix + shortName and compare
secret.Name == expectedName) or apply a strict regex with explicit boundaries
anchored to the full secret.Name so only the intended host secret is selected.

In `@test/extended/edge_topologies/utils/apis/redfish.go`:
- Around line 17-20: The code currently uses strings.TrimPrefix on address which
silently accepts non-`redfish+` inputs; change the logic to explicitly require
the "redfish+" prefix by checking strings.HasPrefix(address, "redfish+") and
returning a clear error if it's missing, then strip the prefix and parse the
remainder (replace the current use of strings.TrimPrefix and the following
url.Parse call accordingly); ensure the returned error message identifies the
input as non-`redfish+` rather than failing later during url.Parse.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a51572b5-235f-40f7-86f5-bbbacdeee8fe

📥 Commits

Reviewing files that changed from the base of the PR and between 428b9a0 and 00087f1.

📒 Files selected for processing (3)
  • test/extended/edge_topologies/tnf_recovery.go
  • test/extended/edge_topologies/utils/apis/baremetalhost.go
  • test/extended/edge_topologies/utils/apis/redfish.go

Comment thread test/extended/edge_topologies/utils/apis/baremetalhost.go Outdated
Comment thread test/extended/edge_topologies/utils/apis/baremetalhost.go
Comment thread test/extended/edge_topologies/utils/apis/redfish.go
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

…lica etcd

Add a new e2e test that rotates BMC fencing credentials via the Redfish API,
runs update-fencing-credentials-sh wiuth the new password, and verifies fencing
still works with updated credentials. Introduces Redfish API helpers and fencing
credentials discovery in the apis package.
@fracappa fracappa force-pushed the fca/update-fencing-credentials-test branch from 2c5b983 to c32869d Compare May 21, 2026 10:09
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/extended/edge_topologies/utils/apis/redfish.go (1)

59-68: 💤 Low value

Scheme information is lost; HTTPS is hardcoded.

ParseRedfishAddress parses both redfish+http:// and redfish+https:// addresses (with appropriate default ports), but this function only receives host and port, then hardcodes https:// at line 63. If a redfish+http://... address is ever used, the curl call would fail.

In practice, BMC Redfish endpoints are virtually always HTTPS, so this is likely acceptable. Consider either documenting this assumption or extending ParseRedfishAddress to also return the scheme for completeness.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/edge_topologies/utils/apis/redfish.go` around lines 59 - 68,
The function ChangeBMCPasswordViaRedfish currently hardcodes https:// when
building baseURL; preserve/propagate the scheme returned by ParseRedfishAddress
(or extend ParseRedfishAddress to return scheme) and use it when constructing
baseURL instead of "https://"; update ChangeBMCPasswordViaRedfish (and its
calls) to accept or derive scheme (so baseURL := fmt.Sprintf("%s://%s", scheme,
authority)), and ensure findRedfishAccountByUsername is called with that
baseURL; alternatively document the HTTPS-only assumption if you choose not to
support http.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/extended/edge_topologies/utils/apis/redfish.go`:
- Around line 59-68: The function ChangeBMCPasswordViaRedfish currently
hardcodes https:// when building baseURL; preserve/propagate the scheme returned
by ParseRedfishAddress (or extend ParseRedfishAddress to return scheme) and use
it when constructing baseURL instead of "https://"; update
ChangeBMCPasswordViaRedfish (and its calls) to accept or derive scheme (so
baseURL := fmt.Sprintf("%s://%s", scheme, authority)), and ensure
findRedfishAccountByUsername is called with that baseURL; alternatively document
the HTTPS-only assumption if you choose not to support http.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 56db3364-d4c5-48b9-8382-e2256b844b0a

📥 Commits

Reviewing files that changed from the base of the PR and between 00087f1 and c32869d.

📒 Files selected for processing (3)
  • test/extended/edge_topologies/tnf_recovery.go
  • test/extended/edge_topologies/utils/apis/baremetalhost.go
  • test/extended/edge_topologies/utils/apis/redfish.go

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label May 21, 2026
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

@fracappa
Copy link
Copy Markdown
Contributor Author

/retest-required

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

@fracappa: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ovn-two-node-fencing c32869d link false /test e2e-metal-ovn-two-node-fencing
ci/prow/e2e-aws-csi c32869d link true /test e2e-aws-csi
ci/prow/e2e-metal-ovn-two-node-fencing-recovery c32869d link false /test e2e-metal-ovn-two-node-fencing-recovery

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants