Empty commit, to trigger new RelVals #47708

fwyzard · 2025-03-27T06:10:28Z

PR description:

Empty commit, to trigger new RelVals.

PR validation:

None.

fwyzard · 2025-03-27T06:13:43Z

test parameters:

enable = gpu
workflows_gpu = 141.044407,141.044483,16850.403,16850.404,17034.403,17034.408,17034.422,17034.424,17050.406,17050.407,17050.408,29834.402,29834.403

cmsbuild · 2025-03-27T06:17:41Z

cms-bot internal usage

cmsbuild · 2025-03-27T06:20:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47708/44267

cmsbuild · 2025-03-27T06:20:40Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

Empty/Package (****)

The following packages do not have a category, yet:

Empty/Package
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@cmsbuild can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

fwyzard · 2025-03-27T06:23:02Z

please test

cmsbuild · 2025-03-27T09:46:52Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/45227/summary.html
COMMIT: 90adbe2
CMSSW: CMSSW_15_1_X_2025-03-26-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47708/45227/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3909207
DQMHistoTests: Total failures: 70
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3909117
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

ROCM Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

fwyzard · 2025-03-27T10:38:19Z

@iarspider are there differences between how we run the ROCm tests on LUMI for the pull requests and for the IBs ?

The IB from last night showed that many ROCm tests failed:

while the tests in this PR all passed:

12834.402_TTbar_14TeV+2024_Patatrack_PixelOnlyAlpaka Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:14 2025-date Thu Mar 27 09:07:23 2025; exit: 0 0 0 0
12834.403_TTbar_14TeV+2024_Patatrack_PixelOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:15 2025-date Thu Mar 27 09:07:23 2025; exit: 0 0 0 0
12834.406_TTbar_14TeV+2024_Patatrack_PixelOnlyTripletsAlpaka Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:15 2025-date Thu Mar 27 09:07:24 2025; exit: 0 0 0 0
12834.412_TTbar_14TeV+2024_Patatrack_ECALOnlyAlpaka Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:14 2025-date Thu Mar 27 09:07:24 2025; exit: 0 0 0 0
12834.422_TTbar_14TeV+2024_Patatrack_HCALOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:12 2025-date Thu Mar 27 09:07:25 2025; exit: 0 0 0 0
12834.423_TTbar_14TeV+2024_Patatrack_HCALOnlyGPUandAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:17:14 2025-date Thu Mar 27 09:07:25 2025; exit: 0 0 0 0
6 6 6 6 tests passed, 0 0 0 0 failed
141.044407_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:22:48 2025-date Thu Mar 27 09:17:45 2025; exit: 0 0 0 0
141.044483_Run3-2023_JetMET2023D_GPUValidation Step0-PASSED Step1-PASSED Step2-PASSED  - time date Thu Mar 27 09:22:08 2025-date Thu Mar 27 09:17:46 2025; exit: 0 0 0
16850.403_ZMM_14+2025_Patatrack_PixelOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:23:38 2025-date Thu Mar 27 09:17:46 2025; exit: 0 0 0 0
16850.404_ZMM_14+2025_Patatrack_PixelOnlyAlpaka_Profiling Step0-PASSED Step1-PASSED Step2-PASSED  - time date Thu Mar 27 09:22:42 2025-date Thu Mar 27 09:17:47 2025; exit: 0 0 0
17034.403_TTbar_14TeV+2025PU_Patatrack_PixelOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 10:01:19 2025-date Thu Mar 27 09:17:47 2025; exit: 0 0 0 0
17034.408_TTbar_14TeV+2025PU_Patatrack_PixelOnlyTripletsAlpaka_Profiling Step0-PASSED Step1-PASSED Step2-PASSED  - time date Thu Mar 27 09:55:17 2025-date Thu Mar 27 09:17:48 2025; exit: 0 0 0
17034.422_TTbar_14TeV+2025PU_Patatrack_HCALOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 10:00:21 2025-date Thu Mar 27 09:17:48 2025; exit: 0 0 0 0
17034.424_TTbar_14TeV+2025PU_Patatrack_HCALOnlyAlpaka_Profiling Step0-PASSED Step1-PASSED Step2-PASSED  - time date Thu Mar 27 09:58:13 2025-date Thu Mar 27 09:17:49 2025; exit: 0 0 0
17050.406_ZMM_14+2025PU_Patatrack_PixelOnlyTripletsAlpaka Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 10:00:38 2025-date Thu Mar 27 09:22:08 2025; exit: 0 0 0 0
17050.407_ZMM_14+2025PU_Patatrack_PixelOnlyTripletsAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 09:57:19 2025-date Thu Mar 27 09:22:42 2025; exit: 0 0 0 0
17050.408_ZMM_14+2025PU_Patatrack_PixelOnlyTripletsAlpaka_Profiling Step0-PASSED Step1-PASSED Step2-PASSED  - time date Thu Mar 27 09:54:55 2025-date Thu Mar 27 09:22:49 2025; exit: 0 0 0
29834.402_TTbar_14TeV+Run4D110PU_Patatrack_PixelOnlyAlpaka Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 10:01:36 2025-date Thu Mar 27 09:23:38 2025; exit: 0 0 0 0
29834.403_TTbar_14TeV+Run4D110PU_Patatrack_PixelOnlyAlpaka_Validation Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Mar 27 11:33:24 2025-date Thu Mar 27 09:54:56 2025; exit: 0 0 0 0
13 13 13 8 tests passed, 0 0 0 0 failed

fwyzard · 2025-03-27T10:38:27Z

One difference that I see in the logs is that the IB tests still use all GPUs for each test:

%MSG-i ROCmService:  (NoModuleName) 27-Mar-2025 03:57:12 EET pre-events
ROCm runtime version 6.3.42134, driver version 6.3.42134, AMD driver version 6.3.6
ROCm device 0: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 1: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 2: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 3: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 4: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 5: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 6: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
ROCm device 7: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
%MSG

while the PR tests use a single GPU per job:

%MSG-i ROCmService:  (NoModuleName) 27-Mar-2025 09:21:07 EET pre-events
ROCm runtime version 6.3.42134, driver version 6.3.42134, AMD driver version 6.3.6
ROCm device 0: AMD Instinct MI250X (gfx90a:sramecc+:xnack-)
%MSG

iarspider · 2025-03-27T12:55:45Z

I think the main difference is that PR tests are run in single-threaded mode

fwyzard · 2025-03-27T20:56:02Z

I think the main difference is that PR tests are run in single-threaded mode

But how does that explain why a job would see 1 GPU vs 8 GPUs ?

smuzaffar · 2025-04-03T08:04:02Z

test parameters:

enable = rocm
workflow_rocm = 141.044406,141.044408,16850.402,17034.404,17034.408,17050.404,17050.408,29834.404
workflow_option_rocm = -t 4

smuzaffar · 2025-04-03T08:05:01Z

please test

smuzaffar · 2025-04-03T11:50:40Z

please test

cmsbuild · 2025-04-03T14:47:11Z

-1

Failed Tests: RelVals-ROCM
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/45362/summary.html
COMMIT: 90adbe2
CMSSW: CMSSW_15_1_X_2025-04-03-1100/el8_amd64_gcc12
Additional Tests: ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47708/45362/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 5 lines from the logs
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3913985
DQMHistoTests: Total failures: 52
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3913913
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

fwyzard · 2025-06-22T09:46:22Z

please test

cmsbuild · 2025-06-22T09:47:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47708/45273

cmsbuild · 2025-06-22T09:48:06Z

Pull request #47708 was updated.

cmsbuild · 2025-06-22T14:25:03Z

-1

Failed Tests: RelVals-ROCM
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/46868/summary.html
COMMIT: 0756b8f
CMSSW: CMSSW_15_1_X_2025-06-22-0000/el8_amd64_gcc12
Additional Tests: ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47708/46868/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-ROCM

141.044408141.044408_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling/step3_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling.log
16850.40216850.402_ZMM_14+2025_Patatrack_PixelOnlyAlpaka/step3_ZMM_14+2025_Patatrack_PixelOnlyAlpaka.log
17034.40417034.404_TTbar_14TeV+2025PU_Patatrack_PixelOnlyAlpaka_Profiling/step3_TTbar_14TeV+2025PU_Patatrack_PixelOnlyAlpaka_Profiling.log

Expand to see more relval errors ...

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4058157
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4058128
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

fwyzard · 2025-07-04T08:04:33Z

please test

cmsbuild · 2025-07-04T08:05:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47708/45405

cmsbuild · 2025-07-04T08:06:13Z

Pull request #47708 was updated.

cmsbuild · 2025-07-04T15:16:12Z

-1

Failed Tests: RelVals-ROCM
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/47079/summary.html
COMMIT: 0c7fa9c
CMSSW: CMSSW_15_1_X_2025-07-03-2300/el8_amd64_gcc12
Additional Tests: ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47708/47079/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/47079/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bf8465/47079/git-merge-result

RelVals-ROCM

17034.41217034.412_TTbar_14TeV+2025PU_Patatrack_ECALOnlyAlpaka/step2_TTbar_14TeV+2025PU_Patatrack_ECALOnlyAlpaka.log
17034.40617034.406_TTbar_14TeV+2025PU_Patatrack_PixelOnlyTripletsAlpaka/step3_TTbar_14TeV+2025PU_Patatrack_PixelOnlyTripletsAlpaka.log
17034.40417034.404_TTbar_14TeV+2025PU_Patatrack_PixelOnlyAlpaka_Profiling/step3_TTbar_14TeV+2025PU_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

You potentially removed 98 lines from the logs
Reco comparison results: 62695 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4067195
DQMHistoTests: Total failures: 561287
DQMHistoTests: Total nulls: 410
DQMHistoTests: Total successes: 3505478
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 6.096 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): 2.651 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 17034.0 ): 2.583 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.133 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.846 KiB SiStrip/MechanicalView
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: found differences in 15 / 48 workflows

fwyzard force-pushed the empty branch from 64bd649 to 90adbe2 Compare March 27, 2025 06:17

cmsbuild added this to the CMSSW_15_1_X milestone Mar 27, 2025

cmsbuild added pending-signatures tests-pending orp-pending new-package-pending code-checks-pending labels Mar 27, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 27, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 27, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 27, 2025

cmsbuild added tests-started and removed tests-approved labels Apr 3, 2025

smuzaffar mentioned this pull request Apr 3, 2025

[ROCM_X] Multiple RelVals failing in ROCM_X IB #46624

Open

cmsbuild added tests-pending code-checks-pending and removed tests-started code-checks-approved labels Jun 22, 2025

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 22, 2025

cmsbuild added tests-rejected and removed tests-started labels Jun 22, 2025

Empty commit, to trigger new RelVals

0c7fa9c

fwyzard force-pushed the empty branch from 0756b8f to 0c7fa9c Compare July 4, 2025 08:03

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Jul 4, 2025

cmsbuild added tests-started and removed tests-pending labels Jul 4, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 4, 2025

cmsbuild added tests-rejected and removed tests-started labels Jul 4, 2025

Empty commit, to trigger new RelVals #47708

Are you sure you want to change the base?

Empty commit, to trigger new RelVals #47708

Conversation

fwyzard commented Mar 27, 2025

PR description:

PR validation:

Uh oh!

fwyzard commented Mar 27, 2025

Uh oh!

cmsbuild commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Mar 27, 2025

Uh oh!

cmsbuild commented Mar 27, 2025

Uh oh!

fwyzard commented Mar 27, 2025

Uh oh!

cmsbuild commented Mar 27, 2025

Comparison Summary

CUDA Comparison Summary

ROCM Comparison Summary

Uh oh!

fwyzard commented Mar 27, 2025

Uh oh!

fwyzard commented Mar 27, 2025

Uh oh!

iarspider commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Mar 27, 2025

Uh oh!

smuzaffar commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Apr 3, 2025

Uh oh!

smuzaffar commented Apr 3, 2025

Uh oh!

cmsbuild commented Apr 3, 2025

Comparison Summary

Uh oh!

fwyzard commented Jun 22, 2025

Uh oh!

cmsbuild commented Jun 22, 2025

Uh oh!

cmsbuild commented Jun 22, 2025

Uh oh!

cmsbuild commented Jun 22, 2025

RelVals-ROCM

Comparison Summary

Uh oh!

fwyzard commented Jul 4, 2025

Uh oh!

cmsbuild commented Jul 4, 2025

Uh oh!

cmsbuild commented Jul 4, 2025

Uh oh!

cmsbuild commented Jul 4, 2025

RelVals-ROCM

Comparison Summary

Uh oh!

Uh oh!

cmsbuild commented Mar 27, 2025 •

edited

Loading

iarspider commented Mar 27, 2025 •

edited

Loading

smuzaffar commented Apr 3, 2025 •

edited

Loading