Migrate LST outputs to Portable SoAs #48409

ariostas · 2025-06-25T15:19:38Z

This PR refactors the LST outputs so that Portable SoAs are used as much as possible. The output of the LST Producer is now a device collection, and the framework takes care of copying it to the host.

This continues the work from #47793, now on the outputs side, and completes the tasks related to CMSSW-LST interfacing in #46746.

c.c. @slava77

cmsbuild · 2025-06-25T15:20:02Z

cms-bot internal usage

cmsbuild · 2025-06-25T15:21:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48409/45307

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/src/classes_def.xml modified in PR(s): SiStripClusterizer, an Alpaka port of the CUDA ClustersFromRawProducerGPU #47629
- File RecoTracker/LSTCore/src/alpaka/LST.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377
- File RecoTracker/LSTCore/src/alpaka/LSTEvent.dev.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377

cmsbuild · 2025-06-25T15:22:06Z

A new Pull Request was created by @ariostas for master.

It involves the following packages:

DataFormats/Common (core)
RecoTracker/LST (reconstruction)
RecoTracker/LSTCore (reconstruction)

@Dr15Jones, @cmsbuild, @jfernan2, @makortel, @mandrenguyen, @smuzaffar can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @VinInn, @VourMa, @dgulhan, @felicepantaleo, @gpetruc, @makortel, @missirol, @mmusich, @mtosi, @rovere, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

slava77 · 2025-06-25T16:02:52Z

test parameters:

enable_tests = gpu
workflows_gpu = 29634.704,29834.704
workflows = 29634.703,29834.703,29834.755,29634.757,29834.757
relvals_opt = -w upgrade,standard
relvals_opt_gpu = -w upgrade,standard

slava77 · 2025-06-25T16:10:37Z

@cmsbuild please test

cmsbuild · 2025-06-25T16:25:06Z

-1

Failed Tests: Build HeaderConsistency
Size: This PR adds an extra 64KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c9102/46917/summary.html
COMMIT: 21a525e
CMSSW: CMSSW_15_1_X_2025-06-25-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48409/46917/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

Copying tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreROCmAsync/libRecoTrackerLSTCoreROCmAsync_rocm.a to productstore area:
Copying tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreCudaAsync/libRecoTrackerLSTCoreCudaAsync_nv.a to productstore area:
cp: cannot stat 'tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreROCmAsync/libRecoTrackerLSTCoreROCmAsync_rocm.a': No such file or directory
cp: cannot stat 'tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreCudaAsync/libRecoTrackerLSTCoreCudaAsync_nv.a': No such file or directory
>> Deleted: tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreCudaAsync/libRecoTrackerLSTCoreCudaAsync_nv.a
gmake: *** [config/SCRAM/GMake/Makefile.rules:1920: tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreCudaAsync/libRecoTrackerLSTCoreCudaAsync_nv.a] Error 1
>> Deleted: tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreROCmAsync/libRecoTrackerLSTCoreROCmAsync_rocm.a
gmake: *** [config/SCRAM/GMake/Makefile.rules:1920: tmp/el8_amd64_gcc12/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreROCmAsync/libRecoTrackerLSTCoreROCmAsync_rocm.a] Error 1
@@@@ Checking for missing symbols was SKIPPED due to NO_LIB_CHECKING flag in BuildFile: libUtilitiesStaticAnalyzers.so
Unknow target lib/el8_amd64_gcc12/RecoTrackerLSTCore_xr.rootmap
Unknow target lib/el8_amd64_gcc12/RecoTrackerLST_xr.rootmap

ariostas · 2025-06-25T17:02:49Z

Sorry about that, that header was deleted by mistake.

cmsbuild · 2025-06-25T17:03:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48409/45311

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/src/classes_def.xml modified in PR(s): SiStripClusterizer, an Alpaka port of the CUDA ClustersFromRawProducerGPU #47629
- File RecoTracker/LSTCore/src/alpaka/LST.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377
- File RecoTracker/LSTCore/src/alpaka/LSTEvent.dev.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377

slava77 · 2025-06-26T12:58:37Z

@slava77 Matti is on vacation and will not be back till the 2nd week of July.

OK, summer time.

In the meantime (while we wait for the main reviewer(s)) it would be nice to get advice/clarification on the plugin issue
#48409 (comment)

Dr15Jones · 2025-06-26T13:13:50Z

For the other failures, could they be false positives? The LSTOutputConverter plugin was moved from plugins to plugins/alpaka, but kept the same name. So could it be finding the poisoned one first? Otherwise, I'm not sure what would need to be updated.

The plugin system used the LD_LIBRARY_PATH environment variable to decide the order in which to look for plugins (this follows how the OS looks for shared libraries). The local work environment's lib directory should be in LD_LIBRARY_PATH before the local work environment's poison directory therefore the new plugin should be found first.

Are you certain the new plugin is actually being built?

@smuzaffar any thoughts?

Dr15Jones · 2025-06-26T13:28:14Z

So I see that the following libraries are built

libRecoTrackerLSTPluginsPortableCudaAsync.so
libRecoTrackerLSTPluginsPortableROCmAsync.so
libRecoTrackerLSTPluginsPortableSerialSync.so

neither of these are an exact match for plugin-poisoned-RecoTrackerLSTPlugins.so as previously, I believe, the LSTOutputConverter was just in the plugins directory which would match the _poisoned` name.

In the conversion to an alpaka module, the module's type name would have been changed as well. It would no longer just be LSTOutputConverter. So in the configuration for these jobs, how is the type of the module specified?

Dr15Jones · 2025-06-26T13:32:57Z

So you need to change

cmssw/HLTrigger/Configuration/python/HLT_75e33/modules/hltInitialStepTrackCandidates_cfi.py

Line 27 in 3c9125e

_hltInitialStepTrackCandidatesLST = cms.EDProducer('LSTOutputConverter',

to

_hltInitialStepTrackCandidatesLST = cms.EDProducer('LSTOutputConverter@alpaka',

and do that for all configurations.

slava77 · 2025-06-26T13:48:02Z

So you need to change

Thank you, Chris. The issue is on our side after all.

Do you think there is a way to improve the error message to indicate that the plugin LSTOutputConverter does not exist? (I can't recall what the error message is if it never existed; if that one is more clear, perhaps it can be added as a possibility)

ariostas · 2025-06-26T13:56:21Z

Thanks for the help, Chris! Should be all working now.

cmsbuild · 2025-06-26T13:59:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48409/45328

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/src/classes_def.xml modified in PR(s): SiStripClusterizer, an Alpaka port of the CUDA ClustersFromRawProducerGPU #47629
- File RecoTracker/LSTCore/src/alpaka/LST.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377
- File RecoTracker/LSTCore/src/alpaka/LSTEvent.dev.cc modified in PR(s): Updated SoA View accessors from raw pointers to span #48377

cmsbuild · 2025-06-26T13:59:36Z

Pull request #48409 was updated. @Dr15Jones, @Martin-Grunewald, @cmsbuild, @fwyzard, @jfernan2, @makortel, @mandrenguyen, @mmusich, @smuzaffar can you please check and sign again.

slava77 · 2025-06-26T14:07:30Z

@cmsbuild please test

cmsbuild · 2025-06-26T18:53:55Z

+1

Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c9102/46936/summary.html
COMMIT: 83731f0
CMSSW: CMSSW_15_1_X_2025-06-26-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48409/46936/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 57
DQMHistoTests: Total histograms compared: 4446263
DQMHistoTests: Total failures: 3306
DQMHistoTests: Total nulls: 8
DQMHistoTests: Total successes: 4442929
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 56 files compared)
Checked 240 log files, 206 edm output root files, 57 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 243 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 4307
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 113319
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 8 files compared)
Checked 32 log files, 36 edm output root files, 9 DQM output files
TriggerResults: no differences found

ROCM Comparison Summary

Summary:

You potentially added 6 lines to the logs
Reco comparison results: 92 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 117626
DQMHistoTests: Total failures: 8904
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 108722
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 8 files compared)
Checked 32 log files, 36 edm output root files, 9 DQM output files

mmusich · 2025-06-26T19:10:22Z

+hlt

changes in HLTrigger packages are anyway minimal and linked to the discussion above.

jfernan2 · 2025-06-27T06:49:39Z

+1

makortel · 2025-07-03T20:29:21Z

RecoTracker/LST/plugins/alpaka/LSTOutputConverter.cc

+
+namespace ALPAKA_ACCELERATOR_NAMESPACE {
+
+  class LSTOutputConverter : public global::EDProducer<> {


What motivated moving the LSTOutputConverter from a "regular module" (in plugins) to an "alpaka module" (in plugins/alpaka and in ALPAKA_ACCELERATOR_NAMESPACE)?

By quick look I didn't see any backend-specific behavior, but maybe I just missed it.

cmsbuild added this to the CMSSW_15_1_X milestone Jun 25, 2025

cmsbuild added reconstruction-pending core-pending pending-signatures tests-pending orp-pending code-checks-pending tracking changes-dataformats labels Jun 25, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 25, 2025

cmsbuild added tests-started and removed tests-pending labels Jun 25, 2025

cmsbuild added tests-rejected and removed tests-started labels Jun 25, 2025

ariostas force-pushed the ariostas/lst_outputs branch from 21a525e to b8eabc3 Compare June 25, 2025 17:01

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Jun 25, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 25, 2025

Migrated LST outputs to SoAs

83731f0

ariostas force-pushed the ariostas/lst_outputs branch from 31df503 to 83731f0 Compare June 26, 2025 13:55

cmsbuild added hlt-pending tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Jun 26, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 26, 2025

cmsbuild added tests-started and removed tests-pending labels Jun 26, 2025

cmsbuild added tests-approved and removed tests-started labels Jun 26, 2025

cmsbuild added hlt-approved and removed hlt-pending labels Jun 26, 2025

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jun 27, 2025

makortel reviewed Jul 3, 2025

View reviewed changes


		namespace ALPAKA_ACCELERATOR_NAMESPACE {

		class LSTOutputConverter : public global::EDProducer<> {

Migrate LST outputs to Portable SoAs #48409

Are you sure you want to change the base?

Migrate LST outputs to Portable SoAs #48409

Conversation

ariostas commented Jun 25, 2025

Uh oh!

cmsbuild commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 25, 2025

Uh oh!

cmsbuild commented Jun 25, 2025

Uh oh!

slava77 commented Jun 25, 2025

Uh oh!

slava77 commented Jun 25, 2025

Uh oh!

cmsbuild commented Jun 25, 2025

Build

Uh oh!

ariostas commented Jun 25, 2025

Uh oh!

cmsbuild commented Jun 25, 2025

Uh oh!

slava77 commented Jun 26, 2025

Uh oh!

Dr15Jones commented Jun 26, 2025

Uh oh!

Dr15Jones commented Jun 26, 2025

Uh oh!

Dr15Jones commented Jun 26, 2025

Uh oh!

slava77 commented Jun 26, 2025

Uh oh!

ariostas commented Jun 26, 2025

Uh oh!

cmsbuild commented Jun 26, 2025

Uh oh!

cmsbuild commented Jun 26, 2025

Uh oh!

slava77 commented Jun 26, 2025

Uh oh!

cmsbuild commented Jun 26, 2025

Comparison Summary

CUDA Comparison Summary

ROCM Comparison Summary

Uh oh!

mmusich commented Jun 26, 2025

Uh oh!

jfernan2 commented Jun 27, 2025

Uh oh!

makortel Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmsbuild commented Jun 25, 2025 •

edited

Loading