[NGT] Introduce NanoAOD flavour for Phase 2 HLT #48091

elenavernazza · 2025-05-15T16:23:22Z

PR description:

The goal of this PR is to introduce a new custom NanoAOD flavour for Phase 2 HLT.
The format is compatible with both the Phase 2 standard menu and the NGT Scouting menu. It includes most of the relevant HLT objects (further developments are foreseen to include Taus and MET) :

Gen-level particles
Primary Vertices
Pixel Tracks
General Tracks
Electrons and Photons
Muons
PF candidates
Jets (with b-tagging score)

The NanoAOD production is triggered automatically when running "TTbar_14TeV" workflows with offsets:

.759 for the standard Phase 2 menu
.771 for the NGT Scouting menu

Otherwise, it can be triggered by adding NANO:@Phase2HLT (when running on the standard Phase 2 menu) or NANO:@NGTScouting (when running on the NGT Scouting menu) in the step2 cmsDriver command, as shown below.

E.g. for the standard Phase 2 menu

cmsDriver.py step2 \
-s L1P2GT,HLT:@relvalRun4,NANO:@Phase2HLT \
--conditions auto:phase2_realistic_T33 \
--datatier GEN-SIM-DIGI-RAW,NANOAODSIM \
--eventcontent FEVTDEBUGHLT,NANOAODSIM \
--geometry ExtendedRun4D110 \
--era Phase2C17I13M9 \
--filein file:step1.root -n 100 \
--fileout file:step2.root

E.g. for the standard NGT Scouting

cmsDriver.py step2 \
-s L1P2GT,HLT:NGTScouting,NANO:@NGTScouting \
--conditions auto:phase2_realistic_T33 \
--datatier GEN-SIM-DIGI-RAW,NANOAODSIM \
--eventcontent FEVTDEBUGHLT,NANOAODSIM \
--geometry ExtendedRun4D110 \
--era Phase2C17I13M9 \
--filein file:step1.root -n 100 \
--fileout file:step2.root

PR validation:

I've run the workflow 29834 with both .759 and .771 offsets. The step1 and step2 are produced correctly, NanoAOD branches are correctly filled.

cmsbuild · 2025-05-15T16:23:56Z

cms-bot internal usage

mmusich · 2025-05-15T16:25:09Z

type ngt

cmsbuild · 2025-05-15T16:25:30Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48091/44835

Found files with invalid states:
- HLTrigger/NGTScouting/python/HLTVertices_cff.py:
  - Added: 1bbcc30
  - Deleted: 1efe67e
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): CA Extension to strips #47090, TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859, Adapt pixel cpe algo to better handle broken clusters #47966
- File HLTrigger/Configuration/python/HLT_75e33/paths/DST_PFScouting_cfi.py modified in PR(s): TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859
- File RecoBTag/ONNXRuntime/plugins/DeepFlavourONNXJetTagsProducer.cc modified in PR(s): Extend onnxruntime gpu interface to producers using onnxruntime #39402

cmsbuild · 2025-05-15T16:25:36Z

A new Pull Request was created by @elenavernazza for master.

It involves the following packages:

Configuration/PyReleaseValidation (upgrade, pdmv)
HLTrigger/Configuration (hlt)
HLTrigger/NGTScouting (****)
PhysicsTools/NanoAOD (xpog)
RecoBTag/ONNXRuntime (reconstruction)

The following packages do not have a category, yet:

HLTrigger/NGTScouting
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@AdrianoDee, @Martin-Grunewald, @Moanwar, @cmsbuild, @DickyChant, @ftorrresd, @hqucms, @jfernan2, @mandrenguyen, @miquork, @mmusich, @srimanob, @subirsarkar can you please review it and eventually sign? Thanks.
@AlexDeMoor, @Martin-Grunewald, @Ming-Yan, @Senphy, @SohamBhattacharya, @VourMa, @andrzejnovak, @castaned, @fabiocos, @gpetruc, @makortel, @missirol, @mmusich, @rovere, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2025-05-15T16:28:21Z

test parameters:

relvals_opt = --what upgrade
workflows = 29834.759, 29834.771

mmusich · 2025-05-15T16:28:30Z

@cmsbuild, please test

cmsbuild · 2025-05-15T17:19:04Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Size: This PR adds an extra 224KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-634315/46170/summary.html
COMMIT: 9ab2dcf
CMSSW: CMSSW_15_1_X_2025-05-15-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48091/46170/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 4 errors in the following unit tests:

---> test test-btvNano-run had ERRORS
---> test test-btvNano-check had ERRORS
---> test test_MC_22_crosscheck had ERRORS
and more ...

RelVals

135.4

----- Begin Fatal Exception 15-May-2025 18:47:35 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'MINIAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODSIMoutput'
   [3] Prefetching for module PATJetSelector/'slimmedJetsAK8'
   [4] Prefetching for module PATJetUpdater/'updatedPatJetsTransientCorrectedSlimmedAK8DeepTags'
   [5] Prefetching for module BoostedJetONNXJetTagsProducer/'pfParticleNetFromMiniAODAK8JetTagsSlimmedAK8DeepTags'
   [6] Prefetching for module ParticleNetFeatureEvaluator/'pfParticleNetFromMiniAODAK8TagInfosSlimmedAK8DeepTags'
   [7] Prefetching for module PATTauHybridProducer/'slimmedTaus'
   [8] Prefetching for module PATJetSelector/'slimmedJetsPuppi'
   [9] Calling method for module PATJetUpdater/'updatedPatJetsTransientCorrectedSlimmedPuppiWithDeepTags'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::AssociationVector<edm::RefToBaseProd<reco::Jet>,std::vector<float>,edm::RefToBase<reco::Jet>,unsigned int,edm::helper::AssociationIdenticalKeyReference>
Looking for module label: pfDeepFlavourJetTagsSlimmedPuppiWithDeepTags
Looking for productInstanceName: probb

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

140.045

----- Begin Fatal Exception 15-May-2025 18:48:29 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 356381 lumi: 60 event: 49955554 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module JetAnalyzer/'jetDQMAnalyzerAk4PFCHSPuppiCleanedMiniAOD'
   [3] Prefetching for module PATJetSelector/'slimmedJetsPuppi'
   [4] Calling method for module PATJetUpdater/'updatedPatJetsTransientCorrectedSlimmedPuppiWithDeepTags'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::AssociationVector<edm::RefToBaseProd<reco::Jet>,std::vector<float>,edm::RefToBase<reco::Jet>,unsigned int,edm::helper::AssociationIdenticalKeyReference>
Looking for module label: pfDeepFlavourJetTagsSlimmedPuppiWithDeepTags
Looking for productInstanceName: probb

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

24834.911

----- Begin Fatal Exception 15-May-2025 19:13:19 CEST-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module JetAnalyzer/'jetDQMAnalyzerAk8PFPUPPICleanedMiniAOD'
   [3] Prefetching for module PATJetSelector/'slimmedJetsAK8'
   [4] Prefetching for module PATJetUpdater/'updatedPatJetsTransientCorrectedSlimmedAK8DeepTags'
   [5] Prefetching for module PATJetUpdater/'updatedPatJetsSlimmedAK8DeepTags'
   [6] Prefetching for module PATJetSlimmer/'slimmedJetsAK8NoDeepTags'
   [7] Prefetching for module JetSubstructurePacker/'packedPatJetsAK8'
   [8] Prefetching for module BoostedJetMerger/'slimmedJetsAK8PFPuppiSoftDropPacked'
   [9] Prefetching for module PATJetSelector/'selectedUpdatedPatJetsAK8PFPuppiSoftDropSubjetsSlimmedDeepFlavour'
   [10] Prefetching for module PATJetUpdater/'updatedPatJetsTransientCorrectedAK8PFPuppiSoftDropSubjetsSlimmedDeepFlavour'
   [11] Calling method for module DeepFlavourONNXJetTagsProducer/'pfDeepFlavourJetTagsAK8PFPuppiSoftDropSubjetsSlimmedDeepFlavour'
Exception Message:
ValueMap::Filler: handle and reference collections should the same size
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

2500.2252500.225_jmeNANOrePuppimc140X/step2_jmeNANOrePuppimc140X.log
2500.2242500.224_jmeNANOmc140X/step2_jmeNANOmc140X.log
4.64.6_MinimumBias2010A/step2_MinimumBias2010A.log

Expand to see more relval errors ...

mmusich · 2025-05-27T11:28:38Z

@cms-sw/xpog-l2 please review.

ftorrresd · 2025-06-02T23:18:34Z

PhysicsTools/NanoAOD/scripts/inspectNanoFile.py

+        <link rel="stylesheet" type="text/css" href="https://cms-nanoaod-integration.web.cern.ch/patsize.css" />
+        <script type="text/javascript" src="https://cms-nanoaod-integration.web.cern.ch/rgraph/RGraph.common.core.js"></script>
+        <script type="text/javascript" src="https://cms-nanoaod-integration.web.cern.ch/rgraph/RGraph.pie.js"></script>
+        <script type="text/javascript" src="https://cms-nanoaod-integration.web.cern.ch/rgraph/RGraph.common.dynamic.js"></script>
+        <script type="text/javascript" src="https://cms-nanoaod-integration.web.cern.ch/rgraph/RGraph.common.tooltips.js"></script>
+        <script type="text/javascript" src="https://cms-nanoaod-integration.web.cern.ch/rgraph/RGraph.common.key.js"></script>


https://cms-nanoaod-integration.web.cern.ch will decommissioned by November. Everything (rgraph included) is being moved to http://cms-xpog.docs.cern.ch.

Can be done in a future PR.

Hi @ftorrresd, thank you for the comment, we take note.
However, further NGT developments are foreseen (already presented to xpog here) and those are beging blocked by this PR (in review since 2 weeks). If you don't have other substantive comments to the overall structure, I would suggest you to sign and let this PR be merged, such that your comment could be acted upon in a follow-up PR.

ftorrresd · 2025-06-03T10:29:15Z

+1

cmsbuild · 2025-06-03T10:29:42Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-06-04T07:39:23Z

+1

artlbv · 2025-06-05T09:41:11Z

FYI I tried to run the NGTScouting wf and while the nano part is in autoNANO, the NGTScouting is missing from HLT:

cmssw/Configuration/HLT/python/autoHLT.py

Lines 4 to 19 in 9938fac

    
           autoHLT = { 
        
             'fake'       : 'Fake', 
        
             'fake1'      : 'Fake1', 
        
             'fake2'      : 'Fake2', 
        
             'relval50ns' : 'Fake', 
        
             'relval25ns' : 'Fake1', 
        
             'relval2016' : 'Fake2', 
        
             'relval2017' : 'Fake2', 
        
             'relval2018' : 'Fake2', 
        
             'relval2022' : 'Fake2', 
        
             'relval2023' : 'Fake2', 
        
             'relval2024' : 'Fake2', 
        
             'relval2025' : '2025v11', 
        
             'relvalRun4' : '75e33', 
        
             'test'       : 'GRun', 
        
           }

I get this error:

$ cmsDriver.py step2 \-s L1P2GT,HLT:@NGTScouting,NANO:@NGTScouting -n 10 \                                                                                                                                                                                          
--conditions auto:phase2_realistic_T33 \
--datatier NANOAODSIM \
--eventcontent NANOAODSIM \
--geometry ExtendedRun4D110 \
--era Phase2C17I13M9 \
--filein file:/eos/cms/store/mc/Phase2Spring24DIGIRECOMiniAOD/GluGluToHHTo4B_node_SM_TuneCP5_14TeV-amcatnlo-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/130000/03474707-bf67-466b-9886-8d6379bf66e9.root \
--fileout file:step2.root
L1P2GT,HLT:@NGTScouting,NANO:@NGTScouting,ENDJOB
We have determined that this is simulation (if not, rerun cmsDriver.py with --data)
entry file:/eos/cms/store/mc/Phase2Spring24DIGIRECOMiniAOD/GluGluToHHTo4B_node_SM_TuneCP5_14TeV-amcatnlo-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/130000/03474707-bf67-466b-9886-8d6379bf66e9.root
Step: L1P2GT Spec: 
Loading P2GT menu from L1Trigger.Configuration.Phase2GTMenus.step1_2024
Step: HLT Spec: ['@NGTScouting']
Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-06-04-2300/bin/el9_amd64_gcc12/cmsDriver.py", line 40, in <module>
    run()
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-06-04-2300/bin/el9_amd64_gcc12/cmsDriver.py", line 16, in run
    configBuilder.prepare()
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02892/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-06-04-2300/src/Configuration/Applications/python/ConfigBuilder.py", line 2340, in prepare
    self.addStandardSequences()
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02892/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-06-04-2300/src/Configuration/Applications/python/ConfigBuilder.py", line 859, in addStandardSequences
    getattr(self,"prepare_"+stepName)(stepSpec = '+'.join(stepSpec))
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02892/el9_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-06-04-2300/src/Configuration/Applications/python/ConfigBuilder.py", line 1679, in prepare_HLT
    raise ValueError('no HLT mapping key "%s" found in autoHLT' % key)
ValueError: no HLT mapping key "NGTScouting" found in autoHLT

For the normal HLT step I get this error but I guess I should check the HLT upgrade docs first before continuing :)

05-Jun-2025 11:35:53 CEST  Closed file file:/eos/cms/store/mc/Phase2Spring24DIGIRECOMiniAOD/GluGluToHHTo4B_node_SM_TuneCP5_14TeV-amcatnlo-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/130000/03474707-bf67-466b-9886-8d6379bf66e9.root
----- Begin Fatal Exception 05-Jun-2025 11:35:53 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
Exception Message:
EDAlias conflicts with data
A product of type 'EBDigiCollection'
with module label 'simEcalUnsuppressedDigis' and instance name ''
already exists.
----- End Fatal Exception -------------------------------------------------

mmusich · 2025-06-05T09:48:21Z

@artlbv

FYI I tried to run the NGTScouting wf and while the nano part is in autoNANO, the NGTScouting is missing from HLT:

There are recipes on how to run at #48091 (comment).
The NGTScouting key is intentionally missing from autoHLT.

elenavernazza · 2025-06-05T11:57:47Z

Hi @artlbv
Sorry I've realised there is a typo in the instructions.
It should be HLT:NGTScouting (not HLT:@NGTScouting), this is the reason for the error message.
I have updated the PR description, let me know if it works now. And thank you for your feedback!

cmsbuild added this to the CMSSW_15_1_X milestone May 15, 2025

cmsbuild added reconstruction-pending hlt-pending pending-signatures tests-pending orp-pending new-package-pending pdmv-pending upgrade-pending code-checks-pending xpog-pending labels May 15, 2025

cmsbuild added code-checks-approved ngt and removed code-checks-pending labels May 15, 2025

cmsbuild added tests-started and removed tests-pending labels May 15, 2025

cmsbuild added tests-rejected and removed tests-started labels May 15, 2025

mmusich force-pushed the ev_HLTNano branch from 9ab2dcf to 41f57bd Compare May 15, 2025 19:21

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels May 15, 2025

cmsbuild added pdmv-approved and removed pdmv-pending labels May 22, 2025

This was referenced May 22, 2025

Update FastSim 2024 Era, add 2025 workflow #48153

Merged

Introduce L1ScoutingNanoAOD for Run3 #48163

Draft

Phase2-hgx364 Make a new scenario Run4D121 using D116 + corrected RPC with the corresponding workflow 34434.0 #48165

Merged

ikrommyd mentioned this pull request May 26, 2025

Add Phase 2 HLT NanoAOD support scikit-hep/coffea#1339

Open

ftorrresd reviewed Jun 2, 2025

View reviewed changes

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels Jun 3, 2025

cmsbuild mentioned this pull request Jun 4, 2025

Phase2-hgx364H Add two new scenario for HGCal V19n and V19ng without cell declaration at G4 geometry level and removing the gaps #48238

Merged

cmsbuild added orp-approved and removed orp-pending labels Jun 4, 2025

cmsbuild merged commit d4d35b7 into cms-sw:master Jun 4, 2025
14 checks passed

mmusich deleted the ev_HLTNano branch June 4, 2025 07:52

cmsbuild mentioned this pull request Jun 5, 2025

[GCC13] Update version to 13.4.0 cms-sw/cmsdist#9914

Merged

This was referenced Jun 6, 2025

add phase-2 NANO HLT workflows to ph2_hlt short matrix #48262

Merged

fix:Duplicate DeepCSV path removal, onnx model update #48256

Merged

artlbv mentioned this pull request Jun 11, 2025

[Phase-2 HLT] Add hltHpsPFTau table to HLT Nano #48295

Merged

mmusich mentioned this pull request Jun 11, 2025

HLT Nano: fix typo in DeepFlavour variables #48298

Merged

mmusich mentioned this pull request Jun 26, 2025

Extend NanoAOD for NGT Phase 2 Scouting with Tau and Gen information #48425

Merged

elenavernazza mentioned this pull request Jul 2, 2025

[NGT] Fix filling of ValueMap for DeepJet scores for NanoAOD Phase2 HLT #48462

Merged

[NGT] Introduce NanoAOD flavour for Phase 2 HLT #48091

[NGT] Introduce NanoAOD flavour for Phase 2 HLT #48091

Uh oh!

Conversation

elenavernazza commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

Uh oh!

cmsbuild commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented May 15, 2025

Uh oh!

cmsbuild commented May 15, 2025

Uh oh!

cmsbuild commented May 15, 2025

Uh oh!

mmusich commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented May 15, 2025

Uh oh!

cmsbuild commented May 15, 2025

Unit Tests

RelVals

RelVals-INPUT

Uh oh!

mmusich commented May 27, 2025

Uh oh!

ftorrresd Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmusich Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ftorrresd commented Jun 3, 2025

Uh oh!

cmsbuild commented Jun 3, 2025

Uh oh!

mandrenguyen commented Jun 4, 2025

Uh oh!

Uh oh!

artlbv commented Jun 5, 2025

Uh oh!

mmusich commented Jun 5, 2025

Uh oh!

elenavernazza commented Jun 5, 2025

Uh oh!

Uh oh!

elenavernazza commented May 15, 2025 •

edited

Loading

cmsbuild commented May 15, 2025 •

edited

Loading

mmusich commented May 15, 2025 •

edited

Loading

ftorrresd Jun 2, 2025 •

edited

Loading

mmusich Jun 3, 2025 •

edited

Loading