fix:Duplicate DeepCSV path removal, onnx model update #48256

uttiyasarkar · 2025-06-05T18:18:17Z

PR description:

Following the presentation in HLT upgrade meeting here dated 03.06.2025, we spotted a bug on the current Phase-2 DeepJet onnx model.
It has a missing Softmax layer in the last step.
The PR is created in cms-data with the fixed model: here
Accordingly, the path names are updated in the python modules.

This PR also includes the removal of "duplicate" BTV paths from the HLT Phase2 simplified menu. Specifically:

These are same as DeepFlavour. These DeepCSV paths have already been removed from the timing variant of the simplified Menu.
This change makes the the simplified menu aligned to the timing variant.

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

cmsbuild · 2025-06-05T18:18:40Z

@uttiyasarkar, CMSSW_15_1_X branch is closed for direct updates. cms-bot is going to move this PR to master branch.
In future, please use cmssw master branch to submit your changes.

cmsbuild · 2025-06-05T18:19:01Z

cms-bot internal usage

cmsbuild · 2025-06-05T18:20:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48256/45072

There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/paths/HLT_PFHT200PT30_QuadPFPuppiJet_70_40_30_30_TriplePFPuppiBTagDeepCSV_2p4_cfi.py modified in PR(s): TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859
- File HLTrigger/Configuration/python/HLT_75e33/paths/HLT_PFHT330PT30_QuadPFPuppiJet_75_60_45_40_TriplePFPuppiBTagDeepCSV_2p4_cfi.py modified in PR(s): TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): A More Flexible And Lightweight CA #47611, TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859

cmsbuild · 2025-06-05T18:20:33Z

A new Pull Request was created by @uttiyasarkar for master.

It involves the following packages:

HLTrigger/Configuration (hlt)

@Martin-Grunewald, @cmsbuild, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @SohamBhattacharya, @VourMa, @missirol, @mmusich, @rovere this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2025-06-05T18:28:48Z

@uttiyasarkar don't forget to remove the paths also from the schedule:

cmssw/HLTrigger/Configuration/python/HLT_75e33_cff.py

Lines 331 to 332 in 7dbf44b

    
           fragment.HLT_PFHT330PT30_QuadPFPuppiJet_75_60_45_40_TriplePFPuppiBTagDeepCSV_2p4, 
        
           fragment.HLT_PFHT200PT30_QuadPFPuppiJet_70_40_30_30_TriplePFPuppiBTagDeepCSV_2p4,

cmsbuild · 2025-06-05T18:38:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48256/45073

There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/HLT_75e33/paths/HLT_PFHT200PT30_QuadPFPuppiJet_70_40_30_30_TriplePFPuppiBTagDeepCSV_2p4_cfi.py modified in PR(s): TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859
- File HLTrigger/Configuration/python/HLT_75e33/paths/HLT_PFHT330PT30_QuadPFPuppiJet_75_60_45_40_TriplePFPuppiBTagDeepCSV_2p4_cfi.py modified in PR(s): TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859
- File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): A More Flexible And Lightweight CA #47611, TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859

cmsbuild · 2025-06-05T18:39:03Z

Pull request #48256 was updated. @Martin-Grunewald, @cmsbuild, @mmusich can you please check and sign again.

mmusich · 2025-06-05T18:44:20Z

test parameters:

pull_request = fix:Phase2 onnx model with added Softmax layer cms-data/RecoBTag-Combined#69
enable = hlt_p2_integration, hlt_p2_timing
workflow_opts= -w upgrade
workflow = 29634.75, 29634.751, 29634.752, 29634.753, 29634.754, 29634.755, 29634.756, 29634.7561, 29634.7562, 29634.757, 29634.77, 29634.759, 29634.771

mmusich · 2025-06-05T18:45:56Z

I am going to wait for the next IB to trigger the tests, to allow having #48253 in the baseline.

mmusich · 2025-06-06T08:11:18Z

@cmsbuild, please test

cmsbuild · 2025-06-06T11:24:09Z

+1

Size: This PR adds an extra 44KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46574/summary.html
COMMIT: c1b43e4
CMSSW: CMSSW_15_1_X_2025-06-05-2300/el8_amd64_gcc12
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/48256/46574/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@Dr15Jones Use steady_clock directly in some services #48259

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46574/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46574/git-merge-result

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 19 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 74 differences found in the comparisons
DQMHistoTests: Total files compared: 60
DQMHistoTests: Total histograms compared: 4304812
DQMHistoTests: Total failures: 26
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4304766
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 59 files compared)
Checked 249 log files, 210 edm output root files, 60 DQM output files
TriggerResults: found differences in 13 / 58 workflows

mmusich · 2025-06-09T08:08:15Z

@cmsbuild, please test

cmsbuild · 2025-06-09T11:05:46Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46606/summary.html
COMMIT: c1b43e4
CMSSW: CMSSW_15_1_X_2025-06-08-2300/el8_amd64_gcc12
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/48256/46606/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46606/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f4cd1f/46606/git-merge-result

HLT P2 Timing: chart

Comparison Summary

There are some workflows for which there are errors in the baseline:
29634.757 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 50 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 72 differences found in the comparisons
DQMHistoTests: Total files compared: 60
DQMHistoTests: Total histograms compared: 4304812
DQMHistoTests: Total failures: 16
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4304776
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 59 files compared)
Checked 249 log files, 210 edm output root files, 60 DQM output files
TriggerResults: found differences in 13 / 58 workflows

mmusich · 2025-06-09T11:24:02Z

Timing change is negligible:

Baseline	This PR

mmusich · 2025-06-09T11:30:39Z

I tried the nanoAOD infrastructure introduce by @elenavernazza at #48091 using the following command:

#!/bin/bash -ex

FILES='file:/eos/cms/store/relval/CMSSW_15_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_Run4D110_PU-v1/2590000/e84a1036-8765-44b5-bf8b-0cb1e4e5cd15.root,file:/eos/cms/store/
relval/CMSSW_15_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_Run4D110_PU-v1/2590000/eaf3b2fc-1fc3-47fc-999f-764b98a7e808.root,file:/eos/cms/store/relval/CMSSW_15_1_0_pre3/Re
lValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realistic_v1_STD_Run4D110_PU-v1/2590000/eb1c9fb1-1f11-4ec5-88de-ea710dffcc43.root,file:/eos/cms/store/relval/CMSSW_15_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIG
I-RAW/PU_150X_mcRun4_realistic_v1_STD_Run4D110_PU-v1/2590000/ef0c6943-4a9b-4eb8-bf1c-44bcad41d9e6.root,file:/eos/cms/store/relval/CMSSW_15_1_0_pre3/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_150X_mcRun4_realis
tic_v1_STD_Run4D110_PU-v1/2590000/f5197ef6-f432-4b4c-a61a-9dd570c9167c.root'

# generate the command
cmsDriver.py Phase2 -s L1P2GT,HLT:NGTScouting,NANO:@NGTScouting \
             --processName=HLTX \
             --conditions auto:phase2_realistic_T33 \
             --geometry ExtendedRun4D110 \
             --era Phase2C17I13M9 \
             --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000 \
             --datatier NANOAODSIM \
             --eventcontent NANOAODSIM \
             --filein=$FILES \
             --mc \
             --inputCommands="keep *, drop *_hlt*_*_HLT, drop triggerTriggerFilterObjectWithRefs_l1t*_*_HLT" \
             -n -1 \
	     --customise_commands='process.options.wantSummary = True' \
             --nThreads 24 \
	     --no_exec

cmsRun Phase2_L1P2GT_HLT_NANO.py &> hlt.log

Here are the results I obtain for the probb score variable

cmssw/HLTrigger/NGTScouting/python/hltJets_cfi.py

Line 25 in f556d3f

    
           DeepFlavour_prob_b = ExtVar(cms.InputTag("hltPfDeepFlavourJetTags:probb"), float, doc="DeepFlavour probability of b", precision=10),

before (baseline CMSSW_15_1_X_2025-06-08-2300)
this PR (fix:Duplicate DeepCSV path removal, onnx model update #48256)

This seems to be inline with the results shown at the HLT ugprade meeting of June 3rd, 2025

mmusich · 2025-06-09T11:38:13Z

+hlt

for uttiyasarkar@c1b43e4
unfortunately it's not possible to assess the impact on the BTV object reconstruction as there is no dedicated Phase-2 BTV DQM nor validation sequences in place (@cms-sw/btv-pog-l2 please take note, see also Add Validation and DQM to Phase2 HLT WF #39362).
Here is a list of observations:
- timing change is negligible, see fix:Duplicate DeepCSV path removal, onnx model update #48256 (comment)
- change in the shape of the scores seems inline with expectations: fix:Duplicate DeepCSV path removal, onnx model update #48256 (comment),
- changes in the TriggerResults: found differences in 13 / 58 workflows seem as expected, generally speaking there is a reduction of rate from the following paths:
  - HLT_PFHT330PT30_QuadPFPuppiJet_75_60_45_40_TriplePFPuppiBTagDeepFlavour_2p4
  - HLT_PFHT200PT30_QuadPFPuppiJet_70_40_30_30_TriplePFPuppiBTagDeepFlavour_2p4
  - HLT_DoublePFPuppiJets128_DoublePFPuppiBTagDeepFlavour_2p4
this PR must be integrated with the companion cms-data PR: fix:Phase2 onnx model with added Softmax layer cms-data/RecoBTag-Combined#69

cmsbuild · 2025-06-09T11:38:36Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-data/RecoBTag-Combined#69

mmusich · 2025-06-09T14:09:39Z

type bug-fix

mandrenguyen · 2025-06-10T13:31:56Z

+1

rovere · 2025-06-11T15:25:21Z

Timing change is negligible:
Baseline This PR

For the record: there is no visible change in timing because the DeepCSV paths were already removed from the timing menu long ago.

mmusich · 2025-06-11T15:26:44Z

For the record: there is no visible change in timing because the DeepCSV paths were already removed from the timing menu long ago.

I wasn't expect any from those. I was actually checking that the new model doesn't slow down the current menu ;)

fix:Duplicate DeepCSV path removal, onnx model update

7dbf44b

cmsbuild changed the base branch from CMSSW_15_1_X to master June 5, 2025 18:18

cmsbuild added this to the CMSSW_15_1_X milestone Jun 5, 2025

cmsbuild added hlt-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jun 5, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 5, 2025

fix:removal from the scheduler

c1b43e4

cmsbuild added code-checks-pending and removed code-checks-approved labels Jun 5, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 5, 2025

cmsbuild added the requires-external label Jun 5, 2025

cmsbuild added tests-started and removed tests-pending labels Jun 6, 2025

cmsbuild mentioned this pull request Jun 6, 2025

TICL-barrel: run CLUE in the barrel calorimeters and first workflows #47859

Merged

cmsbuild added tests-approved and removed tests-started labels Jun 6, 2025

mmusich mentioned this pull request Jun 6, 2025

fix:Phase2 onnx model with added Softmax layer cms-data/RecoBTag-Combined#69

Merged

cmsbuild added tests-started and removed tests-approved labels Jun 9, 2025

cmsbuild added tests-approved and removed tests-started labels Jun 9, 2025

cmsbuild added hlt-approved fully-signed and removed hlt-pending pending-signatures labels Jun 9, 2025

cmsbuild added the bug-fix label Jun 9, 2025

cmsbuild added orp-approved and removed orp-pending labels Jun 10, 2025

cmsbuild merged commit 03f77d9 into cms-sw:master Jun 10, 2025
13 checks passed

cmsbuild mentioned this pull request Jun 10, 2025

Updated ROOT geometry version compatibility #48288

Merged

sinonkt mentioned this pull request Jun 10, 2025

update CRABClient dmwm/CRABServer#9110

Closed

3 tasks

cmsbuild mentioned this pull request Jun 11, 2025

[ROOT] Updated root to tip of branch master cms-sw/cmsdist#9917

Merged

cmsbuild mentioned this pull request Jun 16, 2025

[GCC14] Python packages update cms-sw/cmsdist#9931

Closed

fix:Duplicate DeepCSV path removal, onnx model update #48256

fix:Duplicate DeepCSV path removal, onnx model update #48256

Uh oh!

Conversation

uttiyasarkar commented Jun 5, 2025

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Jun 5, 2025

Uh oh!

cmsbuild commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 5, 2025

Uh oh!

cmsbuild commented Jun 5, 2025

Uh oh!

mmusich commented Jun 5, 2025

Uh oh!

cmsbuild commented Jun 5, 2025

Uh oh!

cmsbuild commented Jun 5, 2025

Uh oh!

mmusich commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Jun 5, 2025

Uh oh!

mmusich commented Jun 6, 2025

Uh oh!

cmsbuild commented Jun 6, 2025

Comparison Summary

Uh oh!

mmusich commented Jun 9, 2025

Uh oh!

cmsbuild commented Jun 9, 2025

Comparison Summary

Uh oh!

mmusich commented Jun 9, 2025

Uh oh!

mmusich commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 9, 2025

Uh oh!

mmusich commented Jun 9, 2025

Uh oh!

mandrenguyen commented Jun 10, 2025

Uh oh!

Uh oh!

rovere commented Jun 11, 2025

Uh oh!

mmusich commented Jun 11, 2025

Uh oh!

Uh oh!

cmsbuild commented Jun 5, 2025 •

edited

Loading

mmusich commented Jun 5, 2025 •

edited

Loading

mmusich commented Jun 9, 2025 •

edited

Loading

mmusich commented Jun 9, 2025 •

edited

Loading