mkFit related non reproducibility of Run-3 HLT tracking results in recent IBs

[mmusich]

After the merge of cms-sw/cmssw#48437 I am seeing in several recent Pull Requests random changes in the trigger results and changes in the tracking HLT DQM, for changes that should not imply any:
Few examples:

• Phase2-hgx364S Modify some parameters for the no cell option for the V19 scenario #48428 (comment)

  • tested with CMSSW_15_1_X_2025-06-29-0000 (has the problem)

• Run3_sim164 Create separate flags for HcalTestNumbering for signal and PU samples to enable different sources for these 2 types of samples #48439 (comment)

  • tested with CMSSW_15_1_X_2025-06-29-0000 (has the problem)

• apply stage2 L1T customizations to TrackToTrackComparisonHists #48440 (comment)

  • tested with CMSSW_15_1_X_2025-06-29-0000 (has the problem)

• apply stage2 L1T customizations to TrackToTrackComparisonHists #48440 (comment)

  • tested with CMSSW_15_1_X_2025-06-29-2300 (has the problem)

Given that:

• Extend NanoAOD for NGT Phase 2 Scouting with Tau and Gen information #48425 (comment)
  • tested with CMSSW_15_1_X_2025-06-26-2300 (looks OK)

I tend to think the non-reproducibility issue comes from the merge of cms-sw/cmssw#48437 that entered CMSSW_15_1_X_2025-06-29-0000.

As the differences in results seem always to be associated with tracking and on-track tracker local reco quantities it's not inconceivable that the issue stems from the integration of one of the recent HLT tracking updates, for instance

• CMSHLT-3570 [TRK] HLT-Tracking updates for 2025 menu V1.2: mkFit for doublet recovery

[mmusich]

assign hlt, tracking-pog

[mmusich]

type tracking

[cmsbuild]

New categories assigned: hlt,tracking-pog

@elusian,@kskovpen,@Martin-Grunewald,@mmasciov,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @mmusich.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

Using the same input data and HLT paths as some of the workflows that failed the trigger results comparisons at #48440 (comment) I tried to reproduce the issue with the following scripts at this commit, unsuccessfully - so far.

[slava77]

is this possibly intel vs AMD (or a different hardware otherwise) case for the bot/PR tests?
having a reproducer would be nice.

[mmusich]

is this possibly intel vs AMD (or a different hardware otherwise) case for the bot/PR tests?

even if this was verified, this is not acceptable.

having a reproducer would be nice.

I think @elusian has reproduced it (he might add more details). Running any Run3 workflow running the HLT menu (be it data or MC) would suffice.

[slava77]

even if this was verified, this is not acceptable.

I agree, I was just looking for a way to reproduce