Skip to content

Publish SDK installer artifacts in CI pipeline #52233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Copilot wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Publish SDK installer artifacts in CI pipeline #52233

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9529521
Initial plan
Copilot Dec 17, 2025
2aa64af
Add step to publish SDK artifacts in CI pipeline
Copilot Dec 17, 2025
3161a3d
Add descriptive comment for SDK artifact publishing
Copilot Dec 17, 2025
d7cc8d2
Include retry number in SDK artifact name to avoid collisions
Copilot Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions eng/pipelines/templates/jobs/sdk-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,17 @@ jobs:
BuildConfig: $(buildConfiguration)
OPENSSL_ENABLE_SHA1_SIGNATURES: 1

############### PUBLISH SDK ARTIFACTS ###############
# Publishes the built SDK installer artifacts (zip, tar.gz, pkg, exe, msi, deb, rpm) to pipeline artifacts.
# This enables validation and troubleshooting, particularly for PRs where local build output isn't available.
- task: ${{ parameters.oneESCompat.publishTaskPrefix }}PublishPipelineArtifact@1
displayName: 🟣 Publish SDK Artifacts
inputs:
targetPath: $(Build.SourcesDirectory)/artifacts/packages/$(buildConfiguration)/Shipping
artifactName: $(System.PhaseName)_SDK
continueOnError: true
condition: succeededOrFailed()

Copy link
Member

@MiYanni MiYanni Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, this will double publish. Meaning, adding this here will cause all builds to publish these assets. Internal builds already publish these assets via the enablePublishBuildAssets and the publish-build-assets.yml template:

sdk/.vsts-ci.yml

Lines 291 to 306 in 2b7468d

############### PUBLISH STAGE ###############
- ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
- stage: publish
displayName: Publish
dependsOn: []
jobs:
- template: /eng/common/templates-official/job/publish-build-assets.yml@self
parameters:
publishUsingPipelines: true
publishAssetsImmediately: true
isAssetlessBuild: true
repositoryAlias: self
pool:
name: $(DncEngInternalBuildPool)
image: 1es-windows-2022
os: windows

Wait, reading this, we moved to isAssetlessBuild. @ViktorHofer Does the internal build publish assets? Because adding this will now publish assets and I believed our goal was to no longer publish assets.

Copy link
Member

@MiYanni MiYanni Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assetless-build PR for reference: #49113

Copy link
Member

@MiYanni MiYanni Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also wondering if we can just use the Arcade mechanism to publish assets instead of doing it ourselves. Meaning, use the publish-build-assets.yml template instead in the .vsts-pr.yml pipeline directly. Then, it would only happen in external PR builds.

Copy link
Member

@ViktorHofer ViktorHofer Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, reading this, we moved to isAssetlessBuild. @ViktorHofer Does the internal build publish assets? Because adding this will now publish assets and I believed our goal was to no longer publish assets.

Yes, sdk shouldn't publish anything in the official assetless build.

Copy link
Member

@ViktorHofer ViktorHofer Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, I see that sdk still builds in the official build. I think that was necessary back a few months ago when we didn't preserve intermediates for compliance scanning tools but that got solved meanwhile. Most/all? other repos that enabled assetless builds don't build anymore.

https://dev.azure.com/dnceng/internal/_build/results?buildId=2863248&view=results

@mmitche do you remember if that was indeed the reason when you made the switch?

Copy link
Member

@ViktorHofer ViktorHofer Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, OneLocBuild should still run in individual repo's builds. Compliance/SDL now runs centrally in the VMR and there's an owner to transfers items to the individual teams. Regarding shipping, no - sdk shouldn't ship any artifacts. I'm pretty sure it doesn't ship anything today as isAssetlessBuild is set to true which skips all that.

Copy link
Member

@MiYanni MiYanni Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compliance/SDL now runs centrally in the VMR and there's an owner to transfers items to the individual teams

Unless the VMR copies a repo's pipeline configuration for SDL/Compliance, I would not trust SDL tool runs from the VMR as they're likely misconfigured. For example, BinSkim requires these glob filters to run properly on this repo:

sdk/eng/pipelines/templates/jobs/sdk-build.yml

Lines 60 to 63 in 6085c30

templateContext:
sdl:
binskim:
analyzeTargetGlob: +:f|eng\**\*.props;+:f|artifacts\bin\**\*.dll;+:f|artifacts\bin\**\*.exe;-:f|artifacts\bin\**\msdia140.dll;-:f|artifacts\bin\**\pgort140.dll;-:f|artifacts\bin\*Tests\**;-:f|**\Microsoft.NET.Runtime.Emscripten**\tools\**;-:f|**\CodeCoverage\**;-:f|artifacts\bin\**\capstone.dll;

Other repos would have specific files for filters or other logic in specific files within the repo that would be declared in the sdl section of the 1ES template. This repo doesn't have anything like that but it would exist here:

sdk/.vsts-ci.yml

Lines 73 to 87 in 6085c30

sdl:
sourceAnalysisPool:
name: $(DncEngInternalBuildPool)
image: 1es-windows-2022
os: windows
policheck:
enabled: true
tsa:
enabled: true
binskim:
enabled: true
${{ if or(eq(parameters.runTestBuild, true), eq(variables['Build.Reason'], 'PullRequest')) }}:
componentgovernance:
# Refdoc: https://docs.opensource.microsoft.com/tools/cg/component-detection/variables/
ignoreDirectories: artifacts, .packages

@RikkiGibson You're still shipping the source package you added as part of our official build pipeline, right?

Copy link
Member

@mmitche mmitche Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source package should also be produced from the VMR and should be shipped from there. Likely there's no material difference, but the VMR is the source of truth

Copy link
Member

@ViktorHofer ViktorHofer Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless the VMR copies a repo's pipeline configuration for SDL/Compliance, I would not trust SDL tool runs from the VMR as they're likely misconfigured. For example, BinSkim requires these glob filters to run properly on this repo:

The VMR is correctly configured. We have an active stream of binskim / CG issues. We are in close contact with the compliance folks in Prague an Redmond and are using a preview feature that does operate on nuget packages and archives instead which makes the above glob unnecessary. @marcpopMSFT knows about all this.

Copy link
Member

@mmitche mmitche Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MiYanni Binkskim in the VMR scans the output artifacts that are uploaded, rather than the. It utilizes binskim's newer functionality to unpack archives. I don't think it needs to be run in the SDK build any longer.

############### TESTING ###############
- ${{ if eq(parameters.runTests, true) }}:
- ${{ if eq(parameters.runAoTTests, true) }}:
Expand Down
Loading