[Fleet] Implement secrets in APM package policy editor #225956

criamico · 2025-07-01T08:51:35Z

Summary

Implement secrets in APM policy editor. This implementation mirrors what we currently have in Integrations' packagePolicyEditor.
❗ In order to work as expected, a new version of APM where secret_token has secret: true needs to be released

In Fleet:

Refactored package_policy_input_var_field in Fleet so the components are now decoupled and SecretInputField can be exported to be used by other plugins
Exported a new function useSecretsStorage that checks if the requirements to enable secrets are available: Fleet server needs to be available and to have at least version 8.0.0 - below that version fleet server doesn't have the capability to read secrets in package policies.

In APM:

Marked the secret_token key as secret: true and added a new type 'secret' to be able to correctly display any secret fields. I also had to pass down an additional property "packageInfo" to be able to check that an input var is marked as "secret" in the package manifest. This aligns with the general functionality in Integrations
Imported Fleet component above to render the secret components
Saving the var as secret works out of the box since the form "submit" is uses Fleet's apis

UI

With secret: true enabled and the correct version of Fleet server installed:

Fallback UI - displays a password field

View policy:

Testing

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

criamico · 2025-07-02T10:50:01Z

...onents/fleet_integration/apm_policy_form/settings_definition/agent_authorization_settings.ts

@@ -57,7 +57,7 @@ export function getAgentAuthorizationSettings(): SettingsRow[] {
      ],
    },
    {
-      type: 'text',
+      type: 'secret',


Changing this value immediately has the effect of using the "fallback" component that renders a EUiFieldPassword, but it doesn't affect the functionality - secret_token keeps being saved as plain text.

I think that this is acceptable but let me know if it's better to change it later when the new APM version with secret: true will be released.

criamico · 2025-07-02T13:25:52Z

@elasticmachine merge upstream

elasticmachine · 2025-07-02T15:26:07Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b368a3f

Failed CI Steps

FTR Configs #104

Test Failures

[job] [logs] FTR Configs #104 / alerting api integration security and spaces enabled - Group 2 Alerts update space_1_all at space1 should handle updates for a long running alert type without failing the underlying tasks due to invalidated ApiKey

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	1366	1369	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1431	1426	-5

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id	before	after	diff
`fleet`	5	3	-2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	2.6MB	2.6MB	+1.6KB
`fleet`	2.1MB	2.1MB	-3.0KB
total			-1.3KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	97	99	+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	166.7KB	169.7KB	+3.1KB

History

💔 Build #314851 failed 7643922

cc @criamico

elasticmachine · 2025-07-02T15:31:50Z

Pinging @elastic/fleet (Team:Fleet)

criamico · 2025-07-03T09:15:10Z

@elasticmachine merge upstream

elasticmachine · 2025-07-03T09:15:22Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

criamico · 2025-07-03T09:27:51Z

When the secret token is saved as secret, it breaks the ESS onboarding. What's the preferred way of addressing it? Replacing it with a "create api key" button like in serverless?

Current onboarding:

crespocarlos · 2025-07-03T09:37:36Z

...lity/plugins/apm/public/components/fleet_integration/apm_policy_form/settings_form/index.tsx

+  if (
+    packageInfo &&
+    packageInfo.policy_templates &&
+    packageInfo.policy_templates.length > 0 &&
+    'inputs' in packageInfo.policy_templates[0] &&
+    Array.isArray(packageInfo.policy_templates[0].inputs) &&
+    packageInfo.policy_templates[0].inputs.length > 0 &&
+    packageInfo.policy_templates[0].inputs[0].vars
+  ) {


Would this work instead?

Suggested change

if (

packageInfo &&

packageInfo.policy_templates &&

packageInfo.policy_templates.length > 0 &&

'inputs' in packageInfo.policy_templates[0] &&

Array.isArray(packageInfo.policy_templates[0].inputs) &&

packageInfo.policy_templates[0].inputs.length > 0 &&

packageInfo.policy_templates[0].inputs[0].vars

) {

if (

packageInfo?.policy_templates?.[0]?.inputs?.[0]?.vars

) {

it works per se if I shorten it but the linter is not happy. This looks a bit funky and I don't like it either, but the alternative is

if ( packageInfo && packageInfo.policy_templates && packageInfo.policy_templates.length > 0 && packageInfo.policy_templates[0].inputs.length > 0 && packageInfo.policy_templates[0].inputs[0].vars )

and some ts-ignore lines here and there. The reason is that inputs exists only in one of the types of this union type:
https://github.com/criamico/kibana/blob/a60bde8d65ce41f14aced142fb273457ef928e30/x-pack/platform/plugins/shared/fleet/common/types/models/epm.ts#L273-L275

so I had to add the type guard as well

Maybe we could break the condition? Just brainstorming ideas to try to simplify the condition.

const { policy_templates = [] } = packageInfo || {}; const policyTemplate = policy_templates[0]; const inputs = 'inputs' in policy_templates[0] ? policy_templates[0].inputs : undefined; if (inputs && Array.isArray(inputs) && inputs.length > 0 && inputs[0].vars) { }

I don't want to block the PR because of this though.

I broke down the condition as suggested :)

simitt · 2025-07-03T09:57:18Z

IMO showing a Create API Key button and nudging users to API Key makes sense for the onboarding flow.
However, there also needs to be a way to see the secret-token, otherwise they cannot configure it - how can that be retrieved after this change? Is this by querying directly from Elasticsearch?
cc @akhileshpok as the PM for APM Server for further input.

criamico · 2025-07-03T10:06:12Z

@simitt

how can that be retrieved after this change? Is this by querying directly from Elasticsearch?

I need to check if there's a way to retrieve the plain text value. Currently we don't do it anywhere in Fleet, we just send the value to the policy and fleet server will read it. The value that is returned in the package policy is like follows, we just see an id that is then internally used to retrieve the value:

 "secret_token": {
                        "value": {
                            "id": "dMlZy5cB3Iz3RJj917hN",
                            "isSecretRef": true
                        },
                        "type": "text"
                    }

Would it be an option to add a warning to the user that they need to write down this value when they enter it - similar to what we do in other areas of the product? I believe it's like that for the main token displayed in serverless onboarding.

...s/apm/public/components/fleet_integration/apm_policy_form/settings_form/form_row_setting.tsx

kpollich · 2025-07-03T14:13:44Z

However, there also needs to be a way to see the secret-token, otherwise they cannot configure it - how can that be retrieved after this change? Is this by querying directly from Elasticsearch?

Kibana cannot retrieve Fleet secrets by design - only Fleet Server can read these secret values. If there's a requirement that users can see the secret token, is just rendering these as an input with type=password enough? Fleet secrets guarantee that kibana_system or any other non-Fleet-Server-users cannot read back the secret once written, so the value of the secret cannot be leaked in dev tools or anywhere else in Kibana.

simitt · 2025-07-03T15:18:51Z

Yeah allowing them to read it back kind of defeats the purpose of making this a secret in the first place. Since this is going out in a minor release version, I just want to ensure that we are not breaking existing workflows.
There is only one secret-token for the apm-server, if we force users to configure a new secret-token, it would mean they have to reconfigure all their apps currently using it. IMO if they can fetch it directly from ES (as long as they have privileges for it) it should be good enough. But PM input would definitely be valued here as this change is pretty user facing.

kpollich · 2025-07-03T15:31:05Z

There is only one secret-token for the apm-server, if we force users to configure a new secret-token, it would mean they have to reconfigure all their apps currently using it.

I wonder if we can do the same thing we did when introducing Fleet Secrets initially here and have a "first time secret" flow where the current, plain-text value of the token is rendered with a warning that it will automatically be converted to a secret and will no longer be readable, so the user should copy it somewhere safe to maintain access to it. IIRC we have some code in Fleet to detect this with the policy saved objects, @criamico.

This might be tough with APM, though, since we automatically upgrade the package behind the scenes today. We want to introduce a user-facing step here where the user must consent to their token being "taken away" and turned into a secret. They need a chance to persist the value.

criamico · 2025-07-07T07:43:25Z

@elasticmachine merge upstream

elasticmachine · 2025-07-07T07:43:38Z

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

Click to trigger kibana-pull-request for this PR!
Click to trigger kibana-deploy-project-from-pr for this PR!
Click to trigger kibana-deploy-cloud-from-pr for this PR!

akhileshpok · 2025-07-07T07:58:46Z

@simitt - Agree that showing a Create API Key button and nudging users to API Key should be incorporated as part of the onboarding flow.
@kpollich - We should not force existing APM users to configure a new secret-token for the apm-server as part of this change. Instead, we should ask for user consent (with accompanying docs) before forcing this change.

criamico · 2025-07-07T13:32:22Z

@kpollich @akhileshpok @simitt I did a local test where I buit the apm version with secret_token marked as secret: true and marked the changelog entry with breaking_changes.
Running the package registry locally I then verified that we have logic in place to detect this type of changes when trying to upgrade the integration:

So I think that this can be a starting point to warn the user when the upgrade is presented.

In addition, I'm checking how to build some logic based on this comment:

I wonder if we can do the same thing we did when introducing Fleet Secrets initially here and have a "first time secret" flow where the current, plain-text value of the token is rendered with a warning that it will automatically be converted to a secret and will no longer be readable, so the user should copy it somewhere safe to maintain access to it.

kpollich · 2025-07-07T16:56:09Z

So I think that this can be a starting point to warn the user when the upgrade is presented.

+1 from me - I'm glad the work we did to add generic breaking change handling can be reused at least to some extent here. Since APM is auto-upgraded, I wonder if we could enhance this to flow to notify the user via a toast when the auto-upgrade failed due to a breaking change and direct them to the view you've screenshotted.

criamico added 2 commits June 30, 2025 10:15

[Fleet] Implement secrets in APM package policy editor

cf82a98

Add secrets component to APM form

fa91503

criamico self-assigned this Jul 1, 2025

criamico added 3 commits July 1, 2025 11:25

Merge main and solve conflicts

edd380f

export useSecretStorage

0fb53ba

Pass packageInfo to APM and use it to check if var is secret

7643922

criamico commented Jul 2, 2025

View reviewed changes

Fix linter

965b6ed

criamico added Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes v9.2.0 backport:skip This commit does not require backporting labels Jul 2, 2025

Merge branch 'main' into 224063_apm_secrets

b368a3f

criamico marked this pull request as ready for review July 2, 2025 15:31

criamico requested review from a team as code owners July 2, 2025 15:31

Merge branch 'main' into 224063_apm_secrets

a60bde8

botelastic bot added the Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team label Jul 3, 2025

crespocarlos reviewed Jul 3, 2025

View reviewed changes

Address code review comments

5d143dc

Merge branch 'main' into 224063_apm_secrets

94b93d3

criamico marked this pull request as draft July 7, 2025 07:43

[Fleet] Implement secrets in APM package policy editor #225956

Are you sure you want to change the base?

[Fleet] Implement secrets in APM package policy editor #225956

Conversation

criamico commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

UI

Testing

Checklist

Uh oh!

criamico Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

criamico commented Jul 2, 2025

Uh oh!

elasticmachine commented Jul 2, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Any counts in public APIs

Async chunks

Public APIs missing exports

Page load bundle

History

Uh oh!

elasticmachine commented Jul 2, 2025

Uh oh!

criamico commented Jul 3, 2025

Uh oh!

elasticmachine commented Jul 3, 2025

Uh oh!

criamico commented Jul 3, 2025

Uh oh!

crespocarlos Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

criamico Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

crespocarlos Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

criamico Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

simitt commented Jul 3, 2025

Uh oh!

criamico commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpollich commented Jul 3, 2025

Uh oh!

simitt commented Jul 3, 2025

Uh oh!

kpollich commented Jul 3, 2025

Uh oh!

criamico commented Jul 7, 2025

Uh oh!

elasticmachine commented Jul 7, 2025

Uh oh!

akhileshpok commented Jul 7, 2025

Uh oh!

criamico commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpollich commented Jul 7, 2025

Uh oh!

Uh oh!

criamico commented Jul 1, 2025 •

edited

Loading

criamico Jul 2, 2025 •

edited

Loading

criamico commented Jul 7, 2025 •

edited

Loading