Add KEDA autoscaling to microservices jsonnet by mapno · Pull Request #6970 · grafana/tempo

mapno · 2026-04-15T13:18:15Z

What this PR does:

Adds KEDA-based horizontal pod autoscaling support to the OSS microservices jsonnet library. These configurations have been running successfully in production.

Distributor: CPU-based autoscaling
Metrics-generator: CPU-based autoscaling
Backend-worker: Prometheus-based autoscaling on outstanding blocks
Block-builder: KEDA kubernetes-workload scaler matching live-store zone-a pod count

All scalers are disabled by default. Configured per component via _config.<component>.keda (matching the existing VPA and PDB patterns).

Which issue(s) this PR fixes:

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copilot

Pull request overview

Adds KEDA-based horizontal pod autoscaling support to the Tempo OSS microservices Jsonnet library, integrating KEDA ScaledObjects and wiring component-specific scaling defaults and tests.

Changes:

Introduces autoscaling.libsonnet to generate KEDA ScaledObjects for distributor, metrics-generator, backend-worker, and block-builder.
Wires autoscaling into the microservices library (imports, configmap adjustments for block-builder partitions) and updates Jsonnet test fixtures.
Vendors keda-libsonnet and updates jb lockfiles plus changelog entry.

Reviewed changes

Copilot reviewed 12 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
operations/jsonnet/microservices/test/test_autoscaling.jsonnet	Adds targeted Jsonnet assertions for autoscaling outputs.
operations/jsonnet/microservices/test/jsonnetfile.lock.json	Locks new Jsonnet test dependencies (KEDA + VPA libs).
operations/jsonnet/microservices/test/jsonnetfile.json	Adds KEDA + VPA Jsonnet dependencies for tests.
operations/jsonnet/microservices/test/environments/default/main.jsonnet	Enables distributor autoscaling in the default test environment.
operations/jsonnet/microservices/tempo.libsonnet	Includes the new autoscaling module in the composed library.
operations/jsonnet/microservices/jsonnetfile.lock.json	Locks KEDA dependency for the microservices Jsonnet library.
operations/jsonnet/microservices/jsonnetfile.json	Adds `keda-libsonnet` as a dependency for microservices Jsonnet.
operations/jsonnet/microservices/configmap.libsonnet	Adjusts block-builder Tempo config to set `partitions_per_instance` when autoscaling is enabled.
operations/jsonnet/microservices/autoscaling.libsonnet	Implements configurable KEDA ScaledObjects and related defaults/overrides.
operations/jsonnet/microservices/Makefile	Runs the new autoscaling Jsonnet test.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/main.libsonnet	Vendored KEDA lib entrypoint for compiled Jsonnet distribution.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/gen.libsonnet	Vendored KEDA generated package wrapper.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/_gen/keda/v1alpha1/triggerAuthentication.libsonnet	Vendored generated KEDA TriggerAuthentication bindings.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/_gen/keda/v1alpha1/scaledObject.libsonnet	Vendored generated KEDA ScaledObject bindings.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/_gen/keda/v1alpha1/main.libsonnet	Vendored generated KEDA v1alpha1 module.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/_gen/keda/v1alpha1/clusterTriggerAuthentication.libsonnet	Vendored generated KEDA ClusterTriggerAuthentication bindings.
operations/jsonnet-compiled/vendor/github.com/jsonnet-libs/keda-libsonnet/2.15/_gen/keda/main.libsonnet	Vendored generated KEDA root module.
operations/jsonnet-compiled/jsonnetfile.lock.json	Updates compiled Jsonnet lockfile to include KEDA dependency.
CHANGELOG.md	Adds user-facing changelog entry for KEDA autoscaling support.

Copilot · 2026-04-15T13:25:20Z

+  tempo_distributor_deployment+:
+    if $._config.autoscaling.distributor.enabled then $.removeReplicasFromSpec else {},
+


When autoscaling is enabled you remove spec.replicas from the Deployment/StatefulSet. Kubernetes defaults replicas to 1 when the field is omitted, which means components with min_replicas > 1 (e.g. distributor=2, backend_worker=3) can come up initially under-provisioned until KEDA reconciles. Is that transient state acceptable for these targets, or should the template set an initial replicas value (or offer a config toggle) to avoid starting below min_replicas?

This is expected and matches standard KEDA usage — the ScaledObject owns the replica count and reconciles quickly. The transient state is short-lived and acceptable for these components.

Copilot

Pull request overview

Copilot reviewed 10 out of 19 changed files in this pull request and generated 3 comments.

Copilot · 2026-04-15T13:43:44Z

+    + scaledObject.spec.withMinReplicaCount(config.min_replicas)
+    + scaledObject.spec.withMaxReplicaCount(config.max_replicas)
+    + scaledObject.spec.withPollingInterval(60)
+    + (


scaledObjectForController hard-codes pollingInterval to 60s, but several configs (e.g. distributor) describe much faster scaling periods (15s). With a 60s polling interval KEDA will only refresh trigger values once per minute, which can significantly delay scale-up/scale-down and makes the “every 15s” behavior unrealistic. Could we either (a) make polling interval configurable (per scaler or globally), or (b) align the comments/defaults so the intended responsiveness matches what the rendered ScaledObject will actually do?

The pollingInterval controls how often KEDA queries the trigger source, while periodSeconds in the HPA scaling policies controls how the HPA acts on the metrics it already has. They're independent — 60s polling with 15s HPA periods is intentional and working well for us.

Copilot

Pull request overview

Copilot reviewed 10 out of 19 changed files in this pull request and generated 1 comment.

zalegrala · 2026-04-15T16:59:01Z

+  local scaleDownBehavior = scaledObject.spec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown,
+
+  _config+:: {
+    autoscaling: {


Do we want to nest under autoscaling? It might be preferable for a flat structure which is something akin to _config.distributor.keda.enabled instead of nesting under autoscaling.

See here for similar k8s specific enablements.
https://github.com/grafana/tempo/blob/main/operations/jsonnet/microservices/vertical-pod-autoscaler.libsonnet
https://github.com/grafana/tempo/blob/main/operations/jsonnet/microservices/pod-disruption-budget.libsonnet

Good call. Refactored to _config.<component>.keda to match the VPA/PDB pattern. Also moved prometheus_address into backend_worker.keda since it's the only consumer.

Add KEDA ScaledObject definitions for distributor (CPU), metrics-generator (CPU), backend-worker (Prometheus outstanding blocks), and block-builder (kubernetes-workload scaler). These configurations have been running successfully in production. All disabled by default and fully configurable.

Matches VPA/PDB convention where each component owns its config subtree. Also moves prometheus_address into backend_worker.keda where it's used.

Copilot

Pull request overview

Copilot reviewed 10 out of 19 changed files in this pull request and generated 3 comments.

javiermolinar

LGTM

Copilot AI review requested due to automatic review settings April 15, 2026 13:18

Copilot started reviewing on behalf of mapno April 15, 2026 13:18 View session

mapno force-pushed the keda-autoscaling branch from 8184838 to 6af06d2 Compare April 15, 2026 13:19

Copilot AI reviewed Apr 15, 2026

View reviewed changes

mapno force-pushed the keda-autoscaling branch from 6af06d2 to 2642532 Compare April 15, 2026 13:26

Copilot AI review requested due to automatic review settings April 15, 2026 13:35

mapno force-pushed the keda-autoscaling branch from 2642532 to dfc1c85 Compare April 15, 2026 13:35

Copilot started reviewing on behalf of mapno April 15, 2026 13:36 View session

mapno force-pushed the keda-autoscaling branch from dfc1c85 to 067e7e5 Compare April 15, 2026 13:39

Copilot AI reviewed Apr 15, 2026

View reviewed changes

mapno marked this pull request as ready for review April 15, 2026 16:08

Copilot AI review requested due to automatic review settings April 15, 2026 16:08

mapno requested review from carles-grafana, electron0zero, ie-pham, javiermolinar, mattdurham, mdisibio, oleg-kozlyuk-grafana, ruslan-mikhailov, stoewer, yvrhdn and zalegrala as code owners April 15, 2026 16:08

Copilot started reviewing on behalf of mapno April 15, 2026 16:09 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

Comment thread operations/jsonnet/microservices/autoscaling.libsonnet

zalegrala reviewed Apr 15, 2026

View reviewed changes

mapno added 3 commits April 20, 2026 16:50

Add header comment and document scale_down_pods sentinel behavior

6958c05

Refactor autoscaling config to _config.<component>.keda pattern

948ba0b

Matches VPA/PDB convention where each component owns its config subtree. Also moves prometheus_address into backend_worker.keda where it's used.

Copilot AI review requested due to automatic review settings April 20, 2026 14:50

mapno force-pushed the keda-autoscaling branch from dd748c0 to 948ba0b Compare April 20, 2026 14:50

Copilot started reviewing on behalf of mapno April 20, 2026 14:51 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread operations/jsonnet/microservices/autoscaling.libsonnet

Comment thread operations/jsonnet/microservices/configmap.libsonnet

Comment thread operations/jsonnet/microservices/autoscaling.libsonnet

Add render-time asserts for HPA scaling policy fields

98eb2af

javiermolinar approved these changes Apr 22, 2026

View reviewed changes

mapno merged commit 2b291ad into grafana:main Apr 22, 2026
44 of 45 checks passed

mapno deleted the keda-autoscaling branch April 22, 2026 08:55

This was referenced Apr 22, 2026

Docs PR check report: merged PRs for 2026-04-15 to 2026-04-22 knylander-grafana/tempo-doc-work#6

Open

docs: cover KEDA autoscaling and partition ring config removal knylander-grafana/tempo-doc-work#7

Closed

knylander-grafana mentioned this pull request Apr 25, 2026

docs: cover KEDA autoscaling and partition ring config removal #7073

Open

3 tasks

		tempo_distributor_deployment+:
		if $._config.autoscaling.distributor.enabled then $.removeReplicasFromSpec else {},

Conversation

mapno commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

mapno Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

mapno Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

zalegrala Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

zalegrala Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

mapno Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javiermolinar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mapno commented Apr 15, 2026 •

edited

Loading