Skip to content

Add ephemeral storage to generator for wal storage#1556

Merged
zalegrala merged 8 commits intografana:mainfrom
zalegrala:generatorWalPvc
Jul 14, 2022
Merged

Add ephemeral storage to generator for wal storage#1556
zalegrala merged 8 commits intografana:mainfrom
zalegrala:generatorWalPvc

Conversation

@zalegrala
Copy link
Copy Markdown
Contributor

@zalegrala zalegrala commented Jul 7, 2022

What this PR does:

Without this change, the generator uses node storage and is not constrianed, and could lead to resource contention on the node.

Here we include a request/limit resource on the generator deployment and an emptydir to constrian the use of local storage.. A side effect of this change is that metrics generator instances will be asked to terminate if the limit is exceeeded, and a new instance will be created. Storage values should be sufficiently high in order to prevent unnecessary churn on the deployment and memberlist. If the requested size is not available, the pod will fail to schedule and could lead to data loss.

Which issue(s) this PR fixes:
Fixes grafana/tempo-squad#91

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Comment thread operations/jsonnet/microservices/generator.libsonnet Outdated
@zalegrala zalegrala force-pushed the generatorWalPvc branch 2 times, most recently from 8fa6110 to 973b065 Compare July 8, 2022 17:32
Comment thread .github/workflows/backport.yml
@zalegrala zalegrala changed the title Generator wal pvc Add ephemeral storage to generator for wal storage Jul 12, 2022
@zalegrala
Copy link
Copy Markdown
Contributor Author

I've made an update here to include an emptyDir and request limits. Let me know your thoughts.

@zalegrala zalegrala marked this pull request as ready for review July 12, 2022 21:32
Comment thread operations/jsonnet/microservices/config.libsonnet Outdated
Comment thread operations/jsonnet/microservices/config.libsonnet Outdated
Copy link
Copy Markdown
Contributor

@annanay25 annanay25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one! LGTM.

@annanay25 annanay25 self-requested a review July 14, 2022 07:09
@zalegrala zalegrala merged commit 2a9efc3 into grafana:main Jul 14, 2022
@zalegrala zalegrala deleted the generatorWalPvc branch July 14, 2022 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants