Add ability to drop instance label from span metrics#5706
Add ability to drop instance label from span metrics#5706ie-pham merged 9 commits intografana:mainfrom
Conversation
ie-pham
requested review from
carles-grafana,
electron0zero,
javiermolinar,
joe-elliott,
mapno,
mattdurham,
mdisibio,
ruslan-mikhailov,
stoewer,
yvrhdn and
zalegrala
as code owners
| - `status_message` (optionally enabled) - The message that details the reason for the `status_code` label | ||
| - `job` - The name of the job, a combination of namespace and service; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` | ||
| - `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` | ||
| - `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` and can be dropped by setting `metrics_generator.processor.span_metrics.drop_instance_label: true |
There was a problem hiding this comment.
@rlankfo isn't instance semi required for target_info to work correctly?
There was a problem hiding this comment.
IE when we go to set the value on the Gauge your likely to override/not override appropriately in SetForTargetInfo
There was a problem hiding this comment.
can you explain why it would not work appropriately?
There was a problem hiding this comment.
the original requirement also states that if instance is empty to skip it entirely, which sounds to me like it's not a hard requirement.
There was a problem hiding this comment.
If there is not a job or instance then the metric will never get recorded, which feels like different behavior than reducing cardinality. The change in behavior would be if there is no job label and drop instance is true then the metric will not be recorded, whereas previously it would be. Though granted that should be pretty rare that a service.name (job) is not declared.
| @@ -206,7 +206,7 @@ func (p *Processor) aggregateMetricsForSpan(svcName string, jobName string, inst | |||
| labelValues = append(labelValues, jobName) | |||
| } | |||
| // add instance label only if job is not blank | |||
There was a problem hiding this comment.
Not related to your code but this comment looks like its wrong.
There was a problem hiding this comment.
Yeah this is a chance to the original requirement. I will change the comments.
| } | ||
| // add instance label to target info only if job is not blank | ||
| if instanceID != "" { | ||
| if instanceID != "" && !p.Cfg.DropInstanceLabel { |
There was a problem hiding this comment.
same comment issue here.
| - `status_message` (optionally enabled) - The message that details the reason for the `status_code` label | ||
| - `job` - The name of the job, a combination of namespace and service; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` | ||
| - `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` | ||
| - `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` and can be dropped by setting `metrics_generator.processor.span_metrics.enable_instance_label: false |
There was a problem hiding this comment.
Thoughts on changing this to say
The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true`
and `metrics_generator.processor.span_metrics.enable_instance_label: true`
2 participants
What this PR does: The
instancelabel in spanmetric series tends to be very high in cardinality. A new config calledenable_instance_labelis added. It is set to true by default and can be set to false to skip the instance label when generating spanmetrics series.Which issue(s) this PR fixes:
Fixes #4873
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]