Add new histogram to generator - messaging system latency

[adirmatzkin]

What this PR does:
This PR adds a new metric calculated by the metrics-generator service-graph processor.
The metric is a histogram for the latency between 2 services communicating through a messaging system.

Which issue(s) this PR fixes:
Fixes #3232

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[joe-elliott]

Thanks for putting some time into this. It's a fairly small change should be a pretty easy review, but we do need this to be configurable.

We are currently trying to keep metrics generator series down and histograms tend to generate a lot of series.

I will give this more time tomorrow.

[adirmatzkin]

We are currently trying to keep metrics generator series down and histograms tend to generate a lot of series.

Just pushed some changes to allow controlling the metric from config.
Not sure I did it in all the relevant places... saw it working in a specific config change I tested though.

Also, the metric is initialized even if the config is set to false... probably not the best, but it felt simpler due to the current structure of the code... Is that fine? Maybe you guys have a different idea?

[joe-elliott]

a few questions, but looking good!

also, we will need a changelog

[joe-elliott]

instead of casting we could check to see if diff > end for overflow. is that better?

[joe-elliott]

also, we recently made a change to protect span metrics from negative latencies:

https://github.com/grafana/tempo/pull/3412/files

@mdisibio should we make sure the same protections are here?

[adirmatzkin]

instead of casting we could check to see if diff > end for overflow. is that better?

I agree we can make it cleaner,
Really liked your idea, very clever 🙃 My only concern about it is how clear it will be. After making the check if diff > end, if underflow did happen, meaning the subtraction yields a negative number - don't we want to return it?

If not, maybe we could make this function a bit more specific and move the negative values "protection" into it.. and then.. return 0?
It could also be that simple:

func unixNanosDiffSec(unixNanoStart uint64, unixNanoEnd uint64) float64 {
	if unixNanoStart > unixNanoEnd {
		// To prevent underflow, return 0.
		return 0
	}
	// Safe subtraction.
	return float64(unixNanoEnd-unixNanoStart) / 1e9
}

[joe-elliott]

let's follow suit with the linked PR for consistency and return 0.

i think your suggested unixNanosDiffSec in this PR is perfect.

[adirmatzkin]

so this one is resolved? 🤔

[adirmatzkin]

Added to changelog.
Will update docs after making a decision about the "negative diffs"

[adirmatzkin]

instead of casting we could check to see if diff > end for overflow. is that better?

I agree we can make it cleaner,
Really liked your idea, very clever 🙃 My only concern about it is how clear it will be. After making the check if diff > end, if underflow did happen, meaning the subtraction yields a negative number - don't we want to return it?

If not, maybe we could make this function a bit more specific and move the negative values "protection" into it.. and then.. return 0?
It could also be that simple:

func unixNanosDiffSec(unixNanoStart uint64, unixNanoEnd uint64) float64 {
	if unixNanoStart > unixNanoEnd {
		// To prevent underflow, return 0.
		return 0
	}
	// Safe subtraction.
	return float64(unixNanoEnd-unixNanoStart) / 1e9
}

[joe-elliott]

Thanks for the work. This is looking good.

heads up you will need this commit in your branch (merge or rebase) to get CI to pass:

7540d57

[adirmatzkin]

heads up you will need this commit in your branch (merge or rebase) to get CI to pass:

Synced from upstream + added docs.
@joe-elliott anything else needed? 🙃

[joe-elliott]

Looking good, just have a few small thoughts.

Also, what do you think about adding the ConnectionType to the expired edge metric? If someone is wanting to do service graph metrics through queues I'm concerned it will create a lot of expired edges. It might be nice for an operator to know if their expired edges are largely queues or synchronous calls.

[adirmatzkin]

Ready to merge this one? @joe-elliott
Just rebased 🙃

[knylander-grafana]

Thank you for updating the docs

[adirmatzkin]

@joe-elliott I think we're ready 🥳

[joe-elliott]

running CI!

[joe-elliott]

Thanks, Adir!