Skip to content

Lack of server label - Multi-Target Support #888

Closed
@rafal-skurka

Description

@rafal-skurka

Some of the metrics don't have a "server" label it's quite problematic in Multi-Target Support because we can't distinguish database servers and some useful graphs in the Grafana dashboard view incorrect data. In my opinion, that label should be added to every single metric to avoid mixing up data between different database servers.

Number of metrics with and without the "server" label

# with
➜  x git:(x) ✗ curl -s localhost:9187/metrics | grep -ci server 
    1465
#without 
➜  x git:(x) ✗ curl -s localhost:9187/metrics | grep -civ server 
    1338

Example metric without server label:

pg_database_size_bytes{container="prometheus-postgres-exporter", datname="green", endpoint="http", instance="192.168.226.57:9187", job="postgres-exporter-prometheus-postgres-exporter", namespace="monitoring", pod="postgres-exporter-prometheus-postgres-exporter-84675c5cfc-hbt77", service="postgres-exporter-prometheus-postgres-exporter"}
pg_database_size_bytes{container="prometheus-postgres-exporter", datname="blue", endpoint="http", instance="192.168.226.57:9187", job="postgres-exporter-prometheus-postgres-exporter", namespace="monitoring", pod="postgres-exporter-prometheus-postgres-exporter-84675c5cfc-hbt77", service="postgres-exporter-prometheus-postgres-exporter"}

Expected output from this metric:

pg_database_size_bytes{container="prometheus-postgres-exporter", datname="green",<trimmed>,server="green-server:5432"}
pg_database_size_bytes{container="prometheus-postgres-exporter", datname="blue",<trimmed>,server="blue-server:5432"}

Activity

SuperQ

SuperQ commented on Aug 14, 2023

@SuperQ
Contributor

Sounds like your relabel config isn't correct. The server label should come from the relabeling.

added a commit that references this issue on Aug 14, 2023
added a commit that references this issue on Aug 15, 2023
added a commit that references this issue on Oct 26, 2023

PMM-12154 pull upstream changes (#153)

82694ff
added a commit that references this issue on Feb 22, 2024
portnov

portnov commented on Apr 22, 2024

@portnov

Hello
I have a similar issue.
Even without Prometheus by itself, I just run prometheus-exporter and go to localhost:9187/metrics. And I see that some metrics have server= label, and some do not. Moreover, some metrics appear twice, once with this label and once without it:

pg_database_size_bytes{datname="pgbench"} 5.1106607e+07
pg_database_size_bytes{datname="pgbench",server="host.docker.internal:5432"} 5.1106607e+07

(the value seem to be always equal for both records).
Going to /probe?target=... does not help, I see the same picture.
Since I see this in exporter output, I do not think it has something to do with prometheus config...

KulykDmytro

KulykDmytro commented on May 8, 2024

@KulykDmytro

Hi
at 0.15 running for of RDS instance (in kubernetes) we observing following behaviour. on default settings + queries.yaml
These collectors provides server label, other not:

  • all metrics passed over queries.yaml
  • pg_settings
  • pg_stat_activity
  • pg_stat_archiver
  • pg_stat_database (only for conflicts_confl section)
  • pg_static
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @portnov@SuperQ@KulykDmytro@rafal-skurka

      Issue actions

        Lack of server label - Multi-Target Support · Issue #888 · prometheus-community/postgres_exporter