Datadog: datadog.hostname setting does not appear to affect host inventory display name

[ringerc]

Component(s)

exporter/datadog

What happened?

Description

The datadog exporter configuration option hostname does not appear to affect the hostname displayed as the preferred hostname in the datadog UI infrastructure view host map or host inventory.

It continues to use the internal cloud provider id for the host; the configured hostname is assigned as a host alias instead. This is not great, as AWS host IDs are i-xxxxxxxxxxx and GCP host-ids are UUIDs like 0b797c2d-36cc-4bd4-bdbb-f33d7a0fcc2b. Many datadog dashboards don't support filtering by host aliases, only by the "main" hostname, so this impacts dashboard usability too.

This is the case whether host_metadata.hostname_source is set to first_resource or config_or_system.

Steps to Reproduce

Configure an OpenTelemetry based datadog agent using the default otel node image otel/opentelemetry-collector-contrib:0.90.1.

Use the recommended datadog configuration with the k8sattributes processor, resourcedetection processor configs for your cloud environment etc.

Add the following to your DaemonSet's env stanza:

          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

Configure the datadog exporter with:

exporters:
  datadog:
    hostname: ${env:K8S_NODE_NAME}
    host_metadata:
      enabled: true
      hostname_source: config_or_system

and add your API key config etc.

Visit the linked DD account. Note that the new node shows up in the "Infrastructure -> Host Map" view under its internal cloud provider id (the host.id detected by the processors) not the configured hostname.

Try changing the hostname_source to first_resource. Repeat. You will still see the internal host.id as the hostname.

Expected Result

I expect to see the value of the host.name or k8s.node.name provided to the datadog exporter, not the internal host.id. This is the behaviour seen with the DD agent.

If the reported preferred hostname changes after initial node registration, the DD UI should reflect the preferred hostname being sent.

Actual Result

No matter what I do, my nodes show up with the host.id for their primary display hostname.

The real hostname (k8s node name) is sometimes shown in the "aliases", sometimes not. I've yet to determine why.

The real hostname is not shown in the overviews:

or usable for selecting hosts in dashboards:

I have verified via kubectl debug based inspection of the otel collector process's /proc/$pid/env that the K8S_NODE_NAME env-var is present and set to the kube node name.

Collector version

0.90.1

Environment information

Environment

k8s on Azure AKS, Google Cloud GKE and AWS EKS.

OpenTelemetry Collector configuration

# My real config is long.
# Use the example from https://docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter/#2-configure-the-datadog-exporter
# and add the config

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}
    hostname: ${env:K8S_NODE_NAME}
    host_metadata:
      enabled: true
      hostname_source: config_or_system

Log output

N/A, nothing relevant is logged.

Additional context

Related issue requesting auto-discovery of host metadata tags: #29700

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ringerc]

I did more digging into this.

The datadog.hostname setting does affect the reported hostname... somewhat. It doesn't seem to override the preferred hostname, but it seems to add an "alias" to the displayed hostname in the infrastructure list if configured.

Without explicit datadog.hostname, hostname_source=config_or_system

Log event will be emitted

{"level":"debug","ts":1702596764.453713,"caller":"provider/provider.go:55","msg":"Unavailable source provider","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"config","error":"empty configuration hostname"}

then cloud provider specific ones from the cloud provider IMDS based discovery are emitted like:

{"level":"info","ts":1702596134.5370073,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"azure","source":{"Kind":"host","Identifier":"c2c7b383-3aa8-4f3b-bb33-d332908e36af"}}

{"level":"info","ts":1702595688.6152918,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"ec2","source":{"Kind":"host","Identifier":"i-0ad96d927c1802a8e"}}

{"level":"info","ts":1702596764.5491843,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"gcp","source":{"Kind":"host","Identifier":"gke-dp-vcpete01clh2q-e2-standard-4-a0-cb95a6fc-vbvs.development-data-381310"}}

these point to the internal cloud provider host names, which are picked as the display names.

Azure IMDS for the node I'm using reports compute.osProfile.computerName = aks-f4sv2a0-24785920-vmss000000, but the discovery plugin seems to be picking the value from compute.vmId = c2c7b383-3aa8-4f3b-bb33-d332908e36af instead.

AWS IMDS query on the node I'm using to http://169.254.169.254/latest/meta-data/hostname returns ip-10-0-11-132.eu-west-1.compute.internal. I have no idea how DD is picking i-0ad96d927c1802a8e instead. Presumably it picks some other metadata source.

For GCP it seems to pick the actual IMDS-reported hostname in discovery.

For Azure and AWS it's IMO not picking the best hostname source. The internal IDs should be aliases, and it should prefer the Azure compute.osProfile.computerName or AWS /latest/meta-data/hostname.

With explicit hostname, hostname_source=first_resource

If you set datadog.hostname to (say) the kube downward-api for the node name and use hostname_source=first_resource, it gets more confusing.

Some reading of the source code suggests that the exporter may log the "Resolved source" message for the exporter's internal discovery even if it actually picked the host name from resource attribute based discovery. It kicks off the config and cloud provider probe based discovery asynchronously, but will then ignore its results if the attribute based discovery matched something it considered satisfactory.

AFAICS there is no log message to indicate whether or not resource attribute based discovery picked the hostname.

So while it will log messages with the k8s node names like

{"level":"info","ts":1702597850.33392,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"config","source":{"Kind":"host","Identifier":"ip-10-0-11-132.eu-west-1.compute.internal"}}

{"level":"info","ts":1702597755.9463413,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"config","source":{"Kind":"host","Identifier":"gke-dp-vcpete01clh2q-e2-standard-4-a0-cb95a6fc-vbvs"}}

{"level":"info","ts":1702598329.3088248,"caller":"provider/provider.go:59","msg":"Resolved source","kind":"exporter","data_type":"metrics","name":"datadog/datadog","provider":"config","source":{"Kind":"host","Identifier":"aks-d4sv4c0-42735447-vmss000000"}}

... it seems to actually ignore those and pick the values it picked from resource attributes instead.

And its attribute based discovery in https://github.com/DataDog/opentelemetry-mapping-go/ is very inconsistent from provider to provider. It seems to prefer to use the node.id instead of node.name for Azure and AWS, but prefers node.name for GCP. It also seems to ignore most of the other available, useful metadata tags from the cloud provider semantic conventions etc. Then there's another layer tacked on top in the hostmetrics exporter itself for some AWS-specific attribute based discovery.

The logic bounces back and forth between opentelemetry-mapping-go and opentelemetry-collector-contrib exporter/datadogexporter/internal/hostmetadata and is tricky to clearly unpick. It really needs a proper review to ensure that it does something sensible and consistent for both attribute based discovery and IMDS based discovery.

[mx-psi]

Hey, thanks for filing this issue. There's a lot to unpack here, so let me reply below to individual points so that I can hopefully clear up some confusion and get your setup to a satisfying state.

Questions about the hostname in the infrastructure list

The datadog exporter configuration option hostname does not appear to affect the hostname displayed as the preferred hostname in the datadog UI infrastructure view host map or host inventory.

The hostname option is the fallback hostname for any telemetry that does not have a hostname attribute already. If your telemetry already has a hostname (e.g. through an OpenTelemetry semantic convention such as host.name, host.id or a similar one that in your case are set by the processors you have set up), then that one will be honored.

If you are familar with the Datadog Agent, the situation there is similar: if you tag something with a host tag, the Agent will use that one, if not, it will put one in for you. The situation is a bit more complex here in that there are multiple host tags (host.name, host.id, k8s.node.name....) but the resulting behavior is the same.

This is the case whether host_metadata.hostname_source is set to first_resource or config_or_system. [...]
Try changing the hostname_source to first_resource. Repeat. You will still see the internal host.id as the hostname.

host_metadata::hostname_source controls what hostname is attached to the host metadata (e.g. things like CPU cores and other system information). It will not change the hostname on your telemetry at all, as you have discovered. If you want to change the hostname on your telemetry you need to use a processor for that.

I expect to see the value of the host.name or k8s.node.name provided to the datadog exporter, not the internal host.id.
This is the behaviour seen with the DD agent.

Can you elaborate on this? The Datadog Agent does not have any host_metadata configuration to my knowledge, nor does it have the resource detection processor or k8sattributes processor, so I am not sure what the equivalence is here.

Some reading of the source code suggests that the exporter may log the "Resolved source" message for the exporter's internal discovery even if it actually picked the host name from resource attribute based discovery.
AFAICS there is no log message to indicate whether or not resource attribute based discovery picked the hostname.

That is true, and I agree it is confusing. The first part is not very easy to change today (the second part is easy), but generally making this simpler is in our roadmap. In general using host_metadata::hostname_source: first_resource is only useful in a handful of exotic infrastructures and edge cases, and may break your setup in unexpected ways in other setups.

Comments about the hostname picked up by default in different cloud providers

For Azure and AWS it's IMO not picking the best hostname source. The internal IDs should be aliases, and it should prefer the Azure compute.osProfile.computerName or AWS /latest/meta-data/hostname.

The current implementation focuses on providing compatibility with the Datadog Agent and other Datadog products hostnames by default. OpenTelemetry provides a lot of flexibility to customize your hostname as you see fit, but the default will remain this because otherwise it wouldn't match that chosen by other Datadog products.

And its attribute based discovery in https://github.com/DataDog/opentelemetry-mapping-go/ is very inconsistent from provider to provider.

There is one exception in AWS, but otherwise it is the same. Here is a table with all the code linked and a brief summary of wht it does

	Provider	Attribute based discovery	Resource detection processor
AWS EC2	instance id if OS hostname is default	host.id	Set host.id to AWS EC2 instance id
Azure VM	VM id	host.id	Set host.id to Azure VM id
GCE	Part of hostname + project ID	part of host.name + cloud.account.id	Set host.name to the hostname, cloud.account.id to the project ID

It seems to prefer to use the node.id instead of node.name for Azure and AWS, but prefers node.name for GCP.

I don't think this is the case. On both the attribute based discovery as well as the builtin provider case all cloud providers are checked before the Kubernetes node name is checked.

It also seems to ignore most of the other available, useful metadata tags from the cloud provider semantic conventions etc.

Not sure what you mean by this, could you elaborate?

Then there's another layer tacked on top in the hostmetrics exporter itself for some AWS-specific attribute based discovery.

Yup, this is there for legacy reasons indeed. Other Datadog products expect this so we fetch it.

The logic bounces back and forth between opentelemetry-mapping-go and opentelemetry-collector-contrib exporter/datadogexporter/internal/hostmetadata and is tricky to clearly unpick. It really needs a proper review to ensure that it does something sensible and consistent for both attribute based discovery and IMDS based discovery.

If you have concrete suggestions about this I am happy to tackle this. The logic itself is pretty unwieldy to be able to support a wide range of cases, so it is fundamentally complex, but I am happy to take feedback as to what to simplify while preserving behavior.

[ringerc]

Thanks very much @mx-psi for your comments.

I definitely do not expect datadog.hostname to change attributes on telemetry (metrics, logs). I am aware I would need processors for that. The k8sattributes processor just does the right thing by default anyway.

I did expect it to change the hostname shown in the DD infrastructure view UI. It does not appear to do so.

The use of the cloud provider's low level node id (usually exposed in OTLP as node.id) instead of the node name (OTLP node.name) when using either attribute discovery or IMDS discovery is surprising and confusing. I expected that when I provide a host name, DD would use that as the host name. It's useful and important to send the host.id, but surely the user-facing display name should be the host.name?

It's particularly painful because many DD dashboards only support filtering by host using this low-level host-id, not by host name aliases or tags. "OpenTelemetry Host Metrics Dashboard" is one such example.

it's ... not great UX.

Putting the table you prepared above into the DD exporter docs, along with the configuration guidance you gave, would be an immense improvement since at least it would be clearly documented that DD expects and requires the low level cloud provider node names, and you cannot get it to use a user-friendly node name instead.

Clarifying the behaviour and recommended use of hostname_source in

opentelemetry-collector-contrib/exporter/datadogexporter/examples/collector.yaml

Line 382 in 0f92b21

## - 'first_resource' picks the host metadata hostname from the resource attributes on the first OTLP payload that gets to the exporter.

per your notes would help too.

In the end though, I don't think it's unreasonable to want to display nodes with the names the user knows those nodes by, not:

Individual comments:

I expect to see the value of the host.name or k8s.node.name provided to the datadog exporter, not the internal host.id.
This is the behaviour seen with the DD agent.

Can you elaborate on this? The Datadog Agent does not have any host_metadata configuration to my knowledge, nor does it have the resource detection processor or k8sattributes processor, so I am not sure what the equivalence is here.

I was unclear. I meant that when hostname_source is set to first_resource, I would expect the provided host name like host.name to be preferred over the low level host identifier like host.id.

The hostname option is the fallback hostname for any telemetry that does not have a hostname attribute

So it's not also supposed to set the host name used in the DD infrastructure node list and infrastructure map views? The docs say "Source for the hostname of host metadata", so I would have expected it to use the configured host name.

See e.g.

The k8s node name injected via downward api and set as hostname: ${env:K8S_NODE_NAME} is completely ignored, and not even shown as an alias

possibly because I currently have hostname_source: first_resource set - but even when config_or_system is used, DD doesn't respect the configured host name and still seems to prefer the node id discovered via IMDS.

It also seems to ignore most of the other available, useful metadata tags from the cloud provider semantic conventions etc.

Not sure what you mean by this, could you elaborate?

Nodes in the infrastructure view have "tags" sections for "datadog", "user", and if explicit tags are configured in the otel collector, "otel". The DD exporter for the otel collector does not set basic tags that the DD UI appears to expect to be defined by default like availability-zone, even for AWS nodes, in its host metadata requests.

so the default view groups everything under "no availability-zone" even AWS nodes that have an AZ

(Ignore the missing cpu data, that's a known issue in my current config, and not relevant to this issue)

Similarly, there are no tags for cloud provider, region etc. Even though suitable tags are provided on metrics by the resourcedetection processor, and are available from the same IMDS endpoints that the DD exporter is using, as seen below:

except for a couple of limited ones for GCP nodes.