s3: Read AWS_EC2_METADATA_SERVICE_ENDPOINT for IAM credential resolution

[jaisonerick]

Summary

The S3 backend's IAM credential provider hardcodes TEST_IAM_ENDPOINT as the only way to override minio-go's default IMDS endpoint (169.254.169.254). This makes it impossible to use custom IMDS-compatible credential providers — such as AWS IAM Roles Anywhere credential helper in serve mode — without setting a test-only environment variable.

This adds AWS_EC2_METADATA_SERVICE_ENDPOINT as a standard fallback, matching the env var used by the AWS SDK for Go. Priority:

1. TEST_IAM_ENDPOINT (backward compat, no behavior change for existing users)
2. AWS_EC2_METADATA_SERVICE_ENDPOINT (standard AWS SDK env var)
3. minio-go default (169.254.169.254)

Motivation

When running Tempo outside of EC2/ECS (e.g., on a Raspberry Pi with IAM Roles Anywhere), the credential helper provides an IMDS-compatible endpoint on a custom address (127.0.0.1:9911). The AWS SDK reads AWS_EC2_METADATA_SERVICE_ENDPOINT to find this endpoint, but Tempo's minio-go credential chain ignores it — causing a 90-second timeout against 169.254.169.254 followed by Access Denied on every S3 operation.

Test plan

• Verified on Raspberry Pi 4 with IAM Roles Anywhere credential helper
• TEST_IAM_ENDPOINT behavior is unchanged (checked first)
• When neither env var is set, minio-go default behavior is preserved

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db2faa8344

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Limit endpoint override to IMDS-only credential retrieval

Passing iamEndpoint() into credentials.IAM.Endpoint here affects more than IMDS in minio-go: IAM.RetrieveWithCredContext reuses Endpoint for the AWS_WEB_IDENTITY_TOKEN_FILE branch as the STS endpoint (see vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go, case identityFile != ""). If AWS_EC2_METADATA_SERVICE_ENDPOINT is set in an IRSA/web-identity environment, credential exchange is sent to the IMDS URL instead of STS and auth fails. This regression is introduced by setting the endpoint unconditionally for all IAM provider modes.

Useful? React with 👍 / 👎.

[jaisonerick]

Good catch on the shared Endpoint field — but this is a pre-existing issue with TEST_IAM_ENDPOINT, not something introduced by this PR. The current code:

Endpoint: os.Getenv("TEST_IAM_ENDPOINT"),

has the exact same behavior: if TEST_IAM_ENDPOINT is set and AWS_WEB_IDENTITY_TOKEN_FILE is also set, the IMDS URL gets used as the STS endpoint. This PR just adds a standard env var as a fallback — it does not change how credentials.IAM.Endpoint is consumed.

The root issue is in minio-go: IAM.Endpoint is overloaded for both IMDS and STS paths. Fixing that belongs upstream in minio-go, not here.

In practice, AWS_EC2_METADATA_SERVICE_ENDPOINT is set in environments where IMDS is the credential source (custom IMDS proxies, IAM Roles Anywhere). It would not be set alongside AWS_WEB_IDENTITY_TOKEN_FILE in an IRSA environment.

[javiermolinar]

Looks reasonable to me. @jaisonerick could you sign the license and add a line to the changelog?