NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[Valve]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v1.25.13

Version of Go (go version)?

1.13.1

What issue did you see?

I'm using ElasticBeanstalk and some requests fail with:

NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Steps to reproduce

This issue is hard to reproduce, happens infrequently, for a small fraction of requests.
There can be days of it not happening, but if it starts occurring, there is a spike of these errors.

I also noticed that when the load on an instance increases, the likelihood of this error also increases.

I'm using the worker type of EBS.

Please let me know what additional information is required from me to fix this.

UPDATE:
I enabled Verbose logging for credentials and here is the full error message:

NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused

This is related to: #1861

[diehlaws]

Hi @Valve, thanks for reaching out to us. The error provided suggests that the instance metadata service is unreachable at the time the SDK is trying to retrieve credentials. In order to better troubleshoot this I'll need some additional information.

• What instance type(s) are you using in Elastic Beanstalk that exhibit this behavior?
• Are you relying on instance profiles attached to your instance to provide credentials to the SDK, or are you providing credentials in a different manner?
• Do you notice correlation between these errors and heavy load on any particular resource type (CPU/storage/network) on your instances?
• Does this occur when interacting with a specific service(s), or with any service?
• Can you provide a code sample showing how you're constructing your session, service client, and request(s) to the service(s) with which you see this behavior?

This does look to be somewhat related to #1861, however that issue deals more with the SDK attempting to reach the instance metadata service over https when the service is listening for http requests. In this case it looks like the SDK is attempting to reach the instance metadata service over http so I think the cause here is different.

That being said, does implementing a timeout within the role credential provider's HTTP client as suggested in this comment help reduce/eliminate the occurrence of these errors? If not, can you implement a check in your code to attempt to reach the instance metadata service without the SDK when you get this error to see if it's just the SDK that cannot reach the service or if it is unavailable to the instance altogether? Something along the lines of the following should do.

if err == credentials.ErrNoValidProvidersFoundInChain {
  resp, reqErr := http.Get("http://169.254.169.254/")
  if reqErr != nil {
    fmt.Println("Unable to reach instance metadata service:", reqErr)
  }
  fmt.Println(resp)
}

[Valve]

Thanks @diehlaws for all the questions and your response.

First of all: after I submitted this issue, I implemented a custom http client with a 10s timeout and haven't seen this issue occur since then.
It's been running for about a week now.
It's too early to tell if this completely fixed it, I will let you know and close the issue if I don't see new occurrences for 1 month.

Now answers to your questions:

What instance type(s) are you using in Elastic Beanstalk that exhibit this behavior?

t3.micro in us-east-1 and eu-central-1

Are you relying on instance profiles attached to your instance to provide credentials to the SDK, or are you providing credentials in a different manner?

Yes, I'm using a built-in IAM instance profile aws-elasticbeanstalk-ec2-role in virtual machine permissions settings.

Do you notice correlation between these errors and heavy load on any particular resource type (CPU/storage/network) on your instances?

Weak correlation between these errors and CPU load. Sometimes they happen during the night, which doesn't make sense, but more often than not they happen under a larger-than-normal CPU load.

Does this occur when interacting with a specific service(s), or with any service?

I'm only using DynamoDB service on those machines, so it's only DynamoDB.

Can you provide a code sample showing how you're constructing your session, service client, and request(s) to the service(s) with which you see this behavior?

func MustCreateSession() *session.Session {
	// configuring too much retries causes overall slowdown,
	// it's better to fail earlier
	return session.Must(session.NewSession(&aws.Config{
		MaxRetries:                    aws.Int(1),
		CredentialsChainVerboseErrors: aws.Bool(true),
		// HTTP client is required to fetch EC2 metadata values
		// having zero timeout on the default HTTP client sometimes makes
		// it fail with Credential error
		// https://github.com/aws/aws-sdk-go/issues/2914
		HTTPClient: &http.Client{Timeout: 10 * time.Second},
	}))
}

// then later in code


func PutItem(tableName TableName, item interface{}) (*dynamodb.PutItemOutput, error) {
	sess := MustCreateSession()
	ddb := dynamodb.New(sess)
	// TODO: handle marshalling error better
	attrValues, _ := dynamodbattribute.MarshalMap(item)
	putItemInput := &dynamodb.PutItemInput{
		TableName: aws.String(tableName),
		Item:      attrValues,
	}
	return ddb.PutItem(putItemInput)
}

If this issue re-occurs, I will implement the HTTP metadata check as you suggested.

Thanks!

[Valve]

@diehlaws now the error is slightly different:

awserr.baseError:
NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused

My HTTP timeout tweak didn't help

[Valve]

@diehlaws some updates here..
The original error is not occurring any more, but I think it was just renamed to a new error, what I'm getting now is:

ServiceUnavailableException: Call to Authorization engine failed status code: 500,
request id: 6E87BP6R9EL3JAHBKTCVD0JAUVVV4KQNSO5AEMVJF66Q9ASUAAJG

This error happens ~10x more frequently, about 1 time per 5K requests, and only happens in EUC1 region.

When using DynamoDB, please let me know if you need me to do anything to be able to diagnose further.

[dmigo]

I have the same error while accessing DynamoDB.
Go version v1.12, aws-sdk-go version v1.25.42

It doesn't seem to be load dependent, since sometimes it happens on 600+ RPS.

Region is EUC1 as well.
We run our code in Kubernetes. We use m5.lagre as our Kubernetes nodes.

[dmigo]

I was able to resolve my problem by reusing the session.Session and dynamodb.DynamoDB client.
I'll try to investigate deeper.

[Valve]

@dmigo I'm also reusing it, but still seeing these issues...

[Valve]

@dmigo please let me know if you're still having this issue

[dmigo]

@Valve I did three things:

1. Upgraded the version of go to v1.13
2. Upgraded aws-sdk-go to v1.28.5
3. Rewritten my code so that the session and the client are always reused

I'm not sure which part cured the problem, but I haven't seen the error ever since.

[brainstorm]

I'm seeing this error consistently in lucagrulla/cw#119 when using AWS SSO v2 CLI-generated credentials.

Related issues:

#3186
hashicorp/terraform-provider-aws#10851

Current workarounds:

https://github.com/benkehoe/aws-sso-credential-process

[ykcab]

I am also getting this error after compiling the program to a binary
go version go1.14.2 linux/amd64

[brainstorm]

We are using the following tool for the time being until this is fixed:

https://github.com/victorskl/yawsso