Unable to get EC2 Role Credentials

[bensont1]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

1.13.20

Version of Go (go version)?

1.9.3

What issue did you see?

I used the following method to simply list the objects in a S3 bucket, however the SDK cannot seem to get the EC2 role which is set to s3:* for * resources. This is the code I'm using to list, as well log debug enabled.

I've verified that the Role is indeed attached to the instance. One thing I did notice, when curl-ing https://169.254.169.254/latest/meta-data/iam/security-credentials from the instance, I get a timeout as well, but when curl is issued for http only, I get a valid response...

        sess := session.New(&aws.Config{
		LogLevel:                      aws.LogLevel(aws.LogDebugWithHTTPBody),
		CredentialsChainVerboseErrors: aws.Bool(true),
		Region: aws.String("us-east-1"),
	})

	val, err := sess.Config.Credentials.Get()
	fmt.Println(val)
	fmt.Println(err)

	svc := s3.New(sess)

	input := &s3.ListObjectsInput{
		Bucket: aws.String("bucket-name"),
	}

	result, err := svc.ListObjects(input)
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case s3.ErrCodeNoSuchBucket:
				fmt.Println(s3.ErrCodeNoSuchBucket, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return
	}

	fmt.Println(result)

Output:

2018/03/24 23:48:26 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.13.20 (go1.9.3; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------

2018/03/24 23:48:56 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.13.20 (go1.9.3; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=***/20180324/us-east-1/es/aws4_request, SignedHeaders=date;host;x-amz-date;x-amz-security-token, Signature=***
Date: 2018-03-24T23:48:26Z
X-Amz-Date: 20180324T234826Z
X-Amz-Security-Token: ***
Accept-Encoding: gzip


-----------------------------------------------------

After a minute or so, I get the following error

{   }
NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
UserHomeNotFound: user home directory not found.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get https://169.254.169.254/latest/meta-data/iam/security-credentials: dial tcp 169.254.169.254:443: i/o timeout
2018/03/24 23:50:26 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.13.20 (go1.9.3; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------

Steps to reproduce

If you have have an runnable example, please include it.

[jasdel]

Thanks for reaching out to us @bensont1, It looks like the host's EC2 Metadata service is timing out. Does your application generally only run into this issue as startup, or are you seeing this issue also periodically during the execution of your application?

If you're running into this issue at startup you could take a look at the connection timeout being used by your application. By default the SDK will use the http.DefaultClient if none is provided when a Session is created. I think the default transport timeout is is 30 seconds. It is possible that your application is starting before the EC2 Metadata service is available. Increasing this timeout or increasing the number of MaxRetires in the Config might be able to resolve the issue.

[bensont1]

@jasdel That would be strange though, the instance itself has been running for a while now, and even then I'm able to curl the metadata from 169.254.169.254 from ssh into the instance. I've tried restarting my application multiple times, and same issue each time, my application (using the code above) has never been able to perform interactions with AWS resources because it never was able to get the proper EC2 iam role.

[bensont1]

@jasdel I did a test with the V2 of the Golang AWS SDK, and using cfg, err := external.LoadDefaultAWSConfig() seems to load EC2 IAM roles perfectly fine. This is with the same EC2 instance that described the previous issue, have not restarted the instance or anything either.

[brianmoran]

I was having the same issue and found that my aws credential var names were lowercase. Coming from the python sdk this is fine. The go sdk on the otherhand is case-sensitive. Once I changed the aws var names to uppercase it worked fine.

[meetsubhojit]

@jasdel both v1 and v2 fails for me. aws cli is able to access the bucket but code is not. below i am pasting both v1 and v2 implementations.

`func main() {

     fmt.Println("v2")
err := func() error {
	cfg, err := externalv2.LoadDefaultAWSConfig(&awsv2.Config{
		LogLevel: awsv2.LogDebugWithHTTPBody,
	})
	if err != nil {
		return err
	}
	cfg.Region = endpointsv2.ApSoutheast1RegionID

	svc := s3v2.New(cfg)
	input := &s3v2.ListObjectsInput{
		Bucket: awsv2.String("some-bucket"),
	}

	result, err := svc.ListObjectsRequest(input).Send()
	if err != nil {
		return err
	}
	fmt.Println(result.Contents)
	return nil
}()
if err != nil {
	fmt.Println("err : ", err.Error())
}
fmt.Println("end v2\n\n")

fmt.Println("v1")
err = func() error {
	sess := session.New(&aws.Config{
		LogLevel:                      aws.LogLevel(aws.LogDebugWithHTTPBody),
		CredentialsChainVerboseErrors: aws.Bool(true),
		Region:                        aws.String("ap-southeast-1"),
	})
	_, err := sess.Config.Credentials.Get()
	if err != nil {
		return err
	}
	svc := s3.New(sess)
	input := &s3.ListObjectsInput{
		Bucket: aws.String("bucket-name"),
	}

	result, err := svc.ListObjects(input)
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case s3.ErrCodeNoSuchBucket:
				fmt.Println(s3.ErrCodeNoSuchBucket, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return err
	}
	fmt.Println(result)
	return nil
}()
if err != nil {
	fmt.Println("err : ", err.Error())
}
fmt.Println("end v1")

}`

OUTPUT -
v1
`
2019/03/11 11:11:26 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.17.14 (go1.10.3 gccgo (Ubuntu 8.2.0-1ubuntu2~18.04) 8.2.0; linux; amd64)
Accept-Encoding: gzip

---

2019/03/11 11:11:26 DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.0 200 OK
Content-Length: 11
Accept-Ranges: bytes
Connection: close
Content-Type: text/plain
Date: Mon, 11 Mar 2019 11:11:26 GMT
Etag: "2818113054"
Last-Modified: Mon, 11 Mar 2019 10:53:28 GMT
Server: EC2ws

---

2019/03/11 11:11:26 social-prod
err : NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /home/ubuntu/.aws/credentials: no such file or directory
EmptyEC2RoleList: empty EC2 Role list
`

v2
v2 err : EmptyEC2RoleList: empty EC2 Role list end v2

[meetsubhojit]

Do we need specific IAM policy attached to get EC2RoleList ?

[meetsubhojit]

fix raised at #2504

[jasdel]

Thanks for creating the PR @TheGUNNER13 I don't think that change fixes the issue as the order of expressions being evaluated doesn't change. Though the PR does improve readability of the method, so we can merge it based on that.

The EC2 Instance Role doc has a good explanation on the steps needed to assign an IAM Role to an EC2 Instance.

With a role assigned correctly the SDK should not need any additional configuration in order to retrieve the credentials for that role from EC2's Instance Metadata Service.

The EmptyEC2RoleList: empty EC2 Role list in the error message means either the EC2 Instance is not configured with an IAM role as expected.

Are you running your application directly within EC2 Instance, or are you using some kind of container such as Docker, or proxy within the EC2 instance?

[meetsubhojit]

I am using an EC2 attached to a IAM role.
Note that all aws cli commands (s3) run on that EC2 works fine, so that tells me the problem is not with EC2. Also with the above code changes, it surprisingly works for me.

If you see the httpDebugLogs, you will see that the role is being fetched from the EC2 metadata, but its not being used.
Will need to debug more, but I dont feel that there is any problem with how the aws roles are setup for the EC2.

[meetsubhojit]

Potentially the a different instance was used for testing?

No, @jasdel, the same machine was being used.

[jasdel]

Thanks for the update @TheGUNNER13 I'm having difficulty reproducing this issue. I'm using the latest version of the SDK (v1.18.5). I see you're using v1.17.14, but that shouldn't cause and issue, but maybe updating to the latest version of the SDK might resolve the issue?

[midN]

Is the timeout for metadata query configurable from outside the SDK?
CodeDeploy uses micro type instances.
There can be slow as f in random req, with low and not configurable timeout for metadata query, that will start failing.

I haven't started digging into code yet