[BUG]: Pagination issue in github_repository_file data block

[cavila-evoliq]

Expected Behavior

When the provider fails to find the file being looked for in the data block, if it does not exist in the first page, it should proceed to the second page.

Actual Behavior

Provider reported the file as not found.

Terraform Version

Terraform v1.6.0
on linux_amd64
integrations/github v5.41.0

Affected Resource(s)

• data.github_repository_file

Terraform Configuration Files

data "github_repository_file" "lookup_files" {
  repository = "<some repo>"
  branch     = "<some branch>"
  file       = "<some file>"
}

provider "github" {
  token = "<some token>"
  owner = "<some owner>"
}

Steps to Reproduce

1. Create a commit with 300+ changes.
2. use data.github_repository_file to lookup a file that shows up on the 2nd page when you query the api.

Debug Output

Obviously wont post the full unredacted log but here's some snippets.

======Request for the commit======
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: -----------------------------------------------------
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: 2023/11/06 14:02:08 [DEBUG] GitHub API Request Details:
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: ---[ REQUEST ]---------------------------------------
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: GET /repos/redacted/redacted/commits/redacted HTTP/1.1
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Host: api.github.com
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: User-Agent: go-github/v55.0.0
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Accept: application/vnd.github.v3+json,application/vnd.github.stone-crop-preview+json
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: X-Github-Api-Version: 2022-11-28
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Accept-Encoding: gzip

=======Failed to find SHA======
2023-11-06T14:02:08.603-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: 2023/11/06 14:02:08 [DEBUG] Found file: <redacted>/<redacted>/path/to/<redacted>, in commit SHA:  <blank>

Panic Output

No panic, just returns with error.

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

👋 Hi! Thank you for this contribution! Just to let you know, our GitHub SDK team does a round of issue and PR reviews twice a week, every Monday and Friday! We have a process in place for prioritizing and responding to your input. Because you are a part of this community please feel free to comment, add to, or pick up any issues/PRs that are labled with Status: Up for grabs. You & others like you are the reason all of this works! So thank you & happy coding! 🚀

[rumenvasilev]

This is the error you get: https://github.com/integrations/terraform-provider-github/blob/main/github/data_source_github_repository_file.go#L141

It is coming from here - https://github.com/integrations/terraform-provider-github/blob/main/github/data_source_github_repository_file.go#L140

As you can tell, there's pagination and it works correctly (just verified) - https://github.com/integrations/terraform-provider-github/blob/main/github/repository_utils.go#L35-L48
It is possible that the paginated request fails but that would be due to other issues - network, rate-limit, etc.

I suspect your issue lies somewhere in this section though: https://github.com/integrations/terraform-provider-github/blob/main/github/repository_utils.go#L53-L67, meaning the following are the possibilities:

• Commit message contains "Merge branch"
• GetCommit() API call returns an error (could be a network issue or perhaps you've exhausted the rate-limit)
• Filename contains some weird characters that aren't processed correctly

Do you really not see any other error message in the debug output, perhaps this: https://github.com/integrations/terraform-provider-github/blob/main/github/repository_utils.go#L70

[cavila-evoliq]

The cannot find file %s in repo %s/%s line is returned when terraform ultimately exits, yes.

You identified where I think the problem is here but the problem I believe is that the GetCommit() also needs to be paginated. This specific commit had 400 changed files and we were able to successfully reproduce the error by trying to lookup any file that was returned from the api on the 2nd page of the list of changed files. Any file looked up that was returned on the first page by the api did not return an error, only things we were able to verify being returned on the 2nd page by the api did.

This is the request from the debug log:

2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: -----------------------------------------------------
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: 2023/11/06 14:02:08 [DEBUG] GitHub API Request Details:
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: ---[ REQUEST ]---------------------------------------
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: GET /repos/<org>/<repo>/commits/<sha> HTTP/1.1
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Host: api.github.com
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: User-Agent: go-github/v55.0.0
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Accept: application/vnd.github.v3+json,application/vnd.github.stone-crop-preview+json
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: X-Github-Api-Version: 2022-11-28
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Accept-Encoding: gzip
2023-11-06T14:02:08.246-0600 [DEBUG] provider.terraform-provider-github_v5.41.0

And we get a successful response, but there is a link being returned indicating that this specific commit has a page 2.

2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: ---[ RESPONSE ]--------------------------------------
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: HTTP/2.0 200 OK
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Access-Control-Allow-Origin: *
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Cache-Control: private, max-age=60, s-maxage=60
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Content-Security-Policy: default-src 'none'
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Content-Type: application/json; charset=utf-8
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Date: Mon, 06 Nov 2023 20:02:08 GMT
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Etag: W/"b0bfbccf4d73bf3dc7cd6d2cbbb77d9ff4e9df7d6c3a19725313216f7e5c4d1e"
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Last-Modified: Thu, 02 Nov 2023 00:13:22 GMT
2023-11-06T14:02:08.594-0600 [DEBUG] provider.terraform-provider-github_v5.41.0: Link: <https://api.github.com/repositories/<repo>/commits/<commit>?page=2>; rel="next", <https://api.github.com/repositories/<repo>/commits/<commit>?page=2>; rel="last"

The debug output successfully lists all files on the first page, but does not list anything on page 2. As I had mentioned before, my guess is this is returning only the first page of files modified by a specific (single) commit and as such rc.Files contains only the first page of changed files.

I should add, after identifying this behavior, we successfully circumvented the problem by adding a white space to all the affected files to generate a new commit with all of the files on the first page of what's returned in the api call. No changes to any file names and both the prior and new commits did not have "Merge branch" in the commit message. For reference, the old/problem commit message was chore: moved apps values to GCP-US folder [no ticket]

[rumenvasilev]

You're right. It's outlined in the REST API documentation - https://docs.github.com/en/rest/commits/commits?apiVersion=2022-11-28#get-a-commit:

Note: If there are more than 300 files in the commit diff, the response will include pagination link headers for the remaining files, up to a limit of 3000 files. Each page contains the static commit information, and the only changes are to the file listing.

[rumenvasilev]

Can you try building the provider yourself from the PR I linked above?
The steps would be: go build . (that would produce the binary in the current directory) and then you can use the instructions from https://developer.hashicorp.com/terraform/cli/config/config-file#development-overrides-for-provider-developers to configure terraform to use that instead of the official version from the registry.

[cavila-evoliq]

Just tested it out and this does resolve the issue.

Before:

{7:43}~/github/github-data-error:main ✗ ➭ terraform apply
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - hashicorp.com/edu/hashicups-pf in /Users/christopheravila/go/bin
│  - integrations/github in /Users/christopheravila/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state
│ to become incompatible with published releases.
╵
data.github_repository_file.lookup_files: Reading...
╷
│ Error: cannot find file junk/file9.txt in repo cavila-evoliq/github-data-error
│ 
│   with data.github_repository_file.lookup_files,
│   on main.tf line 2, in data "github_repository_file" "lookup_files":
│    2: data "github_repository_file" "lookup_files" {
╵

After

╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - hashicorp.com/edu/hashicups-pf in /Users/christopheravila/go/bin
│  - integrations/github in /Users/christopheravila/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state
│ to become incompatible with published releases.
╵
data.github_repository_file.lookup_files: Reading...
data.github_repository_file.lookup_files: Read complete after 1s [id=github-data-error/junk/file9.txt]

Ty!