proxy.golang.org 403 forbidden much like #48107

[martinrode]

Using Go 1.20 in our Github CI chain, we experience 403 errors from certain IPs of our CSP (dedicated runner at our CSPs) when accessing a specific package.

Using CURL or our laptop computers and most IPs from our ISP work fine.

go install github.com/programmfabrik/apitest@latest
go: downloading github.com/programmfabrik/apitest v1.13.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/programmfabrik/golib v0.0.0-20[23](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:24)0614100546-9870ba66917d
go: downloading github.com/spf13/afero v1.9.5
go: downloading github.com/spf13/cobra v1.7.0
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/spf13/viper v1.16.0
go: downloading github.com/moul/http2curl v1.0.0
go: downloading github.com/tidwall/gjson v1.14.4
go: downloading github.com/Masterminds/sprig/v3 v3.2.3
go: downloading github.com/mattn/go-sqlite3 v1.14.17
go: downloading golang.org/x/mod v0.12.0
go: downloading golang.org/x/oauth2 v0.9.0
go: downloading github.com/PuerkitoBio/goquery v1.8.1
go: downloading github.com/clbanning/mxj v1.8.4
go: downloading github.com/tidwall/jsonc v0.3.2
go: downloading golang.org/x/net v0.11.0
go: downloading github.com/antchfx/xmlquery v1.3.17
go: downloading github.com/gabriel-vasile/mimetype v1.4.2
go: downloading github.com/gofrs/uuid v4.4.0+incompatible
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/logrusorgru/aurora v2.0.3+incompatible
go: downloading github.com/yuin/goldmark v1.4.13
go: downloading golang.org/x/crypto v0.10.0
go: downloading golang.org/x/text v0.10.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/cast v1.5.1
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading golang.org/x/sys v0.9.0
go: downloading github.com/tidwall/match v1.1.1
go: downloading github.com/tidwall/pretty v1.2.0
go: downloading github.com/Masterminds/goutils v1.1.1
go: downloading github.com/Masterminds/semver/v3 v3.2.1
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/huandu/xstrings v1.4.0
go: downloading github.com/imdario/mergo v0.3.16
go: downloading github.com/mitchellh/copystructure v1.2.0
go: downloading github.com/shopspring/decimal v1.3.1
go: downloading github.com/andybalholm/cascadia v1.3.2
go: downloading github.com/antchfx/xpath v1.2.4
go: downloading github.com/golang/groupcache v0.0.0-202103312[24](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:25)755-41bb18bfe9da
go: downloading github.com/subosito/gotenv v1.4.2
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.0.8
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/mitchellh/reflectwalk v1.0.2
Error: /home/github-runner/go/pkg/mod/github.com/programmfabrik/golib@v0.0.0-202[30](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:31)614100546-9870ba66917d/mime_type_reader.go:7:2: github.com/gabriel-vasile/mimetype@v1.4.2: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.2.zip: [40](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:41)3 Forbidden

Our CSP is unable to block assigning that IP (if you need it, I can dig it up) to their VMs.

[suzmue]

Hi @martinrode, would you be able to provide us with that IP or let us know which country that IP is originating from? Thanks!

[martinrode]

Hi, we caught at least one affected IP which was 167.235.252.67. Country of origin is probably Germany (or Finnland). BR Martin

[muhammedsaidkaya]

Is there any update?

[enescakir]

Hi @suzmue, we've encountered a similar issue. The affected IP address is 178.63.229.9, which is located in Germany. We have other servers in the same data center, but they remain unaffected.

I've attached the complete logs from the GODEBUG=http2debug=1 go mod download github.com/aws/aws-sdk-go@v1.51.21 command.

The bottom of the logs shows that. How can we find the exact reason why it's not available?

http2: Transport received DATA stream=1 len=193 data="<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>"
go: github.com/aws/aws-sdk-go@v1.51.21: reading https://proxy.golang.org/github.com/aws/aws-sdk-go/@v/v1.51.21.zip: 403 Forbidden

http2-debug-aws.txt

[dmitshur]

CC @samthanawalla, @findleyr, @hyangah.

[findleyr]

@enescakir a geoip lookup for that IP reports a different, embargoed country.
That may be inaccurate, but likely explains the response.