x/sys: does not build on linux/sparc64, stuck on porting attempt

[rrrogalski]

What version of Go are you using (go version)?

$ go version
go version unknown linux/sparc64
$ gcc --version
gcc 12.2.0

Does this issue reproduce with the latest release?

Yes on git master branch

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="sparc64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT="fieldtrack"
GOFLAGS=""
GOHOSTARCH="sparc64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/libexec/gcc/sparc64-unknown-linux-gnu/12.2.0"
GOVCS=""
GOVERSION="unknown"                                                                                                                                    
GCCGO="/usr/sparc64-unknown-linux-gnu/gcc-bin/12.2.0/gccgo"                                                                                            
AR="ar"                                                                                                                                                
CC="gcc"                                                                                                                                               
CXX="g++"                                                                                                                                              
CGO_ENABLED="1"                                                                                                                                        
GOMOD="/var/tmp/tmp/sys-sparc64/go.mod"                                                                                                                
GOWORK=""                                                                                                                                              
CGO_CFLAGS="-g -O2"                                                                                                                                    
CGO_CPPFLAGS=""                                                                                                                                        
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build878311104=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

What did you do?

Running with no changes, I get:

golang.org/x/sys/cpu
golang.org/x/sys/unix
# golang.org/x/sys/cpu
/root/go/pkg/mod/golang.org/x/sys@v0.0.0-20200615200032-f1bc736245b1/cpu/cpu.go:17:30: error: reference to undefined name ‘cacheLineSize’
   17 | type CacheLinePad struct{ _ [cacheLineSize]byte }
      |                              ^
github.com/matrix-org/dendrite/cmd/kafka-producer
# golang.org/x/sys/unix
/root/go/pkg/mod/golang.org/x/sys@v0.0.0-20200615200032-f1bc736245b1/unix/dirent.go:16:12: error: reference to undefined name ‘isBigEndian’
   16 |         if isBigEndian {
      |            ^

After this commit, and trying to write unix/asm_linux_sparc64.s(started with just copying over loong64's), I get this error:

# golang.org/x/sys/unix
unix/asm_linux_sparc64.s:9:10: fatal error: textflag.h: No such file or directory
    9 | #include "textflag.h"
      |          ^~~~~~~~~~~~
compilation terminated.

I assume this is why there was a gc requirement. On sparc64, gccgo is the only option.

So, how would I get an assembly file compileable with gccgo? Is there a way to make it work with textflag.h, or a way to make it work without it?

Thanks for your time, I am really excited about getting x/sys running on my sparcbox, and hopefully soon everyone elses =^D

[ianlancetaylor]

The various asm files in x/sys are not used by gccgo. For example, asm_linux_loong64.s has a go:build line that says that the file is only built with the "gc" tag. That means it is only built when using the gc compiler, not when using gccgo.

You'll need to add a cpu/cpu_sparc64.go file to define cacheLineSize appropriately, and you'll need to figure out why the sparc64 build tag was not used to pick up unix/endian_big.go.

[rrrogalski]

Thanks for the info!! Glad to know I don't need to learn sparc64 asm :D.

I added the cpu_sparc64.go file, and it builds successfully! The bigendian error disappeared after that. Black magic o.o

I am a bit confused for what to set cacheLineSize to. Different sparc64 CPUs have different values, looking at cpu_arm64.go it seems like I should use the highest of the bunch?

Sadly, it looks like it fails a good deal of tests:

root@sparcy /var/tmp/tmp/sys-sparc64 # go test ./...
ok      golang.org/x/sys/cpu    0.174s
ok      golang.org/x/sys/execabs        0.170s
ok      golang.org/x/sys/internal/unsafeheader  0.179s
--- FAIL: TestPktInfo (0.01s)
    --- FAIL: TestPktInfo/udp4 (0.00s)
        ???:1: Got packet from 0.0.0.0, want 127.0.0.2
    --- FAIL: TestPktInfo/udp6 (0.00s)
        ???:1: Got packet from ::, want 2001:db8::1
--- FAIL: TestDevices (0.00s)
    --- FAIL: TestDevices//dev/null_1:3 (0.00s)
        ???:1: for /dev/null Major(0x0) == 0, want 1
        ???:1: for /dev/null Minor(0x0) == 0, want 3
        ???:1: for /dev/null Mkdev(1, 3) == 0x103, want 0x0
    --- FAIL: TestDevices//dev/zero_1:5 (0.00s)
        ???:1: for /dev/zero Major(0x0) == 0, want 1
        ???:1: for /dev/zero Minor(0x0) == 0, want 5
        ???:1: for /dev/zero Mkdev(1, 5) == 0x105, want 0x0
    --- FAIL: TestDevices//dev/random_1:8 (0.00s)
        ???:1: for /dev/random Major(0x0) == 0, want 1
        ???:1: for /dev/random Minor(0x0) == 0, want 8
        ???:1: for /dev/random Mkdev(1, 8) == 0x108, want 0x0
    --- FAIL: TestDevices//dev/full_1:7 (0.00s)
        ???:1: for /dev/full Major(0x0) == 0, want 1                       
        ???:1: for /dev/full Minor(0x0) == 0, want 7                       
        ???:1: for /dev/full Mkdev(1, 7) == 0x107, want 0x0                
    --- FAIL: TestDevices//dev/urandom_1:9 (0.00s)                         
        ???:1: for /dev/urandom Major(0x0) == 0, want 1                    
        ???:1: for /dev/urandom Minor(0x0) == 0, want 9                    
        ???:1: for /dev/urandom Mkdev(1, 9) == 0x109, want 0x0             
    --- FAIL: TestDevices//dev/tty_5:0 (0.00s)                             
        ???:1: for /dev/tty Major(0x0) == 0, want 5                        
        ???:1: for /dev/tty Mkdev(5, 0) == 0x500, want 0x0                 
--- FAIL: TestStatx (0.00s)
    ???:1: Statx: returned stat mode does not match Stat
    ???:1: Statx: returned stat ctime does not match Stat
    ???:1: Statx: returned stat mtime does not match Stat
    ???:1: Statx: returned stat mode does not match Lstat
    ???:1: Statx: returned stat ctime does not match Lstat
    ???:1: Statx: returned stat mtime does not match Lstat
--- FAIL: TestFstatat (0.00s)
    ???:1: Stat/Fstatat: got dev 30, expected 128849018880
    ???:1: Stat/Fstatat: got ino 1, expected 142541374685184
    ???:1: Stat/Fstatat: got mode 33188, expected 0
    ???:1: Stat/Fstatat: got size 8192, expected 1663186338
    ???:1: Lstat/Fstatat: got dev 30, expected 128849018880
    ???:1: Lstat/Fstatat: got ino 1, expected 178116588797952
    ???:1: Lstat/Fstatat: got mode 41471, expected 0
    ???:1: Lstat/Fstatat: got size 8192, expected 1663186338
--- FAIL: TestFchmodat (0.00s)
    ???:1: Fchmodat: failed to change symlink mode: expected -rwxrwxrwx, got ----------
--- FAIL: TestUtimesNanoAt (0.00s)
    ???:1: UtimesNanoAt: wrong mtime: got {0 0}, expected {3333 0}
FAIL
FAIL    golang.org/x/sys/unix   0.988s
ok      golang.org/x/sys/unix/internal/mkmerge  0.290s
?       golang.org/x/sys/windows/mkwinsyscall   [no test files]
FAIL

You can take a look at the cpu_sparc64.go I wrote here.

Any tips on where to go from here? Am a bit lost on where these tests might be failing..

I appreciate your time :)

[ianlancetaylor]

Yes, cacheLineSize should be the largest cache size used on the processor architecture.

The tests suggest that Stat isn't working, which suggests that the Stat_t type in the file ztypes_linux_sparc64.go is somehow incorrect. That is just a guess, though.

[rrrogalski]

aaaaahrg I'm getting stuck on this.

I'm not using the docker method, as I want the information from my running system to be used, to see if there's any differences. Plus, Docker('s deps) requires go 1.18 to build. As far as I'm aware, gcc12 only supports up to go 1.16.

I took a look at Stat_t, and I have 0 idea how to fix this by hand. Looking at mkpost.go, I of course commented out the docker check, but it seems to hang endlessly. Using print messages I narrowed down the freezing to this line:

b, err := ioutil.ReadAll(os.Stdin)

I'm running it directly like this, not sure if this is how I'm supposed to do it:

GOOS="linux" GOARCH="sparc64" go run mkpost.go

I keep looking at the test results, then the tests, and I'm just.... baffled. Like TestPktInfo. The test is so so simple, and it fails miserably. Same thing with the /dev/random test....

I'll keep assuming it's something wrong in ztypes_linux_sparc64.go (or one of the other unix/*linux_sparc64.go files).

I'll keep working on it, but in the meantime, if I can get some hints with this mkpost.go (or any possible alternative ways to get the info I need) I would SUPER greatly appreciate it :D

[rrrogalski]

Update: Writing a script to try every possible value combination in Stat_t. Will leave overnight, maybe, just maybe, by morning I'll have something that passes tests :D

[rrrogalski]

So I finally understood what to do to generate these the right way. I don't really know go, so they were hard for me to follow, and I misunderstood.
Although when I run go tool cgo -godefs linux/types.go, I get an error about not being able to find abi/abi.h. Which, is generated with mkall.go. However when I go run mkall.go, I get an error about needing to be on amd64. An interesting chicken and the egg scenario :D

Anyways, I'm again back to getting the values by hand. The whole script thing I mentioned above didn't work too well :/.

I'm trying to adapt /usr/include/asm/stat.h, and comparing it to ztypes_linux_sparc64.go, but in the ztypes I see this Timespec field for the time values. At the moment I'm leaving that as is, as I don't understand what it is, but is there a chance this could be incorrect?

[rrrogalski]

Thanks for the tip Ian! I'll take a look at it next chance I get :)

[gopherbot]

Change https://go.dev/cl/820000 mentions this issue: unix: avoid unaligned loads in the Ifreq union accessors

[gopherbot]

Change https://go.dev/cl/820180 mentions this issue: unix: fix VMIN, VEOL and VEOL2 on linux/sparc64