can't build lang/go pkgsrc

pfr

This is going to be the last thread I make about failed pkgsrc builds.

trying to build pkgsrc/lang/go (or any package dependent on Go fails with the following error:

[...]
===> Building binary package for go118-1.18.10nb1
=> Creating binary package /home/dave/pkgsrc/packages/All/go118-1.18.10nb1.tgz
===> Installing binary package of go118-1.18.10nb1
=> Returning to build of go121-1.21.10
=> Tool dependency mktools-[0-9]*: found mktools-20220614
=> Tool dependency cwrappers>=20150314: found cwrappers-20220403
=> Full dependency bash-[0-9]*: found bash-5.2.26nb1
=> Full dependency perl>=5.0: found perl-5.38.2
===> Checking for vulnerabilities in go121-1.21.10
===> Overriding tools for go121-1.21.10
===> Extracting for go121-1.21.10
/bin/rm -r -f /home/dave/pkgsrc/lang/go121/work/go/test/fixedbugs/issue27836*
===> Patching for go121-1.21.10
=> Applying pkgsrc patches for go121-1.21.10
===> Creating toolchain wrappers for go121-1.21.10
===> Configuring for go121-1.21.10
=> Substituting "paths" in src/crypto/x509/root_solaris.go
=> Replacing bash interpreter in lib/time/update.bash misc/arm/a misc/cgo/fortran/test.bash misc/wasm/go_js_wasm_exec misc/wasm/go_wasip1_wasm_exec src/all.bash src/bootstrap.bash src/buildall.bash src/clean.bash src/cmd/compile/internal/ssa/_gen/cover.bash src/cmd/go/mkalldocs.sh src/cmd/vendor/golang.org/x/sys/plan9/mkall.sh src/cmd/vendor/golang.org/x/sys/plan9/mkerrors.sh src/cmd/vendor/golang.org/x/sys/unix/mkall.sh src/cmd/vendor/golang.org/x/sys/unix/mkerrors.sh src/cmd/vendor/golang.org/x/sys/windows/mkerrors.bash src/cmd/vendor/golang.org/x/sys/windows/mkknownfolderids.bash src/crypto/internal/boring/build.sh src/crypto/internal/boring/build-boring.sh src/crypto/internal/boring/build-goboring.sh src/go/doc/comment/mkstd.sh src/internal/trace/mkcanned.bash src/cmp.bash src/make.bash src/race.bash src/run.bash src/runtime/race/mkcgo.sh src/syscall/mkall.sh src/syscall/mkerrors.sh src/syscall/mksysnum_plan9.sh.
WARNING: [replace-interpreter] Skipping non-existent file "misc/arm/a".
WARNING: [replace-interpreter] Skipping non-existent file "misc/cgo/fortran/test.bash".
WARNING: [replace-interpreter] Skipping non-existent file "src/cmd/go/mkalldocs.sh".
=> Replacing Perl interpreter in src/net/http/cgi/testdata/test.cgi src/regexp/syntax/make_perl_groups.pl src/syscall/*.pl.
===> Building for go121-1.21.10
cd /home/dave/pkgsrc/lang/go121/work/go/src &&  env  GOROOT_BOOTSTRAP=/usr/pkg/go118  GOROOT_FINAL=/usr/pkg/go121    GOCACHE=/home/dave/pkgsrc/lang/go121/work/.cache/go-build  /usr/pkg/bin/bash ./make.bash -v
Building Go cmd/dist using /usr/pkg/go118. (go1.18.10 netbsd/amd64)
cmd/dist

Building Go toolchain1 using /usr/pkg/go118.
# cd /home/dave; git status --porcelain
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
go tool dist: FAILED: /usr/pkg/go118/bin/go install -tags=math_big_pure_go compiler_bootstrap purego -v bootstrap/cmd/...: exit status 1
*** Error code 2

Stop.
make[2]: stopped in /home/dave/pkgsrc/lang/go121
*** Error code 1

Stop.
make[1]: stopped in /home/dave/pkgsrc/lang/go121
*** Error code 1

Stop.
make: stopped in /home/dave/pkgsrc/lang/go

I then run export GIT_DISCOVERY_ACROSS_FILESYSTEM=1 and try again...

This time I tried building lang/go121

~/pkgsrc/lang/go121 λ doas make install clean
=> Bootstrap dependency digest>=20211023: found digest-20220214
===> Checking for vulnerabilities in go121-1.21.10
===> Building for go121-1.21.10
cd /home/dave/pkgsrc/lang/go121/work/go/src &&  env  GOROOT_BOOTSTRAP=/usr/pkg/go118  GOROOT_FINAL=/usr/pkg/go121    GOCACHE=/home/dave/pkgsrc/lang/go121/work/.cache/go-build  /usr/pkg/bin/bash ./make.bash -v
Building Go cmd/dist using /usr/pkg/go118. (go1.18.10 netbsd/amd64)
cmd/dist

Building Go toolchain1 using /usr/pkg/go118.
# cd /home/dave; git status --porcelain
fatal: not a git repository (or any of the parent directories): .git
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
go tool dist: FAILED: /usr/pkg/go118/bin/go install -tags=math_big_pure_go compiler_bootstrap purego -v bootstrap/cmd/...: exit status 1
*** Error code 2

Stop.
make[1]: stopped in /home/dave/pkgsrc/lang/go121
*** Error code 1

Stop.
make: stopped in /home/dave/pkgsrc/lang/go121

Something is wrong with my system and I have no idea what. Re-installation imminent.

remiliascarlet

pfr fatal: not a git repository (or any of the parent directories): .git

I've seen this so many times in projects that are so Git-centric, they forget that if you put the source code in an archive file, the .git directory won't get included, making the entire project not compilable.
But if you're using amd64, i386, arm, or aarch64 as your CPU architecture, you can get the pre-compiled binary directly from go.dev:
https://go.dev/dl/

On this page, you click "Other ports", and scroll down until you see the 4 binaries for NetBSD.
Since it's already pre-compiled, you can simply put it in /usr/local, and then do ln -s /usr/local/go/bin/go /usr/local/bin/go and ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt.

Cannot reproduce

remiliascarlet This is a workaround, not a proper solution. The thing is, there's something fishy with @pfr system as this is not the first reported failure on packages that are known to be working just fine.

rvp

pfr Something is wrong with my system and I have no idea what.

Try building your packages as a new, bog-standard user, say bld. bld should be a member of wheel so that pkgsrc can su/sudo/... when it wants to become root. Make no other changes to his environment. And, when you want to become bld, use su -l bld or sudo -i -u bld to do a full login.

pfr

rvp could I not simply log in as root?
Because I tried that and got the same error

EDIT

OK, user bld can indeed build go
Time to troubleshoot my user configuration I suppose... But, where to start?

Or, just delete my current user and start over...

pfr Or, just delete my current user and start over...

Sounds like a plan.

pfr Time to troubleshoot my user configuration I suppose... But, where to start?

Import one thing at the time until you find what's breaking it.

rvp

pfr Or, just delete my current user and start over...

Why? Just build everything (kernel & pkgsrc) as bld from now on.

pfr

rvp hmmmm I suppose I could, but I dont really want more than one user on my machine. I think I'd prefer to just start fresh and just find out what is wrong with my current user's config. I dont really want to su -l bld && make install each time I need to install/update packages. Although, I suppose having a seperate user for which to build and install everything under could be good, less likely to run into problems?

I was actually thinking of bootstrapping pkgsrc as --unprivileged for my new user. Haven't decided yet if that's a good idea or not.

I think at this stage, the cleanest way is to move all my config's over to my new user bld and through process of elimination I can find out what is causing issues? I'm not exactly sure how this will work though or what consequences there may be.

rvp

pfr Although, I suppose having a seperate user for which to build and install everything under could be good, less likely to run into problems?

Yes.

pfr I was actually thinking of bootstrapping pkgsrc as --unprivileged for my new user. Haven't decided yet if that's a good idea or not.

I don't think you need to bootstrap on NetBSD.

pfr

does it matter where the pkgsrc tree is located? I suppose for this multi user setup it would be better placed in /usr/..

Ok, I can possibly make this work, however I'll never know what the issue is which will bug me 😣

Will need to add su -l bld to the fzf pkgsrc script, or just move the scripts back to /usr/local/bin so that bld can use the script 😆 Ahhhh fun fun.

kc9udx

You don't need to bootstrap to use pkgsrc on NetBSD. You do, however, if you want to use it without root permission. You don't have to do it per user though, just once for everyone.

You can put pkgsrc wherever you want. mk.conf specifies the location. But I strongly recommend keeping it at /usr/ especially at this point.

pfr

kc9udx You can put pkgsrc wherever you want. mk.conf specifies the location

It's funny you say that, because amongst all this curfuffle I realised that I did not have PKGSRCDIR set in my mk.conf.. I dont think that was causing the issue though. I've since set it to PKGSRCDIR= ${HOME}/pkgsrc and still having the same issues. I'll be sure to fetch a new pkgsrc tree and extract it into /usr/ and make updates to mk.conf appropriately

pfr Also make sure your cpu does have 6 cores

pfr where the pkgsrc tree is located?

pfr need to add su -l bld to the fzf pkgsrc script

pfr move the scripts back to /usr/local/bin

No further comments

kc9udx

Is it even valid to put ${HOME} in mk.conf?

pfr

kc9udx It would appear so...

kc9udx

pfr where did you find that?
I suspect you "can't", that is, it might not always expand to your home directory. You night need to specify/home/bld/pkgsrc (or whatever user)

I guess what I want to say is, since you're having trouble, and (probably) don't have a specific need to have pkgsrc in your home directory, it would probably be best to leave it in its default location, /usr/pkgsrc

rvp

pfr does it matter where the pkgsrc tree is located?

You can unpack the tree anywhere.

kc9udx You do, however, if you want to use it without root permission.

Not needed. Just unpack the tree (anywhere), then make install clean in the correct dir. The pkgsrc machinery will prompt when it wants rootly powers.

pfr I realised that I did not have PKGSRCDIR set in my mk.conf

Don't need to set this on NetBSD, I think--I just built ed without any of that.

kc9udx Is it even valid to put ${HOME} in mk.conf?

mk.conf is parsed by make so you can put anything make understands in it. For instance, I just now installed GNU ed as a new user test, with only this in /etc/mk.conf:

.if defined(BSD_PKG_MK)

# Use sudo(1) if it exists, else go for the default fallback, su(1).
#

.if exists(${LOCALBASE}/bin/sudo)
SU_CMD=	${LOCALBASE}/bin/sudo /bin/sh -c
.endif

.endif

pfr

rvp If I want to use the same in my mk.conf but with doas instead. Should the if statement be placed at the end or the start of the file? does it matter?

And yes, even though we've been on and on and back and forth about suas sued sued runas etc, I still use doas because I just set nopass :wheel so I dont have to enter a password.
does this look ok?

.if defined(BSD_PKG_MK)

# Use doas(1) if it exists, else go for the default fallback, su(1).
.if exists(${LOCALBASE}/bin/doas)
SU_CMD=	${LOCALBASE}/bin/doas /bin/sh -c
.endif

.endif

VARBASE=                    /var
LOCALBASE=                  /usr/pkg
PKG_DBDIR=                  /usr/pkg/pkgdb
PKG_TOOLS_BIN=              /usr/pkg/sbin
PKGINFODIR=                 info
PKGMANDIR=                  man
MAKE_JOBS=                  6
PKGSRC_USE_MKTOOLS=         yes
ALLOW_VULNERABLE_PACKAGES=  yes
ACCEPTABLE_LICENSES+=       intel-microcode-license mozilla-trademark-license
RUST_TYPE=                  src

Also, you'll be pleased to know that building things with bld user are going smoothly. However, I still really want to know what is going on with my main user AND my root user accounts. It would be fine if it was just my main user, because I could delete it and start fresh with a new user, but I can't delete my root user. Is there any way I can reset my root user back to default?