Things required to run bash inside chroot

MG_

Hello, I installed bash in NetBSD using pkg_add. The shell works and I want to run it inside a chroot dir. But the bash executable isn't seen while it's there including 3 binary dependencies.
If I place it in the chroots /bin, which I don't agree with, it still says no such file or directory, while the program is there with executable permissions, recognized executable format and dependencies. It also works from outside the chroot dir, but inside it's "file not found", which is not true. All commands can see the file except when called as chroot parameter from outside.
I found several web mentions of the problem but no cause or solution. What am I missing? Do I have to refresh the dll stack like with ldconfig -R in FreeBSD? Also, I copied most of the native system parts tp the chroot dir. Does anything still need special file flags?

rvp

MG_ You probably forgot to copy ld.elf_so. This list works:

/tmp# find chroot
chroot/
chroot/bin
chroot/bin/bash
chroot/dev
chroot/dev/MAKEDEV
chroot/dev/MAKEDEV.local
chroot/dev/console
chroot/dev/constty
chroot/dev/drum
chroot/dev/kmem
chroot/dev/mem
chroot/dev/null
chroot/dev/full
chroot/dev/zero
chroot/dev/klog
chroot/dev/ksyms
chroot/dev/random
chroot/dev/urandom
chroot/dev/tty
chroot/dev/stdin
chroot/dev/stdout
chroot/dev/stderr
chroot/etc
chroot/etc/passwd
chroot/etc/master.passwd
chroot/usr
chroot/usr/lib
chroot/usr/lib/libterminfo.so.2
chroot/usr/lib/libc.so.12
chroot/usr/lib/libintl.so.1
chroot/usr/libexec
chroot/usr/libexec/ld.elf_so

/tmp# chroot -u rvp /tmp/chroot /bin/bash
bash-5.3$

The files inside dev were created using ./MAKEDEV std, and those in etc were copied from the system (you should, actually, generate a new one with only the reqd. entries). But really, you should just use mksandbox.

MG_

rvp It works now. I didn't notice both /libexec and /usr/libexec have to contain ld_elf.so. Probably a hard link usually but I copied it now. But why doesn't chroot say "missing library"? That would have saved me some time...

rvp

MG_ But why doesn't chroot say "missing library"?

It could if there were an ENOLOADER error in the system. But, there isn't, so when chroot does an exec() of /bin/bash, the kernel will try to load the "interpreter" specified in the binary first, and when that can't be found must return ENOENT -- which is what the exec() in chroot fails with.

MG_ If I copy /bin/ls to /usr/local/bin/aaa in the chroot and mess the content up wirh an editor, chroot says: Shared object has no run-time symbol table. That's better... It's found but can't be executed.

Well, this is very different. Here, ld.elf_so does run. It examines the binary, finds things missing, or messed up, and can print an appropriate error.

FWIW, I think ldd on NetBSD should list the dynamic loader which'll be used, too--just like the GNU ldd on Linux.

nia

The reason it isn't a "missing library" error is that ld_elf.so is literally the interpreter for dynamic executables.

MG_

nia I think it's pretty contradictive to fail at executing a file and then say it can't be found because it's not a statically linked executable like the other programs in /bin, usr/bin,etc. They can also be found, regardless of the content.

If I copy /bin/ls to /usr/local/bin/aaa in the chroot and mess the content up wirh an editor, chroot says: Shared object has no run-time symbol table. That's better... It's found but can't be executed.

MG_

rvp
It seems chroot says nothing on an empty file, assumes running shell input if it's ASCII, gets unpredicrable if the target is binary data interpreted as ahell code and says "file not found" if it's a valid executable but it can't be executed because ld_elf.so is missing.
Shouldn't chroot just say the same as when you delete the native ld_elf.so and try to run a program after that?.

rvp

MG_ It seems chroot says nothing on an empty file, assumes running shell input if it's ASCII, gets unpredicrable if the target is binary data interpreted as ahell code

All these are due to the way exec() works. The first thing the kernel checks is if the executable has a valid signature: the binary starts with \177ELF (ELF binary) or #! (she-bang scripts). If neither of these are found, then
the kernel will return ENOEXEC and the exec() routine in libc will then re-execute the file as /bin/sh filename.

And, for sh

0-length files are perfectly valid.
ASCII files (or, mostly ASCII files) are assumed to be shell-scripts.
binary files may produce syntax errors, or emit a proper message like, Cannot execute ELF binary. Depends on the shell.

This is historical shell behaviour and a lot of things will break if any of these are changed.

MG_ Shouldn't chroot just say the same as when you delete the native ld_elf.so and try to run a program after that?.

chroot is not a shell. Why should it scan the program? If the kernel returns ENOENT, then that's what chroot will print--No such file or directory.

MG_

Btw, I'm trying to "port" my custom FreeBSD RAM-only system to NetBSD as experiment. It's looking good. Almost a working chroot build directory. Next step is compile about 150 packages including compilers that are the build deps for everything. Anyone interested, pm for the code until now.

MG_

rvp Chroot should actually have a -verbose option to indicate what's happening with the given command string because it can be at least 2 different things. I understand this is so fundamental that changing something comes with a security checklist and other work. Not expecting that for only 1 more detailed error message.

It also runs text as script without shebang start line. The current console and shell where chroot is called take over the I/O from the command, right? Couldn't it just give the same reaction as usual, not lifted to a virtual / tree? And start with chroot: to make clear who's talking? Or is this giving information to the outside the actual problem? I didn't think about that...

rvp

MG_ Chroot should actually have a -verbose option to indicate what's happening with the given command string

ktrace is better 😉.

MG_ It also runs text as script without shebang start line.

Yes, it has to. If the kernel returns ENOEXEC, then exec() will run /bin/sh filename

MG_ The current console and shell where chroot is called take over the I/O from the command, right? Couldn't it just give the same reaction as usual, not lifted to a virtual / tree?

Didn't quite get this...

MG_ And start with chroot: to make clear who's talking?

I don't think you'll make this mistake more'n once. If you've see this once, you'll know instantly the next time around.

MG_

rvp If you pass readable text as chroot command parameter, it's processed for the current running console and shell where chroot is started. This is route 1.
If you pass the name of an executable that exists but not runs inside the chroot directory as command parameter, it returns something like file not found, like it doesn't exist and exits with an error. This is route 2.
I think this is done to avoid that programs in a chrootdir reveal things to outside, whatever is there. Kind of paranoid but no problem. Telling it can't be executed without a reason is already giving information: the program exists.

rvp

MG_ If you pass readable text as chroot command parameter, it's processed for the current running console and shell where chroot is started. This is route 1.
If you pass the name of an executable that exists but not runs inside the chroot directory as command parameter, it returns something like file not found, like it doesn't exist and exits with an error. This is route 2.
I think this is done to avoid that programs in a chrootdir reveal things to outside

Hmm, I'm pretty sure that chroot doesn't take 2 routes as you put it. It just calls execve() if you pass a command, or an interactive shell if you don't. It's execve() which runs the command again if the kernel returns ENOEXEC.

MG_

rvp That was kind of my point. Chroot doesn't say ENOEXEC but "doesn't exist".

rvp

MG_ Chroot doesn't say ENOEXEC but "doesn't exist".

If the "interpereter" doesn't exist, then the kernel returns ENOENT in exec() and that's what chroot will print. Don't see where the confusion is...

MG_

rvp The confusion started at the point I placed a working bash executable in a chroot directory but chroot said "not found", which obviously wasn't true. This takes time to find out what's happening because the error is non-substantial. I excpect for security reasons. It's OS details.

Difference with FreeBSD:
mkdir /tmp/a
cp /bin/ls /tmp/a
chroot /tmp/a /ls
result: ELF interpreter /libexec/ld-elf.so.1 not found, error 2
Abort trap

After adding ld-elf.so:
ld-elf.so.1: Cannot open "/usr/lib/libstdbuf.so"

rvp

MG_ Difference with FreeBSD:

mkdir /tmp/a
cp /bin/ls /tmp/a
chroot /tmp/a /ls
result: ELF interpreter /libexec/ld-elf.so.1 not found, error 2
Abort trap

Oh, that's just a convenience the FreeBSD kernel provides:
https://github.com/freebsd/freebsd-src/blob/main/sys/kern/imgact_elf.c#L1092

OpenBSD, however's, just recently removed this from theirs:
https://github.com/openbsd/src/commit/7db5826dada7d33c09da396a91ea673ce2e93737

One obvious problem with the kernel printing directly to userspace is this:

$ uname -a
FreeBSD CoreBook.local 15.0-RELEASE-p4 FreeBSD 15.0-RELEASE-p4 GENERIC amd64
$ mkdir a
$ cp /bin/sh a
$ sudo chroot -u rvp /tmp/a ./sh >log.txt 2>&1
ELF interpreter /libexec/ld-elf.so.1 not found, error 2
Abort trap		sudo chroot -u rvp /tmp/a ./sh > log.txt 2>&1
$ ls -l log.txt
-rw-------  1 rvp wheel 0 20 Mar 22:41 log.txt
		#      ^^^
		# where's my log, dude!?
$

redirections don't work; which is very confusing too.

Anyway, as I said before, this isn't much of a hassle to figure out if you've seen it before. The real headache is for shared objects loaded withdlopen()--like PAM modules, or i18n support objects. With those, one has to run strings(1) on the binaries (and the libraries, as well as any config. files!) to get a clue as to what modules are missing.

MG_

rvp Hm, never tried to log chroot output from outside. What happens if you precede with stdbuf -o 0 ? It could be 1 time stdout buffer getting lost when chroot ends. And &3 also shows nothing? Some programs use it but I never understood the purpose.

rvp

MG_ What happens if you precede with stdbuf -o 0 ?

It's the kernel who's printing directly to your terminal, so nothing userspace can do will work--incl. redirections (which is what I wanted to show).

MG_

rvp The kernel is looking for the file that's given as chroot parameter. I think chroot not giving feedback is actually considered because mentioning that a dll or the actual file does or doesn't exist inside the separate directory tree is revealing things to a user that doesn't have to have access, even while logged in as a host user. Theoretically, chroot could be used to circumvent the systems permission wall and retrieve confidential information of other users like files existing.

If the kernel says "file not found" or something like that, it can also tell the path and filename it was looking for, but it gives an intentionally bogus error message. It wasn't really looking for a file in some storage in this case.