RUST_BACKTRACE

Anyone here running OpenBSD and FreeBSD having the Rust compiler installed that would't mind doing me a favour?

TIA

rvp

pin Lay it on me. I was gonna to finish-up your other rust-stuff this coming week. Better send an email, though, eh?

rvp Thanks! Create test.rs and put this inside

fn main() { panic!("explode!") }. Compile it, rustc test.rs and run the resulting binary ./test

Now, could I get the output of
RUST_BACKTRACE=1 ./test and RUST_BACKTRACE=full ./test on OpenBSD and FreeBSD?

No more, no less. Appreciated!

@rvp Thank you for these 🤗

So OpenBSD behaves the same as NetBSD and FreeBSD behaves like Linux and you actually get debug symbols.

EDIT: Adding the outputs here for reference.

Rust 1.51.0 on OpenBSD 6.9
==========================

Script started on Sat May 14 07:51:02 2022
$ rustc --version
rustc 1.51.0
$ uname -a
OpenBSD x202e.localdomain 6.9 GENERIC#464 amd64
$ env RUST_BACKTRACE=1 ./test
thread 'main' panicked at 'explode!', test.rs:2:5
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
$ env RUST_BACKTRACE=full ./test
thread 'main' panicked at 'explode!', test.rs:2:5
stack backtrace:
   0:      0xb7ce45f70cd - <unknown>
   1:      0xb7ce461f7bd - <unknown>
   2:      0xb7ce45ef484 - <unknown>
   3:      0xb7ce4609df4 - <unknown>
   4:      0xb7ce460996f - <unknown>
   5:      0xb7ce460a40c - <unknown>
   6:      0xb7ce45ed7e7 - <unknown>
   7:      0xb7ce45ed418 - <unknown>
   8:      0xb7ce45ed72f - <unknown>
   9:      0xb7ce45ed52c - <unknown>
  10:      0xb7ce45ed58a - <unknown>
  11:      0xb7ce45ed44d - <unknown>
  12:      0xb7ce45edf80 - <unknown>
  13:      0xb7ce460b69d - <unknown>
  14:      0xb7ce45edf5c - <unknown>
  15:      0xb7ce45ed552 - <unknown>
  16:      0xb7ce45ec921 - <unknown>
$ ^D

Script done on Sat May 14 07:52:12 2022

Rust 1.59.0 on FreeBSD 13.0-RELEASE
===================================

Script started on Sat May 14 07:36:56 2022
$ uname -a
FreeBSD x202e.localdomain 13.0-RELEASE-p11 FreeBSD 13.0-RELEASE-p11 #0: Tue Apr  5 18:54:35 UTC 2022     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
$ rustc --version
rustc 1.59.0
$ env RUST_BACKTRACE=1 ./test
thread 'main' panicked at 'explode!', test.rs:2:5
stack backtrace:
   0: std::panicking::begin_panic
   1: test::main
   2: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
$ env RUST_BACKTRACE=full ./test
thread 'main' panicked at 'explode!', test.rs:2:5
stack backtrace:
   0:     0x2957cf5a752b - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h58a1696b4cc8c064
   1:     0x2957cf5c374c - core::fmt::write::h36702066c14463f8
   2:     0x2957cf597672 - std::io::Write::write_fmt::h9ee00d1919dd7f6a
   3:     0x2957cf59c832 - std::panicking::default_hook::{{closure}}::he0727aa3c06036d7
   4:     0x2957cf59c44b - std::panicking::default_hook::hbbb3aab2e413a40f
   5:     0x2957cf59ce22 - std::panicking::rust_panic_with_hook::h889bd528eff56827
   6:     0x2957cf59237e - std::panicking::begin_panic::{{closure}}::h9c91aa21e456736b
   7:     0x2957cf592665 - std::sys_common::backtrace::__rust_end_short_backtrace::hf50bf842715648bf
   8:     0x2957cf592302 - std::panicking::begin_panic::hbfb85dc97b129898
   9:     0x2957cf592759 - test::main::h7289cb927c7e1740
  10:     0x2957cf592823 - core::ops::function::FnOnce::call_once::h2cbbaaad17aaab73
  11:     0x2957cf592609 - std::sys_common::backtrace::__rust_begin_short_backtrace::h090f8eb0a2e90c6e
  12:     0x2957cf592fc9 - std::rt::lang_start::{{closure}}::h2a7c3802ff305c48
  13:     0x2957cf594371 - std::rt::lang_start_internal::h07827b591a12348c
  14:     0x2957cf592fb1 - std::rt::lang_start::hc47027af7a72bc7d
  15:     0x2957cf5927c3 - main
$ ^D

Script done on Sat May 14 07:38:33 2022

rvp

pin Actually, the symbol table is present in the file (readelf -s ...), but, it looks like support for frame unwinding (with or without PIC/PIE) is not there yet.

If you compile the code without PIC, then you can get a backtrace with addr2line:

$ rustc -g -C relocation-model=dynamic-no-pic test.rs 
$ env RUST_BACKTRACE=full ./test 2>&1 |
> awk '$2 ~ /0x[[:xdigit:]]+$/ { print $2 }' |
> addr2line -fp -e test | c++filt -s rust
<std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt at ??:?
core::fmt::write at ??:?
std::io::Write::write_fmt at ??:?
std::panicking::default_hook::{{closure}} at std.f6bd66af-cgu.13:?
std::panicking::default_hook at std.f6bd66af-cgu.13:?
std::panicking::rust_panic_with_hook at ??:?
std::panicking::begin_panic::{{closure}} at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/panicking.rs:526
std::sys_common::backtrace::__rust_end_short_backtrace at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/sys_common/backtrace.rs:142
std::panicking::begin_panic at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/panicking.rs:525
test::main at /tmp/test.rs:2
core::ops::function::FnOnce::call_once at /work/lang/rust/work/rustc-1.58.1-src/library/core/src/ops/function.rs:227
std::sys_common::backtrace::__rust_begin_short_backtrace at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/sys_common/backtrace.rs:126
std::rt::lang_start::{{closure}} at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/rt.rs:145
std::rt::lang_start_internal at ??:?
std::rt::lang_start at /work/lang/rust/work/rustc-1.58.1-src/library/std/src/rt.rs:144
main at ??:?
___start at ??:?
$

addr2line similarly fails to print backtraces for C programs if they're compiled as PIC/PIE.

Edit: All this is on NetBSD 9.2_STABLE with rust from 2022Q1.

Anyone here with OpenBSD-7.1 that can repeat the test? @rvp rust-1.51 is rather old and I wounder if the behaviour is still the same with a more recent compiler. I'd guess it is but, if someone could test...
Thanks!

Thank you!

rvp

pin Actually, if you use the correct library (libexecinfo instead of libunwind) then you can get a proper backtrace even with PIE executables:

unwind.c

/**
 * cc -pie -fPIE unwind.c -lexecinfo
 * ./a.out
 */
#include <stdio.h>
#include <stdlib.h>

static void f1(void);
static void f2(void);
static void f3(void);
static void f4(void);
static void show_backtrace(void);

int
main(void)
{
	f1();
	return 0;
}

static void f1(void) {
	f2();
}
static void f2(void) {
	f3();
}
static void f3(void) {
	f4();
}
static void f4(void) {
	show_backtrace();
}

#include <execinfo.h>
static void
show_backtrace(void)
{
	enum { MAX = 256 };
	void* locs[MAX];
	char** s;
	size_t i, n;

	if ((n = backtrace(locs, MAX)) <= 0)
		return;
	s = backtrace_symbols_fmt(locs, n, "\t%a - %n");
	if (s == NULL)
		return;
	for (i = 0; i < n; i++)
		printf("%s\n", s[i]);
	free(s);
}

$ cc -pie -fPIE unwind.c -lexecinfo
$ ./a.out 
        0x20c8a2d8bbec - show_backtrace
        0x20c8a2d8bbc9 - f4
        0x20c8a2d8bbb9 - f3
        0x20c8a2d8bba9 - f2
        0x20c8a2d8bb99 - f1
$

Somebody's gotta teach this to rust.

@pin: a quick test would be to add execinfo for the NetBSD builds also, like for FreeBSD (in rust-1.60.0):

./library/std/src/sys/unix/mod.rs:271
Add:
#[link(name = "execinfo")]

rvp Even better 😃

he32

The patch to link with -lexecinfo was apparently added in NetBSD's pkgsrc version of rust 1.50.0, ref.

https://github.com/NetBSD/pkgsrc-wip/blob/master/rust/patches/patch-library_std_src_sys_unix_mod.rs

However, the rust build log doesn't actually contain the string "execinfo", so I don't quite understand whether that's actually taking effect. Also, recently the execinfo functions were made visible via a patch to the libc crate, ref.

https://github.com/NetBSD/pkgsrc-wip/blob/master/rust/patches/patch-vendor_libc-0.2.116_src_unix_bsd_netbsdlike_netbsd_mod.rs

patterned after what's done for OpenBSD and FreeBSD already (yes, I patched the other two libc crates as well, but this appears to be the one which is being used, it's the only one which needed its checksum fixed to complete the build.)

However, I suspect that rust has not yet been taught to use the execinfo functions to implement backtrace functionality, and only relies on LLVM's unwind library, ref. what I find in rustc-1.60.0-src/library/unwind/src/libunwind.rs -- everywhere else I search I only see references to LLVM's unwind library for this functionality. Am I just not good enough at navigating the dependencies and the conditionals for pulling in the various parts?

rvp

he32 [...] yes, I patched the other two libc crates as well,[...]

Looks like you got them all. (I figured it using:)

$ fgrep -r backtrace rustc-1.60.0-src/ | fgrep freebsd
$ fgrep -r backtrace rustc-1.60.0-src/ | fgrep netbsd

he32 However, I suspect that rust has not yet been taught to use the execinfo functions

$ fgrep -r -- -lexecinfo rustc-1.60.0-src
rustc-1.60.0-src/vendor/openssl-src/openssl/Configure:                push @{$config{ex_libs}}, "-lexecinfo";
rustc-1.60.0-src/src/llvm-project/clang/lib/Driver/ToolChains/CommonArgs.cpp:    CmdArgs.push_back("-lexecinfo");
rustc-1.60.0-src/src/llvm-project/compiler-rt/test/msan/lit.cfg.py:  clang_msan_cflags += ["-lexecinfo", "-fPIC"]
rustc-1.60.0-src/src/llvm-project/compiler-rt/lib/orc/unittests/CMakeLists.txt:  append_list_if(COMPILER_RT_HAS_LIBEXECINFO -lexecinfo ORC_UNITTEST_LINK_FLAGS)
rustc-1.60.0-src/src/llvm-project/compiler-rt/lib/xray/tests/CMakeLists.txt:  append_list_if(COMPILER_RT_HAS_LIBEXECINFO -lexecinfo XRAY_UNITTEST_LINK_FLAGS)
$

Only one file, Driver/ToolChains/CommonArgs.cpp seems relevant here and that does link -lexecinfo on all the BSDs--but, I think that it's brought in only for Sanitizers, and, their tests.

he32 everywhere else I search I only see references to LLVM's unwind library for this functionality.

The unwind library will only get you the stack frames, I think. You need to grub around in the executable's ELF sections (either in .dynsym or .symtab and their string-table sections depending on whether -rdynamic is passed or not when compiling) to translate the IP location into a function name:

unwind.c

/**
 * cc -pie -fPIE unwind.c
 * ./a.out
 
https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-s.html
https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-AMD64/LSB-Core-AMD64/libgcc-s.html
https://dwarfstd.org/Home.php

 */
#include <stdio.h>
#include <stdlib.h>
#include <unwind.h>

static void f1(void);
static void f2(void);
static void f3(void);
static void f4(void);
static void pr_bt(void);
static _Unwind_Reason_Code bt_fn(struct _Unwind_Context *ctx, void *arg);

int
main(void)
{
	f1();
	return 0;
}

static void f1(void) {
	f2();
}
static void f2(void) {
	f3();
}
static void f3(void) {
	f4();
}
static void f4(void) {
	pr_bt();
}

static void
pr_bt(void)
{
	_Unwind_Backtrace(bt_fn, NULL);	
}

static _Unwind_Reason_Code
bt_fn(struct _Unwind_Context *ctx, void *arg)
{
	_Unwind_Ptr ip = _Unwind_GetIP(ctx);
	if (ip == (_Unwind_Ptr)NULL)
		return _URC_END_OF_STACK;
	printf("0x%lx\n", ip);
	return _URC_NO_REASON;
}

rvp

he32 In rustc-1.60.0-src/library/backtrace/src/symbolize/gimli.rs line 179, netbsd has to be added because it has dl_iterate_phdr
Also in:

rustc-1.60.0-src/library/backtrace/src/symbolize/gimli/elf.rs
rustc-1.60.0-src/vendor/backtrace/src/symbolize/gimli/elf.rs

fn debug_path_exists() will need fixing up for /usr/libdata/debugon NetBSD.

he32

Thanks for the hints! I have now committed an update to pkgsrc-wip's rust package which makes RUST_BACKTRACE work on NetBSD (again?). The commit can be seen at

https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=commit;h=5c8a552378a868042241aab6a7af3a14a2a7e102