Few NetBSD-specific bugs

remiliascarlet

Since most of you seem to be using NetBSD as your main BSD, I want to ask a few questions on how to fix certain things.

First one is about the terminal.
Whenever I do something like git diff, the message "WARNING: terminal is not fully functional" shows up before it executes the command, and when I do top, I get "top: can't open termcap file".
Terminal emulator is Suckless st, and shell is ZSH.

Second one is more specific about ZSH.
The home, end, and delete keys don't seem to function as expected in the shell.
In Neovim, the home key works, the end key doesn't, and delete key opens help.txt in a separate buffer.

Third thing is that even though I'm using OhMyZSH, NetBSD is seemingly the only OS where I can't seem to go up in history based on user input.
On OpenBSD, FreeBSD, Linux, OpenIndiana, and even macOS, if I write "ne" and do arrow up, it'll rotate between "neofetch" and "neomutt", and pressing tab will highlight words that start with "ne" so I can autocomplete it.
But on NetBSD, writing "ne" and arrow up will just give me whatever the last command was, and pressing tab will spawn a list of programs starting with "ne", but are not selectable at all, so it behaves much more like GNU Bash.

And the forth thing is more something I can't understand, and that's Neofetch reporting I'm using up 7 GiB of RAM out of 8 GiB, even though I have nothing running apart from ST, CWM, and SSH in the background.
Very similar to how memory usage shows up in FreeBSD, except in FreeBSD I'm using ZFS, but on NetBSD that's just UFS.
So why is that?

As for the first 3 problems, any ideas how to fix those?

nortonham

remiliascarlet
The git error I dont know much about.

remiliascarlet "top: can't open termcap file".

Does this error only occur with ST and ZSH?

remiliascarlet Second one is more specific about ZSH.
The home, end, and delete keys don't seem to function as expected in the shell.

I remember having that issue on FreeBSD (iirc). It's something you have to edit in your .zshrc.

Maybe this link would be helpful:

https://stackoverflow.com/questions/161676/home-end-keys-in-zsh-dont-work-with-putty

remiliascarlet Third thing is that even though I'm using OhMyZSH, NetBSD is seemingly the only OS where I can't seem to go up in history based on user input.

When you installed zsh, did you go through the setup menu? What does your .zshrc look like?

remiliascarlet And the forth thing is more something I can't understand, and that's Neofetch reporting I'm using up 7 GiB of RAM out of 8 GiB, even though I have nothing running apart from ST, CWM, and SSH in the background.

How accurate is neofetch when it displays RAM usage vs. top, free, etc? My guess would be not very, but that's obviously just a hunch on my part. Also, OS's use memory differently. I've seen discussions online in different forums about how linux vs freebsd use RAM, and how top and htop report RAM usage differently. Maybe someone more informed can speak to how it is used on netbsd.

rvp

remiliascarlet WARNING: terminal is not fully functional

Something is messed up in your install, and your issues 1) to 3) all stem from that.
Can you post the output of these commands:

echo $TERM
ls -l /etc/term* /usr/share/misc/terminfo*
infocmp -1

For issue 4), there was this thread quite recently.

remiliascarlet

nortonham The git error I dont know much about.

I gave Git as an example simply because it's the lowest hanging fruit, but happens with other programs too.

nortonham Does this error only occur with ST and ZSH?

It's the only combination I normally use, and the only combination I ever used on NetBSD, so I guess so.

nortonham I remember having that issue on FreeBSD (iirc). It's something you have to edit in your .zshrc.

Maybe this link would be helpful:

https://stackoverflow.com/questions/161676/home-end-keys-in-zsh-dont-work-with-putty

This solved for the home and end keys, but delete is still dysfunctional.

nortonham When you installed zsh, did you go through the setup menu?

No, because I simply rsync over my ~/.zshrc and ~/.local/share/oh-my-zsh across all my computers, and have a little script to apply OS-specific configurations to only specific OS's.
Although I had to make 1 adjustment in OhMyZSH itself before to get the grep command working on OpenBSD, because no --color.

rvp Can you post the output of these commands:

$ echo $TERM
st-256color
$ ls -l /etc/term* /usr/share/misc/terminfo*
zsh: no matches found: /etc/term*
$ infocmp -1
infocmp: no terminal definition found in internal database

nortonham How accurate is neofetch when it displays RAM usage vs. top, free, etc? My guess would be not very, but that's obviously just a hunch on my part. Also, OS's use memory differently. I've seen discussions online in different forums about how linux vs freebsd use RAM, and how top and htop report RAM usage differently. Maybe someone more informed can speak to how it is used on netbsd.

In Neofetch:
Memory: 7861MiB / 7888MiB

In vmstat:

 procs    memory      page                       disks   faults      cpu
 r b      avm    fre  flt  re  pi   po   fr   sr w0 d0   in   sy  cs us sy id
 0 0  4187392  36416 53820  0   0    0    0    1  6  6  502 51260 5647 4 5 91

In free -m:

             total       used       free    buffers
Mem:          7621       7589         32       6029
Swap:            0          0          0

In top:
Memory: 4086M Act, 2003M Inact, 51M Wired, 35M Exec, 5994M File, 48M Free
Swap: / Pools: 1362M Used / Network: 3K In, 3K Out

rvp For issue 4), there was this thread quite recently.

The biggest resource hogs are X (188M), and 2 times st (100M each).
If I add up the RAM usage of all of them, it doesn't add up.

rvp

remiliascarlet

$ ls -l /etc/term* /usr/share/misc/terminfo*
zsh: no matches found: /etc/term*

Gah! bloody zsh (it's as bad as csh). Can you redo the ls -l after:

setopt nonomatch

$ infocmp -1
infocmp: no terminal definition found in internal database

Or maybe, no database found; or, it's corrupted: can't tell. The ls -l will show. The files should be about this size:

$ ls -lh /usr/share/misc/terminfo*
-r--r--r-- 1 root wheel 1.2M Mar 18 22:47 /usr/share/misc/terminfo
-r--r--r-- 1 root wheel 1.9M Mar 18 22:47 /usr/share/misc/terminfo.cdb
$

If they don't look kosher, extract them from the base.tar.xz tarball of your NetBSD version:

tar -C/ -xvf /tmp/base.tar.xz ./usr/share/misc/terminfo*

That should fix things because both 9.3- and 10.0-RELEASE have definitions for st-256color

In top:
Memory: 4086M Act, 2003M Inact, 51M Wired, 35M Exec, 5994M File, 48M Free
Swap: / Pools: 1362M Used / Network: 3K In, 3K Out

That's 6GB for the file-cache which can easily be "evicted" if needed. Try reducing the various caches like I do here.

The biggest resource hogs are X (188M), and 2 times st (100M each).

Are those SIZE (aka VSZ virtual size) or RES (aka RSS resident-set size)? Prob'ly the former (otherwise, st sucks).

If I add up the RAM usage of all of them, it doesn't add up.

This isn't that straightforward to calculate: You can add up the RES column for a rough estimate of the memory used by only processes (rough because of shared pages), but then you have to factor in the kernel memory used for book-keeping (see vmstat -m). There's a lot of overlap in the memory-usage stats reported by top and ps.

nia

Top shows your system using 6GB of memory as file cache. This is fine, normal, and can easily happen after the scheduled tasks run.

st is not present in your terminfo database. Few options here, either you can set TERM to something reasonable (vt100? rxvt?), use a different terminal emulator, or add it.

rvp otherwise, st sucks

🤣 not so suckless

remiliascarlet

rvp $ ls -l /etc/term* /usr/share/misc/terminfo*
zsh: no matches found: /etc/term*
if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) {
window.hljsLoader.highlightBlocks(document.currentScript.parentNode);
}

Gah! bloody zsh (it's as bad as csh). Can you redo the ls -l after:
setopt nonomatch
if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) {
window.hljsLoader.highlightBlocks(document.currentScript.parentNode);
}

$ infocmp -1
infocmp: no terminal definition found in internal database

No use...

$ ls /etc/t*
/etc/ttyaction  /etc/ttys

rvp Or maybe, no database found; or, it's corrupted: can't tell. The ls -l will show. The files should be about this size:

Yes, I confirm seeing the same thing.

$ ls -lh /usr/share/misc/terminfo*
-r--r--r--  1 root  wheel  1.1M Mar 28 17:33 /usr/share/misc/terminfo
-r--r--r--  1 root  wheel  1.7M Mar 28 17:33 /usr/share/misc/terminfo.cdb

rvp If they don't look kosher, extract them from the base.tar.xz tarball of your NetBSD version

Where can I get the individual filesets from?
I mean, in OpenBSD they're available on their FTP servers and most of their mirrors provide it too, but inspecting the NetBSD FTP servers, I can't seem to find it.

Edit: nevermind, found it: https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/amd64/binary/sets/

rvp That's 6GB for the file-cache which can easily be "evicted" if needed. Try reducing the various caches like I do here.

I'll give this a try, thanks.
Very helpful stuff!

rvp Are those SIZE (aka VSZ virtual size) or RES (aka RSS resident-set size)? Prob'ly the former (otherwise, st sucks).

I read from SIZE.
I didn't even notice RES until now.
But RES for X is 29M, and for st is 11M.

rvp This isn't that straightforward to calculate: You can add up the RES column for a rough estimate of the memory used by only processes (rough because of shared pages), but then you have to factor in the kernel memory used for book-keeping (see vmstat -m). There's a lot of overlap in the memory-usage stats reported by top and ps.

Yeah, I guess it's the symptoms of having used Linux for about 30 years, and only having progressively been switching to the BSD's over the past 1 year (starting with servers, then laptops, and then finally the main desktop), and started using the BSD's almost exclusively (still using Linux for gaming and statically linking C programs for Linux) since last month.
Pretty much coinsiding with CRUX adding PAM to core, and OpenBSD 7.5 and NetBSD 10.0 releasing almost at the same time.

Results in thinking that if free -m reports a certain RAM usage, everything else will report like so too.
It kinda works on OpenBSD too, except of course it doesn't come with free.
But FreeBSD and NetBSD seem to work differently in this aspect.

rvp

pin Yeah, that was the joke intended 😃. For reference, xterm uses between ~10 MB (only bitmap fonts) to ~25 MB of RSS (when showing colour emojis + TT fonts).

rvp

remiliascarlet Edit: nevermind, found it: https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/amd64/binary/sets/

Fine. Also extract the curses and term* libraries--just in case those're also somehow old or mismatched:

tar -C/ -xvf base.tar.xz ./lib/libterm* ./usr/lib/libterm* ./usr/lib/libcurses*

This, plus the updated terminfo database should solve 1).

For 2) and 3), download and extract the pkgsrc source tree (use the branch corresponding to the binaries you currently have), then recompile st-term with this line removed from the Makefile:

SUBST_SED.configdefh+=		-e 's,st-256color,xterm,g'

This hack is no longer needed (and is what messes up the HOME/END key recognition for oh-my-zsh) since both 9.X and 10.X have the definition for st-256color. (You could also simply compile st standalone to test this part.)

And, contact the maintainer to get rid of that hack.

remiliascarlet

rvp I did what you suggested, but I see 0 difference.
I actually manually compile my own st build, since I have applied some patches to it.
But I tried to install the one in pkgsrc, which I already pulled when I installed NetBSD, but this too didn't seem to make any difference.
The terminfo stuff still doesn't exist, neither does it in /etc.

However, what I did find is this:

ls -l /usr/pkg/share/terminfo/s/st*
-rw-r--r--  2 root  wheel  2639 Apr  8 22:10 /usr/pkg/share/terminfo/s/st
-rw-r--r--  1 root  wheel  2412 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.6
-rw-r--r--  1 root  wheel  2731 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.7
-rw-r--r--  1 root  wheel  2637 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.8
-rw-r--r--  2 root  wheel  2799 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-16color
-rw-r--r--  2 root  wheel  2729 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-256color
-rw-r--r--  1 root  wheel  2675 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-direct
-rw-r--r--  8 root  wheel   856 Apr  8 22:10 /usr/pkg/share/terminfo/s/st52
-rw-r--r--  4 root  wheel  1966 Apr  8 22:10 /usr/pkg/share/terminfo/s/st52-color
-rw-r--r--  8 root  wheel   856 Apr  8 22:10 /usr/pkg/share/terminfo/s/st52-m
-rw-r--r--  1 root  wheel   562 Apr  8 22:10 /usr/pkg/share/terminfo/s/st52-old
-rw-r--r--  2 root  wheel  2639 Apr  8 22:10 /usr/pkg/share/terminfo/s/stterm
-rw-r--r--  2 root  wheel  2799 Apr  8 22:10 /usr/pkg/share/terminfo/s/stterm-16color
-rw-r--r--  2 root  wheel  2729 Apr  8 22:10 /usr/pkg/share/terminfo/s/stterm-256color
-rw-r--r--  1 root  wheel   916 Apr  8 22:10 /usr/pkg/share/terminfo/s/stv52
-rw-r--r--  1 root  wheel   848 Apr  8 22:10 /usr/pkg/share/terminfo/s/stv52pc

Meaning the definitions do exist, just not in the place ZSH can recognize.
So I altered the .zshrc file a bit to make it look more like this:

if [ "$platform" = 'netbsd' ]; then
  export TERMINFO_DIRS="$XDG_DATA_HOME/terminfo:/usr/share/terminfo:/usr/pkg/terminfo"
else
  export TERMINFO_DIRS="$XDG_DATA_HOME/terminfo:/usr/share/terminfo"
fi

But this still didn't do the trick.
I also tried to copy the entire terminfo directory to ~/.local/share, and that too didn't work.
Even after all that, I'd still get "can't find terminal definition for st-256color".

The $platform variable is sourced from a different script on which I detect what OS and distro (on Linux only) I'm running, and $XDG_DATA_HOME is simply just ~/.local/share.

rvp

remiliascarlet I did what you suggested, but I see 0 difference.

Damn! I was so sure I had fixed this--however, ...

remiliascarlet I actually manually compile my own st build, since I have applied some patches to it.

That explains why TERM=st-256color. The pkgsrc st is forced to xterm which leads to the other problem (in prev. post).

remiliascarlet The terminfo stuff still doesn't exist, neither does it in /etc.

It shouldn't exist in /etc -- which is what I wanted to confirm.

remiliascarlet However, what I did find is this:

ls -l /usr/pkg/share/terminfo/s/st*
-rw-r--r--  2 root  wheel  2639 Apr  8 22:10 /usr/pkg/share/terminfo/s/st
-rw-r--r--  1 root  wheel  2412 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.6
-rw-r--r--  1 root  wheel  2731 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.7
-rw-r--r--  1 root  wheel  2637 Apr  8 22:10 /usr/pkg/share/terminfo/s/st-0.8
[...]

That's part of the ncurses(w) package. Try:

pkg_info -Fe /usr/pkg/share/terminfo/s/st-256color

However, zsh in pkgsrc uses NetBSD's own curses. Try:

ldd $(which zsh)

The layout and structure of the two databases are completely different.

remiliascarlet So I altered the .zshrc file a bit to make it look more like this:

if [ "$platform" = 'netbsd' ]; then
  export TERMINFO_DIRS="$XDG_DATA_HOME/terminfo:/usr/share/terminfo:/usr/pkg/terminfo"
else
  export TERMINFO_DIRS="$XDG_DATA_HOME/terminfo:/usr/share/terminfo"
fi

But this still didn't do the trick.

This! This is what's causing all your problems. Change that test to say [ $platform != netbsd ] and do nothiing for NetBSD and it should all be OK.

remiliascarlet Even after all that, I'd still get "can't find terminal definition for st-256color".

Self-inflicted wound. 😉

BTW, that TERMINFO_DIRS variable has a misleading name. You put the actual database filenames there (typically w/o the .cdb extension) instead of directories containing the DB files. Feel free to file a PR about this too.

remiliascarlet

rvp This! This is what's causing all your problems. Change that test to say [ $platform != netbsd ] and do nothiing for NetBSD and it should all be OK.

This right here is indeed what caused the problem!
Everything is fine now at last.