With GNU coreutils and FreeBSD I have become used to seeing the largest items in a directory with the command du -sh ./* | sort -h . In this post I'll show an alternative I have implemented for NetBSD, which I recently started using. Here is an example of the output of du -sh ./* | sort -h : dbohdan@debian ~/sort-h-test> du -sh ./* | sort -h 4.0K ./compbench 20K ./lsblk 11M ./hicolor 22M ./netbsd.bak 43M ./caddy.tgz 76M ./caddy 333M ./go if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } The NetBSD version of sort(1) does not implement -h ( FreeBSD man page for comparison) to sort numbers with unit prefixes. It made me look for a workaround. I did not want to implement my own sort(1), so instead I chose to "humanize" already sorted du output. For this I wrote a script I called humsize . Its output looks like this: dbohdan@netbsd ~/sort-h-test> du -sk ./* | sort -n | humsize 2.0K ./compbench 18.0K ./lsblk 10.4M ./hicolor 21.5M ./netbsd.bak 42.2M ./caddy.tgz 74.9M ./caddy 322.0M ./go if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Here is the source code of the script (with a permissive license): #! /usr/bin/awk -f # humsize -- humanize du(1) output. This is useful for systems without # `sort -h`. Use the output of `du -k`. For example, # $ du -sk ./* | sort -n | humsize # # https://opensource.org/licenses/Fair # # Copyright 2023 D. Bohdan. # # Usage of the works is permitted provided that this instrument is # retained with the works, so that any entity that uses the works is # notified of this instrument. # # DISCLAIMER: THE WORKS ARE WITHOUT WARRANTY. BEGIN { # Options. Set with `humsize -v varname=value`. if (f == "") { # What field to humanize. f = 1 } if (format == "") { # The format string: first the size, then the unit prefix. format = "%6.1f%s" } u_last = 0 u[++u_last] = "K" u[++u_last] = "M" u[++u_last] = "G" u[++u_last] = "T" } # Pass through headers and such. $f + 0 != $f { print } $f + 0 == $f { size = $f i = 1 while (size > 1024 && i < u_last) { size /= 1024 i++ } $f = sprintf(format, size, u[i]) print } if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); }

rvp The problem with this approach is that fields may have the same value, or one may be a substring of another. > cat test foo foo bar > ./subst.awk 1 new < test new foo bar > ./subst.awk 2 new < test new foo bar if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } You could check if the substitution was ambiguous and return an error value. Edit: Rephrased the last sentence to remove a double negative.

`du -h | sort -h` alternative for NetBSD

dbohdan

With GNU coreutils and FreeBSD I have become used to seeing the largest items in a directory with the command du -sh ./* | sort -h. In this post I'll show an alternative I have implemented for NetBSD, which I recently started using.

Here is an example of the output of du -sh ./* | sort -h:

dbohdan@debian ~/sort-h-test> du -sh ./* | sort -h
4.0K    ./compbench
20K     ./lsblk
11M     ./hicolor
22M     ./netbsd.bak
43M     ./caddy.tgz
76M     ./caddy
333M    ./go

The NetBSD version of sort(1) does not implement -h (FreeBSD man page for comparison) to sort numbers with unit prefixes. It made me look for a workaround. I did not want to implement my own sort(1), so instead I chose to "humanize" already sorted du output. For this I wrote a script I called humsize. Its output looks like this:

dbohdan@netbsd ~/sort-h-test> du -sk ./* | sort -n | humsize
   2.0K ./compbench
  18.0K ./lsblk
  10.4M ./hicolor
  21.5M ./netbsd.bak
  42.2M ./caddy.tgz
  74.9M ./caddy
 322.0M ./go

Here is the source code of the script (with a permissive license):

#! /usr/bin/awk -f
# humsize -- humanize du(1) output. This is useful for systems without
# `sort -h`. Use the output of `du -k`. For example,
# $ du -sk ./* | sort -n | humsize
#
# https://opensource.org/licenses/Fair
#
# Copyright 2023 D. Bohdan.
#
# Usage of the works is permitted provided that this instrument is
# retained with the works, so that any entity that uses the works is
# notified of this instrument.
#
# DISCLAIMER: THE WORKS ARE WITHOUT WARRANTY.

BEGIN {
  # Options. Set with `humsize -v varname=value`.
  if (f == "") {
    # What field to humanize.
    f = 1
  }
  if (format == "") {
    # The format string: first the size, then the unit prefix.
    format = "%6.1f%s"
  }

  u_last = 0
  u[++u_last] = "K"
  u[++u_last] = "M"
  u[++u_last] = "G"
  u[++u_last] = "T"
}

# Pass through headers and such.
$f + 0 != $f {
  print
}

$f + 0 == $f {
  size = $f
  i = 1
  while (size > 1024 && i < u_last) {
    size /= 1024
    i++
  }

  $f = sprintf(format, size, u[i])
  print
}

rvp

dbohdan Heh. Looks like we wrote similar code. (wrote this sometime back):

#!/usr/bin/awk -f
#
# Humanize the 1st field (KB assumed) in the output of commands like
# du(1) or sort(1)--if they don't provide a `-h' (humanize) flag.
#
# Eg: du ~ | sort -n -k1,1 | hum.awk
BEGIN {
	K = 1024
	N = split("KMGTPEZY", SUF, "")
}
function H(num,	i) {		# num (x KB)
	for (i = 1; i < N && num >= K; i++)
		num /= K
	return sprintf("%.1f%c", num, SUF[i])
}
$1 ~ /^[[:digit:]]+$/ {		# num something...
	printf "%6s%s\n", H($1), substr($0, length($1)+1)
	next
}
{ print }			# something...

dbohdan

rvp

Wow, this is very similar code, down to the comments. :-D I like your use of split; it is shorter and less error-prone than a[++index] array initialization. I am going to borrow it. :-) I should also use next to avoid negating the pattern.

#! /usr/bin/awk -f
# humsize -- humanize du(1) output. This is useful for systems without
# `sort -h`. Use the output of `du -k`. For example,
# $ du -sk ./* | sort -n | humsize
#
# https://opensource.org/licenses/Fair
#
# Copyright 2023 D. Bohdan.
#
# Usage of the works is permitted provided that this instrument is
# retained with the works, so that any entity that uses the works is
# notified of this instrument.
#
# DISCLAIMER: THE WORKS ARE WITHOUT WARRANTY.

BEGIN {
  # Options. Set with `humsize -v varname=value`.
  if (f == "") {
    # What field to humanize.
    f = 1
  }
  if (format == "") {
    # The format string: the size, then the unit prefix.
    format = "%6.1f%s"
  }

  u_last = split("K M G T", u, " ");
}

# Pass through headers and such.
$f + 0 != $f {
  print
  next
}

# $f is a number.
{
  size = $f
  i = 1
  while (size > 1024 && i < u_last) {
    size /= 1024
    i++
  }

  $f = sprintf(format, size, u[i])
  print
}

Since at least two people independently wrote them, there should probably be a wiki page with our scripts.

rvp

dbohdan I like your use of split [...]

And, I like your use of $f + 0 == $f to match numbers (which works for floats too). It's a trick I keep seeing and then forgetting...

(awkward silence...)
(loud, forced male laughter)

dbohdan there should probably be a wiki page with our scripts.

Go ahead. As far as I'm concerned, all the stuff I write for the board is public-domain.

BTW, assigning back to a field like this: $f = sprintf(format, size, u[i]) causes field-splitting--which'll mess up filenames if they contain multiple adjacent blanks.

dbohdan

rvp

rvp Go ahead. As far as I'm concerned, all the stuff I write for the board is public-domain.

Yay. A wiki page is coming soon. Meanwhile, I have created a Git repo for my script, with tests and CI. I have also fixed a bug and implemented two new options: units (e.g., B,KiB,MiB) and zero (to give size zero a special representation if you want to, for example, avoid "0.0 bytes").

rvp BTW, assigning back to a field like this: $f = sprintf(format, size, u[i]) causes field-splitting--which'll mess up filenames if they contain multiple adjacent blanks.

Thanks! I didn't consider this. I've added a comment warning the user the script can mess up whitespace.

I'd call it more "field-joining" than "field-splitting" 🙂, since asigning to a field causes $0 to be reconstructed from $1–$NF joined together with OFS.

I could avoid this problem if I only humanized the first field, but I like the feature that lets you select which field to humanize. To preserve the whitespace with a user-selectable field, I'd have to do my own AWK-style parsing manually—in AWK, which would be a hassle. I could use split with seps if I were targeting Gawk, but I am not. I probably won't fix it. The user shouldn't try to parse the output of du anyway, since it doesn't handle newlines in filenames.

(If someone wants one, I have implemented an AWK-style field parser that preserves the separators between the fields for future use, but in Tcl, where it is more convenient. It is part of a "what if awk(1) but SQL?" thing I wrote.)

rvp

dbohdan To preserve the whitespace with a user-selectable field, I'd have to do my own AWK-style parsing manually—in AWK, which would be a hassle.

How 'bout something like this:

subst.awk

#!/usr/bin/awk -f
#
# Usage: subst.awk fldno repl-str
# Eg.:   subst.awk 3 hello	# replace 3rd field with "hello"

# Replace `t' at position `pos' in `s' with `r'.
#
function repl(s, t, pos, r,	n, s1, s2) {
	if (!t || pos < 1)
		return s
	s1 = substr(s, 1, pos - 1)
	s2 = substr(s, pos + length(t))
	return s1 r s2
}

# Get the position where field `fno' starts in string `s'.
#
function getpos(s, fno,		i, l, n, N) {
	if (fno == 1)
		return index(s, $fno)
	for (i = 1; i <= fno; i++) {
		l = (i < fno) ? length($i)-1 : 0
		n = index(s, $i) + l
		s = substr(s, n + 1)
		N += n
	}
	return N
}

BEGIN {
	ME = "subst.awk"
	if (ARGC != 3) {
		print "Usage: " ME " fldno repl-str" > "/dev/stderr"
		exit 1
	}
	fno = ARGV[1]; ARGV[1] = ""
	str = ARGV[2]; ARGV[2] = ""
}
{
	pos = getpos($0, fno)
	print repl($0, $fno, pos, str)
}

dbohdan

rvp

The problem with this approach is that fields may have the same value, or one may be a substring of another.

> cat test
    foo     foo  bar
> ./subst.awk 1 new < test
    new     foo  bar
> ./subst.awk 2 new < test
    new     foo  bar

You could check if the substitution was ambiguous and return an error value.

Edit: Rephrased the last sentence to remove a double negative.

dbohdan

rvp

Now humsize updates the field f without affecting whitespace using a function based on yours. Thanks! I have credited you in a comment. Please tell me if you want to be credited differently.

dbohdan

Ta-da. The page is up.

rvp

dbohdan The problem with this approach is that fields may have the same value, or one may be a substring of another.

Yeah, just discovered that... 🙁

Should be able to disambiguate if we used the rest of the string also, I think. Hmm.

EDIT: OK, script updated. Should be fixed now.

dbohdan

rvp

The new version should be good for most uses of awk. It works if a field can't be a substring of a string literal field separator or a substring of a match in the case of a regular expression separator. If a field is such a substring, it can misbehave. This not an issue with the default separator. Maybe I should implement something like this in my script.

Here is an example of it misbehaving. Suppose we set FS to <separator>. Then we can have

> cat test
c<separator>d<separator>e
> ./subst.awk 1 new < test
new<separator>d<separator>e
> ./subst.awk 2 new < test
c<separator>new<separator>e
> ./subst.awk 3 new < test
c<separator>d<snewparator>e

rvp

dbohdan It works if a field can't be a substring of a string literal field separator or a substring of a match in the case of a regular expression separator. If a field is such a substring, it can misbehave.

Yeah. Fixing this to handle all types of FS would be tricky. For instance, the default value of FS is a single SPACE, but, all awks treat that as SP, TAB, NL. But, this is not the same as setting FS=[[:space:]]+ (as one would intuitively expect):

$ echo $'\t a \t b \t c ' | awk '{ print $3 }'
c
$ echo $'\t a \t b \t c ' | awk -F' ' '{ print $3 }'              # leading spaces not significant
c
$ echo $'\t a \t b \t c ' | awk -F'[[:space:]]+' '{ print $3 }'   # _now_ thet are!
b
$

Then there is the FS="" case where each char. becomes a field...

dbohdan I have credited you in a comment. Please tell me if you want to be credited differently.

Thanks for that, but, no credit is needed. Really! (It's all PD-stuff as I said.)