Moritz Systems | BSD Tips and Tricks

kamil

Please share your BSD Tips and Tricks and we can publish a selection of them on the aggregator page:

https://www.moritz.systems/bsd-tips-and-tricks/

Enjoy!

rvp

EDIT: removed stray ; in the c function.

kamil 3 selections:

Remove duplicate lines in files (eg. rc.conf files, shell history files):

$ cat ~/bin/nuniq.sh
#!/bin/sh
#
# nuniq.sh: Remove duplicate lines, leaving only the last one.
# (based on histsort.awk from "GAWK: Effective AWK Programming")

exec awk '
{
	lines[NR] = $0
	if ($0 in array)
		delete lines[array[$0]]
	array[$0] = NR
}
END {
	for (i = 1; i <= NR; i++)
		if (i in lines)
			print lines[i]
}' "$@"

Show diff output/patch files in colour:

$ cat ~/bin/cpatch.sh
#!/bin/sh
#
# Colour unified diffs.

org=$(tput sgr0)		# Reset attrs.
old=$(tput setf 4 bold)		# Bold Red
new=$(tput setf 6 bold)		# Bold Yellow

exec sed -e "s/^+.*$/${new}&${org}/; s/^-.*$/${old}&${org}/" -- "$@"

Use as: diff -urN DIR1 DIR2 | cpatch.sh OR cpatch.sh a.patch b.patch ...

Emulate clear in ~~restricted~~ constrained environments which don't have a clear and the associated termcap / terminfo database and library--eg. an OpenBSD USB install stick 😠:

c() {
	printf '\033[H\033[J'
}

Works on all ANSI-/VT100-compatible terminals.

kamil

"Find the largest files in the current directory" by Kamil Rytarowski

find . -type f -exec stat -f "%z %N" {} \; | sort -r -n -k1

https://www.moritz.systems/bsd-tips-and-tricks/find-largest-files-in-the-current-directory/

#FreeBSD #NetBSD #OpenBSD

@kamil ...and with some eye-candy 😉

I'm really liking this broot tool.

kamil

rvp Is this form OK: alias clr='printf "\033[H\033[J"' ?

I needed to delete the semicolon character as otherwise it was printed in the termnial (NetBSD, ksh).

kamil

To reduce the cycle of asking people for permission. Pasting here the tips and tricks means that I can publish them on the website freely, without asking and with the credit for submitter.

qik

A tip which is trivial, but overlooked: Use -C to specify the directory in which make should run, as in:

make -C /usr/pkgsrc/devel/git-base install clean clean-depends

rvp

qik A tip which is trivial, but overlooked: Use -C to specify the directory in which make should run

Yes, just like patch -d /some/dir ...

yeti

MVMPIYTGA!

Make viewing man pages in your terminal great again!

alias man='w3mman'

rvp

kamil I needed to delete the semicolon character

I swear that wasn't there when I typed it in. It's these semicolons I tell you--they sneak right in somehow. Thank you for eradicating the beast.

I made the clear work-alike a function because a) I prefer functions, and b) some shells don't have an alias command. However, alias is in POSIX, so go right ahead with them.

As for permissions, etc: All code/scripts I write and post on this board can be considered to be in the public-domain. Use as you wish. Code & scripts I did not write, I will try to attribute correctly, and for these, you'll have to use your own judgement.

Some sed(1) one-liners:

The sed(1) "list" command is very useful when dealing with unprintable (and invisible!) chars. I use it all the time because the other methods of dealing with these pesky characters is not very convenient: hexdump(1) is great, but, it doesn't preserve line endings; and vis(1) is a BSD-only thing.

$ printf 'MS-DOS line ending\r\n' | sed -n l 
MS-DOS line ending\r$
$ printf 'Non-breaking space: '$'\u00A0''\n' | LC_ALL=C sed -n l
Non-breaking space: \302\240$

Insert thousands-separators into numbers. The -h option to commands like du(1), df(1), etc. are nice and all, but, sometimes I want to see whole numbers in outputs. Then separators are useful to make sense of large numbers.

#!/usr/bin/sed -Ef
#
# Insert US-style thousands-separators (commas) into numbers.

: loop
s/(^|[[:blank:]]+)([0-9]{1,})([0-9]{3})([0-9,]*)([[:blank:]]+|$)/\1\2,\3\4\5/
t loop

Use as: du -a /some/dir | comma, or netstat -b -w 1 -I alc0 | comma, etc.

Number lines in file.

#!/bin/sh
#
# Simulate nl(1) or cat(1) -n

exec sed -e = -- ${1+"$1"} | sed -nEe 'N;s/([[:digit:]]+)./\1'$'\t''/;p'

NetBSD doesn't come with a tree(1) command so here's one in sed(1):

#!/bin/sh
#
# Print directory tree

exec find -- ${1:-.} | sed -Ee '2,$s:[^/]*/:   :g; s:   ([^ ]+)$:|_ \1:'

Note: it doesn't draw a fully-connected tree. Drawing a full tree is very hairy--and, completely out of our self-imposed one-liners category.

Reverse-print a file--aka tac(1):

#!/bin/sh
#
# Implement tac(1) using sed(1).

exec sed -ne '1h; 2,${x;H;}; ${x;p;}' -- "$@"

Same thing in ed(1); using the standard idiom:

#!/bin/sh
#
# Implement tac(1) using ed(1).

: ${1?"Usage: ${0##*/} FILE"}

exec ed -s -- "$1" <<\EoF
g/^.*$/m0
%p
EoF

kamil

rvp

NetBSD doesn't come with a tree(1) command so here's one in sed(1):

How about ls -R ?

kamil

Emulate clear in restricted environments

alias clr='printf "\033[H\033[J"'

Works with VT100-compatible terminals.

https://www.moritz.systems/bsd-tips-and-tricks/emulate-clear-in-restricted-environments/

#FreeBSD #NetBSD #OpenBSD

rvp

kamil I remember that char. order (H,J) using the mnemonic HiJack (or, HiJinks--more apt). Every programmer knows \033 is ESC, and \033[ is the standard ANSI CSI.

I see you're only taking the simple stuff. OK, then, here's another golden oldie:

How to reset a messed up terminal (eg. after you accidentally cat a binary file and the terminal goes completely wonky. You may have to type this in blind if the terminal is severely messed up.):

Ctrl-J stty sane Ctrl-J

Ie. Press Ctrl-J instead of ENTER, since the binary garbage could've changed the default line-ending char. too. After that, you can do reset; clear.

Edit: I would change "restricted" to "constrained" in the heading.
My English, you see, sometimes she not so good.

kamil

rvp

I see you're only taking the simple stuff.

Not really. I just take them one at a time! I plan to publish one entry weekly.

I would change "restricted" to "constrained" in the heading.

It's too late!

Thank you for your entries!

rvp

kamil How about ls -R ?

That's no tree. This is a tree (written long long ago):

tree.c

/**
 * Print a directory `tree'.
 */
#include <sys/types.h>
#include <sys/stat.h>

#include <dirent.h>
#include <errno.h>
#include <limits.h>			/* for PATH_MAX */
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>			/* for getopt() */

/* Declarations */
struct entinfo {			/* describes a directory entry */
	struct dirent dent;		/* for entry name and type */
	int islast;			/* last entry in a directory if 1 */
};

/* Functions */
static char* concat(const char*, const char*);
static int dircmp(const void*, const void*);
static int do_opts(const int, char*[]);
static int isdir(const struct entinfo*);
static int isdot(const char*);
static int walk(const char*, const int, int*);
static size_t getents(const char*, struct entinfo***);
static struct entinfo* direntcpy(const struct dirent*);
static void error(const char*, ...);
static void inittchars(void);
static void freeents(struct entinfo**, const size_t);
static void prtree(const struct entinfo*, const int, const int*);
static void setlastdent(struct entinfo**, const size_t);
static void setlastfent(struct entinfo**, const size_t);
static void usage(void);
static void* Realloc(void*, const size_t);

/* Globals */
static char* prog;			/* program name */
static struct options {			/* program options */
	int allents;			/* show files in addition to dirs. if 1 */
	int nosort;			/* do not sort directory entries if 1 */
	int vtgraph;			/* use VT100 graph. chars. for tree if 1 */
} opts;
static const char dirsep = '/';		/* directory separator character */
static const char* tree_vbar;		/* string/esc. sequence for vertical bar */
static const char* tree_norm;		/* string/esc. sequence for normal entries */
static const char* tree_last;		/* string/esc. sequence for last entry in dir. */


/**
 * M A I N
 */
int
main(int argc, char* argv[])
{
	char* path;
	int branch[PATH_MAX];		/* print vert. bar in level if 1 */
	int argidx;

	prog = argv[0];
	memset(branch, 0, sizeof branch);

	argidx = do_opts(argc, argv);
	argc -= argidx;
	argv += argidx;

	path = (argc == 0 || **argv == '\0') ? "." : *argv;

	inittchars();			/* setup strings used for tree printing */
	printf("%s:\n", path);
	return walk(path, 0, branch);
}

/**
 * Setup the strings used for tree printing.
 */
static void
inittchars(void)
{
	if (opts.vtgraph == 1) {	/* use VT100 graph. chars. for tree */
		tree_vbar = "\033(0x\033(B";
		tree_norm = "\033(0tqq\033(B";
		tree_last = "\033(0mqq\033(B";
	} else {			/* use plain ASCII chars. for tree */
		tree_vbar = "|";
		tree_norm = "|--";
		tree_last = "`--";
	}
}

/**
 * Recursively walk a directory tree starting at path.
 * `level' is the current depth in the tree.
 */
static int
walk(const char* path, const int level, int* branch)
{
	struct entinfo** eip;		/* file/dir entries in the current directory */
	size_t neip;			/* no. of entries in array `eip' */
	size_t i;			/* current directory entry number */

	/*
	 * Get list of entries in specified directory.
	 */
	neip = getents(path, &eip);
	if ((ssize_t)neip == -1)	/* error */
		return EXIT_FAILURE;

	/*
	 * Sort the entries in lexicographic order.
	 */
	if (opts.nosort != 1)
		qsort(eip, neip, sizeof(struct entinfo* ), dircmp);

	/*
	 * Mark the last entry or the last directory entry
	 * in the array after sorting.
	 */
	if (opts.allents == 1)
		setlastfent(eip, neip);
	else
		setlastdent(eip, neip);

	for (i = 0; i < neip; i++) {
		/*
		 * Check if entry is a directory and also not a symlink.
		 */
		if (isdir(eip[i]) == 1) {
			char* newpath;
			/*
			 * If this is the last entry in a directory,
			 * then no more branches are possible. So we
			 * don't print any branch chars for this level.
			 */
			if (eip[i]->islast == 1)
				branch[level] = 0;	/* no branch */
			else
				branch[level] = 1;	/* print branch */

			prtree(eip[i], level, branch);
			/* recurse */
			newpath = concat(path, eip[i]->dent.d_name);
			walk(newpath, level + 1, branch);
			free(newpath);
		}
		/*
		 * Entry is a file. Files don't change the
		 * structure of a directory, so there is no
		 * change in printing of branches.
		 */
		else if (opts.allents == 1)
			prtree(eip[i], level, branch);
	}

	freeents(eip, neip);		/* free entries and array */
	return EXIT_SUCCESS;
}

/**
 * Construct an array containing the entries in a directory.
 * Store array and return the no. of entries read.
 */
static size_t
getents(const char* path, struct entinfo*** eip)
{
	DIR* dir;			/* current directory */
	struct entinfo** ents;		/* entries in the current directory */
	struct dirent* dent;		/* directory entry */
	size_t entsz;			/* size of array `ents' */
	size_t nent;			/* current directory entry number */

	dir = opendir(path);
	if (dir == NULL) {
		error("%s: %s: can't read directory.", prog, path);
		*eip = NULL;		/* nothing read */
		return (ssize_t)-1;	/* error */
	}

	nent = entsz = 0;
	ents = NULL;
	while ((dent = readdir(dir)) != NULL) {
		/*
		 * Grow array if needed.
		 */
		if (nent == entsz) {
			entsz = (entsz == 0) ? 64 : entsz * 2;
			ents = Realloc(ents, entsz * sizeof(struct entinfo*));
		}
		/*
		 * Store entries in array except `.' or `..'.
		 */
		if (isdot(dent->d_name) == 0) {
			struct entinfo* e;
			if ((e = direntcpy(dent)) != NULL)
				ents[nent++] = e;
		}
	}

	closedir(dir);
	*eip = ents;			/* store pointer to entries in current dir. */
	return nent;			/* return no. of entries in current dir. */
}

/**
 * Mark the last entry in the array `eip'.
 */
static void
setlastfent(struct entinfo** eip, const size_t neip)
{
	if (neip > 0)
		eip[neip - 1]->islast = 1;
}

/**
 * Mark the last _directory_ entry in the array `eip'.
 */
static void
setlastdent(struct entinfo** eip, const size_t neip)
{
	if (neip > 0) {
		size_t i, lastdir;	/* tmp, index to last directory entry */

		lastdir = 0;
		for (i = 0; i < neip; i++)
			if (isdir(eip[i]) == 1)
				lastdir = i;
		eip[lastdir]->islast = 1;
	}
}

/**
 * Print tree branches for directory entry `eip'
 * at depth `level'.
 *
 * The last entry in a directory is printed differently.
 */
static void
prtree(const struct entinfo* eip, const int level, const int* branch)
{
	int i;

	printf("\t");
	for (i = 0; i < level; i++)
		if (branch[i] == 1)
			printf("%s   ", tree_vbar);
		else
			printf("    ");
	printf("%s ", eip->islast ? tree_last : tree_norm);
	printf("%s\n", eip->dent.d_name);
}

/**
 * Append a directory entry to a path and return
 * new allocated pathname.
 */
static char*
concat(const char* path, const char* ent)
{
	char* newpath;

	newpath = Realloc(NULL, strlen(path) + strlen(ent) + 2);
	if (path[strlen(path) - 1] != dirsep)	/* don't add extra slashes */
		sprintf(newpath, "%s%c%s", path, dirsep, ent);
	else
		sprintf(newpath, "%s%s", path, ent);

	return newpath;
}

/**
 * Make a copy of dirent and stat structures associated with
 * a file.
 */
static struct entinfo*
direntcpy(const struct dirent* dentp)
{
	struct entinfo* eip;
	
	eip = Realloc(NULL, sizeof (struct entinfo));
	memset(eip, 0, sizeof (struct entinfo));

#define FIX_CLANG_ADDRESS_SANITIZER_ERROR 1
#if FIX_CLANG_ADDRESS_SANITIZER_ERROR
	strcpy(eip->dent.d_name, dentp->d_name);
	eip->dent.d_type = dentp->d_type;
#else
	eip->dent = *dentp;
	//	OR
	// memcpy(&eip->dent, dentp, sizeof (struct dirent));
#endif
	if (eip->dent.d_type == DT_UNKNOWN) {
		struct stat st;
		if (lstat(eip->dent.d_name, &st) != -1) {
			switch (st.st_mode & S_IFMT) {
			case S_IFDIR:	eip->dent.d_type = DT_DIR; break;
			case S_IFLNK:	eip->dent.d_type = DT_LNK; break;
			}
		}
	}
	eip->islast = 0;

	return eip;
}

/**
 * Check if arg is `.' or `..'.
 */
static int
isdot(const char* name)
{
	return !strcmp(name, ".") || !strcmp(name, "..");
}

/**
 * Check if entry is a directory and also not a symlink.
 */
static int
isdir(const struct entinfo* eip)
{
	return eip->dent.d_type==DT_DIR && eip->dent.d_type!=DT_LNK;
}

/**
 * Check if 2 directory entries are the same.
 */
static int
dircmp(const void* ent1, const void* ent2)
{
	struct entinfo* eip1;
	struct entinfo* eip2;

	eip1 = *(struct entinfo**)ent1;
	eip2 = *(struct entinfo**)ent2;

	return strcmp(eip1->dent.d_name, eip2->dent.d_name);
}

/**
 * Free the elements in a directory entry array
 * and the array.
 */
static void
freeents(struct entinfo** eip, const size_t neip)
{
	size_t i;

	if (eip != NULL) {
		for (i = 0; i < neip; i++)
			if (eip[i] != NULL)
				free(eip[i]);
		free(eip);
	}
}

/**
 * Process program options.
 */
static int
do_opts(int argc, char* argv[])
{
	int opt;

	/* defaults */
	opts.allents = 0;		/* don't show files */
	opts.nosort  = 0;		/* sort directory entries */
	opts.vtgraph = 0;		/* use plain ASCII chars for tree */

	while ((opt = getopt(argc, argv, "afg")) != -1)
		switch (opt) {
		case 'a':		/* show files too */
			opts.allents = 1;
			break;
		case 'f':		/* don't sort directory entries */
			opts.nosort = 1;
			break;
		case 'g':		/* use VT100 graphic chars for tree */
			opts.vtgraph = 1;
			break;
		default:
			usage();
			exit(EXIT_FAILURE);
		}

	return optind;
}

/**
 * realloc wrapper.
 */
static void*
Realloc(void* ptr, const size_t size)
{
	void* p;

	p = realloc(ptr, size);
	if (p == NULL) {
		error("%s: realloc() failed.", prog);
		exit(EXIT_FAILURE);
	}

	return p;
}

/**
 * Print usage information.
 */
static void
usage(void)
{
	fprintf(stderr, "Usage: %s [OPTION] [DIRECTORY]\n", prog);
	fprintf(stderr, "%s: Print a tree like directory listing\n", prog);
	fprintf(stderr, "\n");
	fprintf(stderr, "  -a    Print all entries (default: dirs. only)\n");
	fprintf(stderr, "  -f    Do not sort entries\n");
	fprintf(stderr, "  -g    Use VT100 graphic chars to print tree\n");
}

/**
 * Print an error message.
 */
static void
error(const char* fmt, ...)
{
	va_list ap;

	fflush(stdout);

	va_start(ap, fmt);
	vfprintf(stderr, fmt, ap);
	va_end(ap);

	if (errno != 0) {
		fprintf(stderr, " %s", strerror(errno));
		errno = 0;
	}
	fprintf(stderr, "\n");
	fflush(stderr);
}

kamil

rvp What is the source of this code? It looks impressive. (Same question regarding the license.)

Today, I think I would go for ftw(3) or fts(3), unless I would target non-unix or some restricted very portable codebase.

rvp

kamil What is the source of this code?

Expository code I wrote for junior colleagues 15+ years ago. Which is why it's so heavily commented. Licence: again, I place it in the public-domain. Incidentally, after I posted the code, I discovered a "deluxe" tree-printer in pkgs.

kamil Today, I think I would go for ftw(3) or fts(3), unless I would target non-unix or some restricted very portable codebase.

So would I, but portability was the main concern here: at the time I wrote it, glibc didn't have fts/ftw.

kamil

Show diff output in color

https://www.moritz.systems/bsd-tips-and-tricks/show-diff-output-in-color/

alias cpatch='sed -e "s/^+.*$/`tput setf 6 bold`&`tput sgr0`/" \
                  -e "s/^-.*$/`tput setf 4 bold`&`tput sgr0`/"'

Explanation

tput sgr0

Turn off all attributes.

tput setf 4 bold

Set foreground color to Bold Red.

tput setf 6 bold

Set foreground color to Bold Yellow.

Check the documentation for tput(1) and terminfo(5) for more terminal capability definitions.

Example output

rvp

kamil Bold Olive

Err. I thought Olive was a shade of green (and it is Cthulhu's favourite colour, I think.). That looks distinctly yellow to me.

Some more before I lose my DSL and go off-line (moving to new flat):

Finding files with a set of names you want is easy: find PATH \( -name '*.c' -o -name '*.h' \), but recently, I had to find some filenames which began with mod_ and were not either .c or .obj files. Here's how I did it:

# Find files with a common name, excluding those with a specific
# extension. For example, find all mod_* files which don't end
# in *.c or *.obj

$ find /usr/src/ -iname mod_* -a \! \( -iname *.c -o -iname *.obj \)
...output elided...
$

The -iname option is for a case-insensitive pattern match.

Read single characters in a POSIX shell:

$ cat ~/bin/bl.sh
#!/bin/sh
#
# Set LCD backlight on *BSD.

RC=1			# default is failure.
trap 'stty -cbreak echo; exit $RC' exit hup int quit term
stty cbreak -echo
while :
do	c=$(dd bs=1 count=1 status=none)
	case $c in
	\<|,|\-)	intel_backlight decr
		;;
	\>|.|\+)	intel_backlight incr
		;;
	q)	RC=0	# exit success
		break
		;;
	esac
done

bash(1) has a read -n 1 -s which does the same thing.

Change directories conveniently using abbreviations:

#
# To use this function, source this file.
#
# Create test dirs.:
# for n in {100..150}; do mkdir " "$'\n'"$n"$'\n'"\"d\\'\\ir\""; done

# Bourne-shell string quoting
# Turns:
#	hello world   -> 'hello world'
#	hello 'world' -> 'hello '\''world'\''
#
q ()
{
	printf '%s' "$d" | sed -Ee "s/'/'\\\\''/g; 1s/^/'/; \$s/\$/'/"
}

ch ()
{
	local arg dirs d i

	# short-circuit
	if [ $# -eq 0 ]
	then	cd
		return
	fi

	arg="$1"
	for d in $(printf '%s' "$arg" | sed -Ee 's/[^/]+/&*/g')
	do	test -d "$d" || continue
		dirs="$dirs $(q $d)"
	done

	eval "set -- $dirs"

	# no match
	if [ $# -eq 0 ]
	then	printf 1>&2 '%s: %s: no match\n' "$0" "$arg"
		return 1
	fi

	# single match
	if [ $# -eq 1 ]
	then	cd "$1"
		return
	fi

	# multiple matches
	i=0
	for d
	do	i=$((i + 1))
		printf '%d) %s\n' "$i" "$d"
	done
	printf '%s\n? ' '-----'
	read num

	# do nothing if no number
	if [ "$num" == "" ]
	then	return 0
	fi

	# invalid number
	if [ "$num" -lt 1 -o "$num" -gt $# ]
	then	printf 1>&2 '%s: %s: bad number\n' "$0" "$num"
		return 1
	fi

	eval "cd \"\${$num}\""
}

Use as follows:

work$ source ~/bin/chdir.sh
work$ ch /u/l/b
1) /usr/libexec/bsdconfig
2) /usr/libexec/bsdinstall
3) /usr/local/bin
-----
? 3
/usr/local/bin$

Use more chars. to disambiguate and avoid the prompt. The ch function should work even if the directory name contains blanks and new-lines. Report bugs if you find them.

kamil

rvp I used the terminology from aha

Terminal yellow is painted as olive.

https://www.computerhope.com/cgi-bin/htmlcolor.pl?c=808000

rvp

kamil I used the terminology from aha

Terminal yellow is painted as olive.

I like his code, but, I wouldn't try his martinis--he obviously likes strange garnishes in his.