leah blogs: July 2013

For the final installment of my “Summer of Scripts”, I’m showing a generic tool I wrote and a specialized version of it useful to Arch Linux users.

tarhash computes checksums for files inside tarballs without unpacking them.

% tarhash ~/src/mutt-1.5.21.tar.gz | head -3
9cc2ec57dc43e6768516898ebb90f3d76cb24d72  ./mutt-1.5.21/ABOUT-NLS
a87360b6b5b8d6d2cdeb83d54b3aa4a0a35bf090  ./mutt-1.5.21/BEWARE
5d1b9cfe259891e3408938afa6bdd3821953973f  ./mutt-1.5.21/COPYRIGHT

It defaults to SHA1, but you can specify other hashes easily:

% tarhash --sha512 ~/src/mutt-1.5.21.tar.gz | head -3
808297837049d5b84b54ba780f87f08c22fb83ebbc62edaf3085966428593e76d28a7bf08cc4f029ee24a3a455fa292aac064b01ab8700240cb9ab0cc0284fae  ./mutt-1.5.21/ABOUT-NLS
b0ac0f3c9297c0bf26c20ce58bf7bb234bd2ab84e5ee545345f39142e83f4d93ca1eaf406d77fb8ffab8ac748bb25ea8891412f6dc3d0058db73de73442b38eb  ./mutt-1.5.21/BEWARE
68c306e6fc7a0b9a1dc47bbc700f034bc40c6c4e2125c35ce24deba44a95eb8113ce8dbd81a9fd9ad7208d28108e36a8f6bb078de416e42d7ad46271b13cca77  ./mutt-1.5.21/COPYRIGHT

Also, since it uses the powerful bsdtar of libarchive, it supports other archive formats as well:

% tarhash /usr/lib/python3.3/test/zipdir.zip
da39a3ee5e6b4b0d3255bfef95601890afd80709  ./a/b/c

Since the hashes of tarballs themselves easily can change (due to changed metadata, different order of files, etc…), this tool is nice to compare tarballs contentwise.

However, tarhash actually is a by-product of pacverify, which tries to find files that have changed in your Arch Linux installation, compared to the original packages. Simply run it and after some time you’ll see output like:

cpupower 3.10-1: /./etc/default/cpupower: FAILED
cpupower 3.10-1: sha256sum: WARNING: 1 computed checksum did NOT match

Of course, it’s ok that some config files have been changed, but that is your job to decide.

pacverify is also good if you think some (possibly undetected?) filesystem corruption took place (or someone fiddled in your files, but be sure to compare against verified package files them).

That’s it for the summer. I hope you had fun and learned something. :)

NP: Toxoplasma—Alte Zeichen

ssh-chain wasn’t written by me, but it’s so incredibly useful and I use it a lot, which makes it a perfect fit to be listed here.

Often (admittedly, with the advent of IPv6 less often) you want to ssh to a machine you can’t connect to directly, and have to “hop” over a proxy host. Many people configure such hops statically in their .ssh/config, for example (the latter variant works with modern versions of OpenSSH only):

Host office
  ProxyCommand ssh -qAx remote.example.org nc -q5 workstation.example.org 22

Host office2
  ProxyCommand ssh -W workstation.example.org:22 remote.example.org

However, these configurations get nasty quickly if you need multiple hops or want to use different hops depending on other things.

ssh-chain is a simple Perl script you put into the path of your remote machines, and add the following line to .ssh/config:

Host *^*
  ProxyCommand ssh-chain %h %p

Now, you can use that host to hop to other machines over SSH, by just giving a “path” (think UUCP ;)) of immediate hosts (which all need ssh-chain installed):

ssh faraway^hop3^hop2^hop1

Since ssh is so central to many tools, this syntax also works for scp, rsync, git, and many others. Really useful if you work with non-trivial network topologies.

NP: Toxoplasma—Weltverbesserer

now-playing is the script generating these “now playing”-lines you can find at the bottom of this post, but I also use it to post the currently playing song to IRC, for example.

Thanks to mpd this is really easy. Back when I used OS X, it was a bit more complicated, having to interface with AppleScript (yikes) and so on. I restored it from an old backup:

#!/bin/sh

osascript -e '
tell application "iTunes"
        set theTitle to name of current track
        set theArtist to artist of current track
        return {theArtist,"---", theTitle} as string
end tell'

NP: Toxoplasma—Alles oder nichts

1t is an old but useful tool I wrote. I like to have small windows showing logfiles with tail -F on my desktop, and it always annoyed me that this wastes one line, showing nothing but the cursor.

1t to the rescue, which disables the cursor and only outputs the final newline after the next line has arrived, thus making use of the last display line as well:

Below:

% tail -F /var/log/messages.log | 1t

NP: Fliehende Stürme—Alles falsch

spongegrep is like grep, but for whole files: if any line matches the regular expression given, all lines are output:

% utter foo bar baz | spongegrep foo
foo
bar
baz
% utter foo bar baz | spongegrep quux || echo not found
not found

It also supports negation:

% utter foo bar baz | spongegrep -v quux
foo
bar
baz
% utter foo bar baz | spongegrep -v foo

The benefit over using grep -q foo file && cat file is that spongegrep works on streams and doesn’t need rewinding.

A good use is checking logs or command outputs for error messages, and eating them up if nothing worrisome happened (also nice in crontabs):

% sensors | spongegrep ALARM

NP: Fliehende Stürme—Es gibt mich nicht!

The stee command made an appearance at day 1 already, but it’s so useful to be it deserves its own mention.

While tee(1) copies standard input to standard output as well as the file argument, stee is silent tee and doesn’t copy to standard output (well, it throws the output away).

What’s that good for, you may wonder? Well, it allows you to write into a file without having to setup a shell direction (nor fiddle with argument strings as with dd of=...). This saves you an additional layer of quoting:

% date | ssh localhost 'cat >/tmp/out'
% date | ssh localhost stee /tmp/out

Also, an often overlooked feature of tee is that it supports output to multiple files. stee of course does as well:

% fortune | stee a b c
% sum a b c
51569     1 a
51569     1 b
51569     1 c

# echo min_power | stee /sys/class/scsi_host/*/link_power_management_policy

(Of course, a good shell can do that already internally.)

NP: Die Schnitter—Klöne

A classic “failure” in shell scripting is to grep the output of ps, which easily generates an erroneous additional result:

% ps ax |grep emacs
 2941 ?        SN    48:31 emacs
21771 pts/27   SN+    0:00 grep emacs

Some people try to fix that this way:

% ps ax |grep emacs |grep -v grep
 2941 ?        SN    48:32 emacs

But that is lame of course, because just making the argument not match itself literally is enough:

% ps ax |grep [e]macs
 2941 ?        SN    48:32 emacs

Clearly, there has to be a better way. I simply use this:

# px -- verbose pgrep
px() {
  ps uwwp ${$(pgrep -d, "${(j:|:)@}"):?no matches}
}

Since it uses pgrep, it searchs inside the program string only:

% px emacs
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
chris     2941  0.0  0.9 162184 76056 ?        SN   Jun08  48:33 emacs

% px swap
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        35  0.0  0.0      0     0 ?        S    Jun08   9:22 [kswapd0]

NP: Die Schnitter—Orange

sel is straight and simple: it selects and raises an X11 window by class. I use it for two purposes: first, when I “lost” some window deep below others, then I use it from dmenu (this happens rarely, and cwm can find windows by name already, anyway). Secondly, I use it for its side-effect of returning with an error code if the window cannot be found. For example, I run my command launcher tkexec like this:

tkexec ... "em:sel Emacs || exec emacs" ...

Thus, pressing the em button will focus Emacs, or start a new one if none can be found. This makes it work quite like the OS X dock.

NP: Die Schnitter—Ich will dich noch einmal sehen

There are many tools to format narrow lines into multiple columns, for example Plan 9 mc, BSD rs or column:

% utter foo{1..16} | 9 mc
foo1  foo3  foo5  foo7  foo9  foo11 foo13 foo15
foo2  foo4  foo6  foo8  foo10 foo12 foo14 foo16
% utter foo{1..16} | rs  
foo1   foo3   foo5   foo7   foo9   foo11  foo13  foo15  
foo2   foo4   foo6   foo8   foo10  foo12  foo14  foo16  
% utter foo{1..16} | column
foo1    foo3    foo5    foo7    foo9    foo11   foo13   foo15
foo2    foo4    foo6    foo8    foo10   foo12   foo14   foo16

One problem with these tools is that a few long elements bloat the output:

% utter foo{1..5} foobarquuxmeh6 foo{7..16} | 9 mc
foo1           foo5           foo9           foo13
foo2           foobarquuxmeh6 foo10          foo14
foo3           foo7           foo11          foo15
foo4           foo8           foo12          foo16

My tool cmc is made for formatting such lists:

% utter foo{1..5} foobarquuxmeh6 foo{7..16} | cmc 
foo1    foo2    foo3    foo4    foo5    foobarquuxmeh6  foo7    foo8    foo9
foo10   foo11   foo12   foo13   foo14   foo15   foo16

Essentially, it aligns the contents into multiples of some column width, for example 6:

% ls / | cmc -t 6
afs   altboot     bin   boot  data  dev   dump  etc   home  lib   lib64
lost+found  mnt   opt   proc  root  run   sbin  service     srv   sys   tmp
usr   var

This format can be a bit quirky, but it’s still easier to scan than a completely unformatted list, and it takes far less space than a strict column layout.

NP: Die Schnitter—Nichstdestotrotz

fp is an interactive file picker based on dmenu. It tries to detect “projects” based on a simple heuristic (finding a .git, .hg, or a Makefile) and then runs dmenu with all files recursively except for some boring stuff.

Using dmenu has the nice benefit that typing any part of the file name makes selection very quick, even within deep directory structures. It’s also nice to use with command substitution:

% vim `fp`

NP: Die Schnitter—Ich Fliege Über Das Meer

hyp looks up the German Wiktionary to figure out how to hyphenate words and outputs in a TeX compatible format:

% hyp Transistor
Tran\-si\-stor
% hyp Determinativkompositum
De\-ter\-mi\-na\-tiv\-kom\-po\-si\-tum

It also works for some foreign words, because the German(!) Wiktionary actually contains a few entries for some other languages as well (unfortunately, other Wiktionary editions don’t have hyphenation data):

% hyp apple
ap\-ple

NP: Die Schnitter—Thomas Müntzer

822par is a wrapper around the powerful but obscure par text re-formatter to reformat e-mails without making them invalid.

It reformats the mail headers specially, creating continuation lines when needed, and filters the rest of the mail through par with appropriate settings.

% cat mail
Subject: This is a badly formatted mail with a very very long subject, in fact much more than 72 characters.

A nice quote:

> There once was a master programmer who wrote unstructured programs.  A 
> novice programmer,
> seeking to imitate him, also began to write unstructured programs.  
> When the novice asked
> the master to evaluate his progress, the master criticized him for 
> writing unstructured
> programs, saying: "What is appropriate for the master is not 
> appropriate for the novice.
> You must understand the Tao before transcending structure."
-- Geoffrey James, "The Tao of Programming"

% 822par mail
Subject: This is a badly formatted mail with a very very long subject, in
        fact much more than 72 characters.

A nice quote:

> There once was a master programmer who wrote unstructured programs.
> A novice programmer, seeking to imitate him, also began to write
> unstructured programs.  When the novice asked the master to evaluate
> his progress, the master criticized him for writing unstructured
> programs, saying: "What is appropriate for the master is not
> appropriate for the novice.  You must understand the Tao before
> transcending structure."
-- Geoffrey James, "The Tao of Programming"

NP: Die Schnitter—Traumgeburt

base is really simple program to convert numbers between various bases.

% base 255
255 = 11111111 255 0377 0xff
% base 0644
0644 = 110100100 420 0644 0x1a4
% base 0b111
0b111 = 00000111 7 07 0x7
% base 0xdeadbeef
0xdeadbeef = 11011110101011011011111011101111 3735928559 033653337357 0xdeadbeef

It also supports the Ruby syntax for writing integer literals, as well as arbitrarily large numbers (easy, because it’s written in Ruby):

% base 100_000
100_000 = 11000011010100000 100000 0303240 0x186a0

NP: Lia Ices—Ice Wine

frep is a pretty special tool, a mix of cut and uniq. I use it to get an overview of very similar data. For example, given a log file:

% cat log
2013-06-15 foo
2013-06-15 bar
2013-07-16 foo
2013-07-16 bar
2013-07-17 baz

We only want to see one line per day, so we filter consecutive duplicates of the first field (or, rather, we print when the first field changes):

% frep 1 < log
2013-06-15 foo
2013-07-16 foo
2013-07-17 baz

And sometimes we want to see the last line for each duplicate value:

% frep -l 1 < log
2013-06-15 bar
2013-07-16 bar
2013-07-17 baz

Finally, we also can refine the field seperator and for example, only list one line per month:

% frep -F- 1 2 < log
2013-06-15 foo
2013-07-16 foo

Another example would be to list which program uses most CPU, for each user:

% ps -eo pid,user,pcpu,comm --sort user,-pcpu | frep 2 
  PID USER     %CPU COMMAND
 3479 chris     2.8 firefox
  971 dbus      0.0 dbus-daemon
25256 dovecot   0.0 anvil
  973 mpd       0.0 mpd
  995 ntp       0.0 ntpd
 8724 postfix   0.0 qmgr
 2474 root      1.2 X
 4040 unbound   0.0 unbound

NP: Cameron Boucher—Life in Vain (Daniel Johnston Cover)

pacsrc is probably my most used Arch helper script: it parses the build description for a package (the PKGBUILD) and outputs the sources used. I often use this when I want to look at the source code of some tool.

% pacsrc zsh
# http://projects.archlinux.org/svntogit/packages.git/plain/trunk/PKGBUILD?h=packages/zsh
http://downloads.sourceforge.net/zsh/zsh-5.0.2.tar.bz2
http://projects.archlinux.org/svntogit/packages.git/plain/trunk/zprofile?h=packages/zsh

The first line is the link to the PKGBUILD, if I want to look up how it’s configured and compiled. The other lines are the sources used, so fetching the tarball is just a simple wget away.

NP: Temple Of The Dog—Wooden Jesus

b is my “I need Internet NOW”-browser: it’s completely command-driven and very limited, but plenty fast for quick lookups:

% b birthday of tbl
   Search [1]Images [2]Maps [3]Play [4]YouTube [5]News [6]Gmail [7]Drive
   [8]More »

   [9]Web History | [10]Settings | [11]Sign in

   [12][logo_sm.gif]  _________________________________________   Search
   [13]Advanced Search
   [14]Preferences

   Web About 2,950,000 results (0.56 seconds)

   [15]Tim Berners-Lee - Wikipedia, the free encyclopedia

   Tim Berners-Lee at the Home Office, London, on 11 March 2010 ... In the
   1997
   Queen's Birthday Honours he was appointed an Officer of the Order of
   the British
    ...
   en.wikipedia.org/wiki/Tim_Berners-Lee - 220k - [16]Cached - [17]Similar
   pages
...
% b 15
   #[1]Wikipedia (en) [2]copyright [3]Wikipedia Atom feed

Tim Berners-Lee

   From Wikipedia, the free encyclopedia
   Jump to: [4]navigation, [5]search
   [6]Page semi-protected
   Sir Tim Berners-Lee
   [7]Tim Berners-Lee-Knight-crop.jpg
   Berners-Lee in 2008
   Born Timothy John Berners-Lee
   (1955-06-08) 8 June 1955 (age 58)^[8][1]
   London, England
   United Kingdom^[9][1]
   Residence United States and United Kingdom^[10][2]
   Nationality British
   [11]Alma mater [12]Queen's College, Oxford
   Occupation [13]Computer scientist
   Employer
     * [14]World Wide Web Consortium
     * [15]University of Southampton
     * [16]Massachusetts Institute of Technology
...

b uses lynx -dump to render the HTML pages, but navigation is quicker by entering numbers than cursor keys.

I use this often via Mosh Connectbot if my connection is really slow, because it’s much faster than using an Android browser.

NP: The Vageenas—Hole In My Head

My tool com is a shell script port of Tom Duff’s tool with the same name.

com allows you to put commands into files (usually compile instructions) and run them:

% cat hello.c 
/*% cc -o # %
 */
main() {
        printf("hello, world!\n");
}
% com hello.c
hello.c: In function ‘main’:
hello.c:4:2: warning: incompatible implicit declaration of built-in function ‘printf’ [enabled by default]
  printf("hello, world!\n");
  ^
% com
hello.c: In function ‘main’:
hello.c:4:2: warning: incompatible implicit declaration of built-in function ‘printf’ [enabled by default]
  printf("hello, world!\n");
  ^

Note that the second com invocation doesn’t pass the arguments since they are saved in .comfile.

I have extended it to also support the magic sequence #% in addition to /*% to support more languages.

NP: Molotow Soda—Letzte Schritte

One of the best Plan 9 features is the dump file system, a nightly snapshot of the filesystem. I use a similar scheme to backup my own machines: Everyday, I run a script dailydump which runs my rdumpfs wrapper around rsync to create a snapshot of my notebook SSD to the hard disk. Files are compared to yesterday’s revision and hardlinked in case they did not change. This makes backups pretty efficient: I snapshot the whole filesystem and the dump increases only by around 250 to 400 MB daily. (That contains all system package updates.) I keep the last month of daily dumps on the machine and later put them to my home RAID for archival. I keep a monthly copy on my notebook as well.

Since rdumpfs supports backing up to and from remote machines, I also use this to backup various machines on my home network that don’t have enough local disk space for a dump.

The tool yday is akin to Plan 9’s yesterday and can be used to compute old file names in the dump. For example, to list all available versions of my .zshrc that are older than 100 days:

% yday -a -n 100 ~/.zshrc
/dump/2013/0401/home/chris/.zshrc
/dump/2013/0314/home/chris/.zshrc
/dump/2013/0301/home/chris/.zshrc
/dump/2013/0214/home/chris/.zshrc
/dump/2013/0201/home/chris/.zshrc
/dump/2013/0114/home/chris/.zshrc
/dump/2013/0101/home/chris/.zshrc
/dump/2012/1231/home/chris/.zshrc
/dump/2012/1207/home/chris/.zshrc

This tool saved my files a few times already. :)

(Another benefit: When some package breaks, I can fiddle with $PATH and $LD_LIBRARY_PATH and almost transparently run old programs against old libaries.)

NP: Molotow Soda—Everything

Today’s shell function is for the Arch Linux users:

pacgrep() {
  PATTERN=${1:?pattern missing}
  shift
  pacman -Qlq "$@" | xargs grep -d skip -e "$PATTERN"
}
_pacgrep() {
  _pacman  # force autoload
  _arguments : '1:pattern:' '*:package:_pacman_completions_installed_packages'
}
compdef _pacgrep pacgrep

I even include command line completion for it. :)

It’s a really simple variant of g actually: it limits the search to files belonging to the given Arch Linux packages. E.g. when I tried to figure out where lhs2TeX mangled my >> in the files, I can search for it:

% pacgrep '>>' lhs2tex
Binary file /usr/bin/lhs2TeX matches
Binary file /usr/share/doc/lhs2tex/doc/Guide2.pdf matches
/usr/share/lhs2tex-1.18.1/lhs2TeX.fmt:%format >>         = "\sequ "
/usr/share/lhs2tex-1.18.1/lhs2TeX.fmt:%format >>=        = "\bind "

Or, I can get a list of Perl scripts included with Git:

% pacgrep bin/perl git
/usr/bin/git-cvsserver:#!/usr/bin/perl
/usr/lib/git-core/git-add--interactive:#!/usr/bin/perl
/usr/lib/git-core/git-archimport:#!/usr/bin/perl
/usr/lib/git-core/git-cvsexportcommit:#!/usr/bin/perl
...

This function is often useful for finding where error messages come from or which internal files are used by the package, without knowing where they are.

NP: Canal Terror—Staatsfeind

When lr is ls the way I want it to be, utter is the equivalent for echo.

utter outputs each argument according to some rules: by default, it outputs each argument on its own line:

% utter foo bar baz
foo
bar
baz

But you can override the strings printed before, after, and inbetween the arguments:

% utter -b 'Simon says: ' -a $'!\n' foo bar baz
Simon says: foo!
Simon says: bar!
Simon says: baz!

There are two predefined modes. One for generating NUL-separated data:

% utter -z foo bar baz | od -c
0000000   f   o   o  \0   b   a   r  \0   b   a   z  \0
0000014

And one for verifying argument quoting:

% utter -q 'foo bar' baz
»foo bar« »baz«

utter does no escape processing at all, your shell probably can do that already.

I mostly use utter as in the first example, and frequently with -q. Why is it better than printf '%s\n'? It doesn’t output anything when no arguments are given.

NP: Fliehende Stürme—Die Axt

uni is a small script that uses the Perl Unicode tables to search for Unicode characters by name, glyph or hexadecimal codepoint.

% uni gamma 
Ɣ       0194    LATIN CAPITAL LETTER GAMMA
ɣ       0263    LATIN SMALL LETTER GAMMA
ˠ       02E0    MODIFIER LETTER SMALL GAMMA
Γ       0393    GREEK CAPITAL LETTER GAMMA
γ       03B3    GREEK SMALL LETTER GAMMA
Ϝ       03DC    GREEK LETTER DIGAMMA
...

% uni ツ
ツ      30C4    KATAKANA LETTER TU

% uni 00B1
±       00B1    PLUS-MINUS SIGN

The nice thing about this script is that it works everywhere a half-way modern Perl is installed and doesn’t need additional tables or internet access.

NP: Fliehende Stürme—Höhlen sind dunkel

A great addition to yesterday’s script, f is total, which simply sums up the n-th column.

% lr -l | f 5 | total
8595
% du -bs
8595    .

Again, it’s a pretty trivial short-cut, yet a very useful one.

NP: Toxoplasma—Gut und böse

This script actually isn’t by me, but by Mark J. Dominus. It’s very simple, but I use it often: it simply outputs the n-th field.

% lr -l | f 5
220
10
41
...

Compared to cut -f, it has a few advantages: it can output the last fields (by negative index) and it seperates on arbitrary whitespace (some BSD now have cut -w for this, a great addition).

And, of course, it’s quick to type.

NP: Toxoplasma—Weltverbesserer

Another script originating from a debugging session (like swaptop) is zombies, a convenient way to list all zombie processes and what you need to kill to get rid of them:

% zombies
xdg-open/2776 zombie child of hotot/29421
firefox-limited/26341 zombie child of firefox/26337

The implementation is simple, but uses the neat trick of tree-sorting the processes to ensure parents processes are seen before their children.

NP: Toxoplasma—Heile Welt

What l is to file names, g is for file contents: A quick way to search by regular expressions:

g() {
  local p=$argv[-1]
  [[ -d $p ]] && { p=$p/; argv[-1]=(); } || p=''
  grep --exclude "*~" --exclude "*.o" --exclude "tags" \
    --exclude-dir .bzr --exclude-dir .git --exclude-dir .hg --exclude-dir .svn \
    --exclude-dir CVS  --exclude-dir RCS --exclude-dir _darcs \
    --exclude-dir _build \
    -r -P ${@:?regexp missing} $p
}

A popular incarnation would be, for example:

src/emacs-24.3% g '^main' src 
src/Makefile:maintainer-clean: distclean
src/makefile.w32-in:maintainer-clean: distclean
src/xrdb.c:main (int argc, char **argv)
src/Makefile.in:maintainer-clean: distclean
src/buffer.c:maintained internally by the Emacs primitives.  Enabling or disabling
src/emacs.c:main (int argc, char **argv)
src/tparam.c:main (int argc, char **argv)
src/termcap.c:main (int argc, char **argv)

As you can see, it filters some useless junk and again offers the ability to specify a different directory to search in as the last argument (just like l).

I use this all the time to navigate both foreign and my own source code. With a SSD, it’s plenty fast even on large file trees.

NP: Toxoplasma—Polizeistaat

Of course, for serious development I use Git, but sometimes I just hack around on a script, prepare a draft, or golf down some code and want to keep old versions of a file. For this, I use keep:

% keep foo
‘/home/chris/mess/2013/27/foo’ -> ‘/home/chris/mess/2013/27/foo.1’
% vi foo
% keep foo
‘/home/chris/mess/2013/27/foo’ -> ‘/home/chris/mess/2013/27/foo.2’
% keep foo
/home/chris/mess/2013/27/foo.2 not modified

As you can see, it creates numbered backup files—if the file has changed. The major benefit over doing this manually (with cp foo foo.1) is that it won’t ever overwrite an old revision accidentally because you blindly used your shell history.

I have not yet felt the need to add diffs or restoring of old versions.

(Tom Duff has an interesting, but more complicated version of this program.)

NP: Lia Ices—After Is Always Before

A good start to debug various slowdowns in programs is to analyze virtual memory usage, especially on machines with low amounts of RAM.

I wrote swaptop to calculate which processes use swap, and how much swapped:

% swaptop
  293240  hotot 29421
  140880  firefox 26337
   90760  evince 1856
   84472  skype 1674
...

It shows kbytes of swap used by the process and its pid.

Since some kernel version the swap usage is presented directly in /proc/$$/status, which makes the script pretty trivial. But before that, you had to calculate swap usage by adding up swapped pages from /proc/$$/smaps, which makes the script a bit slow on such old kernels.

NP: Die Schnitter—Aderlass

As you can guess by their names, l and lr are tools I use frequently. l was there first and simply lists all files recursively that possibly match a given pattern.

l() {
  local p=$argv[-1]
  [[ -d $p ]] && { argv[-1]=(); } || p='.'
  find $p ! -type d | sed 's:^./::' | egrep "${@:-.}"
}

It’s nice to naviate large codebases:

src/emacs-24.3% l term.c
etc/e/eterm-color
etc/e/eterm-color.ti
src/term.c
src/w32term.c
src/xterm.c
lisp/term/common-win.elc
lisp/term/cygwin.elc
lisp/term/cygwin.el
lisp/term/common-win.el

Note that it uses extended regular expressions instead of globs.

Also, you can pass a directory as the last argument to search there:

% l utmp /usr/share/man  
/usr/share/man/man8/systemd-update-utmp-runlevel.service.8.gz
/usr/share/man/man8/systemd-update-utmp.service.8.gz
/usr/share/man/man8/utmpset.8.gz
/usr/share/man/man8/systemd-update-utmp.8.gz
/usr/share/man/man0/utmpx.h.0p.gz
/usr/share/man/man5/utmp.5.gz
/usr/share/man/man5/utmpx.5.gz
/usr/share/man/man1/utmpdump.1.gz
/usr/share/man/man3/utmpxname.3.gz
/usr/share/man/man3/utmpname.3.gz
/usr/share/man/man3/getutmpx.3.gz
/usr/share/man/man3/getutmp.3.gz

l is a very handy tool that I use often.

It’s bigger and newer brother is lr, which emulates ls -R in a more reasonable way. It’s quite a mouthful:

lr() {
  zparseopts -D -E S=S t=t r=r h=h U=U l=l F=F d=d
  local sort="sort -t/ -k2"                                # by name (default)
  local numfmt="cat"
  local long='s:[^/]* /::; s:^\./\(.\):\1:;'               # strip detail
  local classify=''
  [[ -n $F ]] && classify='/^d/s:$:/:; /^-[^ ]*x/s:$:*:;'  # dir/ binary*
  [[ -n $l ]] && long='s: /\./\(.\): \1:; s: /\(.\): \1:;' # show detail
  [[ -n $S ]] && sort="sort -n -k5"                        # by size
  [[ -n $r ]] && sort+=" -r"                               # reverse
  [[ -n $t ]] && sort="sort -k6" && { [[ -n $r ]] || sort+=" -r" } # by date
  [[ -n $U ]] && sort=cat                                  # no sort, live output
  [[ -n $h ]] && numfmt="numfmt --field=5 --to=iec --padding=6"  # human fmt
  [[ -n $d ]] && set -- "$@" -prune                        # don't enter dirs
  find "$@" -printf "%M %2n %u %g %9s %TY-%Tm-%Td %TH:%TM /%p -> %l\n" |
    $=sort | $=numfmt |
    sed '/^[^l]/s/ -> $//; '$classify' '$long
}

A plain lr is like a find except for the stupid ./ at the beginning of every line:

% lr
.
bar
bar/bla
bar/quux
foo/
foo/a/
foo/a/b
foo/a/b/c

However, you can pass -l to get more detail (It even uses a reasonable date format. ;)):

% lr -lF
drwxrwxr-x  4 chris users        80 2013-06-29 16:28 ./
drwxrwxr-x  3 chris users        80 2013-06-29 16:28 bar/
drwxrwxr-x  2 chris users        40 2013-06-29 16:28 bar/bla/
-rw-rw-r--  1 chris users        57 2013-06-29 16:28 bar/quux
drwxrwxr-x  3 chris users        60 2013-06-29 16:28 foo/
drwxrwxr-x  3 chris users        60 2013-06-29 16:28 foo/a/
drwxrwxr-x  2 chris users        60 2013-06-29 16:29 foo/a/b/
-rw-rw-r--  1 chris users       317 2013-06-29 16:29 foo/a/b/c

The advantage is has is that it can sort over all arguments given. (By size, name, or date.) Compare:

% ls -lFRS bar foo/a/b 
bar:
total 4
-rw-rw-r-- 1 chris users 57 Jun 29 16:28 quux
drwxrwxr-x 2 chris users 40 Jun 29 16:28 bla/

bar/bla:
total 0

foo/a/b:
total 4
-rw-rw-r-- 1 chris users 317 Jun 29 16:29 c


% lr -lFS bar foo/a/b
drwxrwxr-x  2 chris users        40 2013-06-29 16:28 bar/bla/
-rw-rw-r--  1 chris users        57 2013-06-29 16:28 bar/quux
drwxrwxr-x  2 chris users        60 2013-06-29 16:29 foo/a/b/
drwxrwxr-x  3 chris users        80 2013-06-29 16:28 bar/
-rw-rw-r--  1 chris users       317 2013-06-29 16:29 foo/a/b/c

Also, it keeps relative paths for copy and pasting:

% ls -lF ../22
total 4
-rwxrwxr-x 1 chris users 2160 Nov  3  2004 fix-maildir-timestamps.pl*

% lr -lF ../22
drwxrwxr-x  2 chris users      4096 2013-06-03 21:28 ../22/
-rwxrwxr-x  1 chris users      2160 2004-11-03 18:32 ../22/fix-maildir-timestamps.pl*

If you look closely, you’ll see that other arguments are passed to find verbatim, so you can do more complex searches:

% lr -lF /usr/lib -size +80M
-rwxr-xr-x  1 root root  87749264 2013-06-20 06:25 /usr/lib/chromium/chromium*
-rw-r--r--  1 root root 143132030 2013-04-21 19:53 /usr/lib/ghc-7.6.3/ghc-7.6.3/libHSghc-7.6.3_p.a
-rw-r--r--  1 root root  85721136 2013-06-06 13:08 /usr/lib/maxima/5.30.0/binary-sbcl/maxima.core

I think lr is pretty much to how I actually want ls to work.

NP: Die Schnitter—Rebellenlied

I generally dislike unasked-for color output of various tools, especially grep (grep is there to “print lines matching a pattern” and not confuse me with highlighted terms). That’s why I usually disable such things.

However, sometimes I have to debug ugly or very verbose output and want to highlight some words. For these cases, I (ab)use the --color function of grep in a zsh function hl:

hl() {
  egrep --color=always -e '' -e${^*}
}

The purpose of grep is defeated by searching for the empty string, so all lines will be shown, but the matches still are being highlighted:

% head -c4k /dev/urandom | xxd | hl f00
...
0000f20: 3205 ab77 a1a0 5eeb c5e6 2c6e 42d6 1195  2..w..^...,nB...
0000f30: 7b0a 5ad0 3d6f 79ef 0dfc f0aa 6819 d168  {.Z.=oy.....h..h
0000f40: 6aea 3280 38f2 e437 54d9 372d 1b4f 5f00  j.2.8..7T.7-.O_.
0000f50: 6a90 d459 afc3 da2b b0bb 2349 b5a5 dc44  j..Y...+..#I...D
0000f60: a1e8 d21d f8ba db6a 0908 eb1c b2e3 ad48  .......j.......H
0000f70: c871 8dde a3ee 019c 6053 2a4a b205 2136  .q......`S*J..!6
...

Also note that several patterns can be given on the command line:

% objdump -d =grep | hl push pop | less -R

NP: Die Schnitter—Widerstand

lsort is a very simple tool (in fact, just a Perl one-liner), but it does a job that is not very easy to do with classic Unix on-board equipment: sort lines by their length.

An example application would be to cheat at some word games: what’s the longest word that contains all vocals in reverse alphabetic order?

% grep u.*o.*i.*e.*a /usr/share/dict/words |lsort
unnoticeable
subcontinental
uncomplimentary

Together with head and tail, this tool is also useful for finding the shortest and longest lines (e.g. to find very long filenames or huge entries in symbol tables):

% nm -D /usr/lib/chromium/chromium | lsort | tail -1

(Actually, wc -L can output the longest line length, but not the longest line.)

NP: Aimee Mann—Driving Sideways

Apart from urxvt and Emacs, the program I spend most time with is Firefox. Thus, it’s natural I want to control it from these other programs too. Opening links is simple by just executing firefox, but to get access to its internal state, I’ve found the following two tools very helpful:

lstab lists all tabs that are currently open with their URL and title:

% lstab
https://mail.google.com/mail/u/0/?ui=2&shva=1 # Gmail
http://www.techism.de/ # Techism - Events, Projekte, User Groups in München!
http://www.ilxor.com/ILX/ # ilXor.com
...

I use this tool to “synchronize” the tabs between my notebook and my desktop. E.g. if I was surfing on my desktop (which is always on), but need to switch to my notebook, I just run ssh hecate lstab and click on the few links I wanted to read.

On the other hand, if I surf on my notebook and want to go home, I just copy the open tabs to the desktop machine:

lstab |ssh hecate stee mess/current/tabs

(This simple—but very helpful to me—command line inspired this series of posts, by the way.) stee will be explained later, essentially it writes stdin to the argument file name. The purpose of mess/current has been explained here already.

The companion tool is curtab, which simply outputs the URL of the last selected tab:

% curtab
http://www.imdb.com/title/tt0052561/

I use this most often when preparing Trivium, where a simple M-x insert-curtab pastes the current URL into my Emacs buffer.

How do these scripts work? They parse Firefox’s sessionstore.js file which keeps all this data formatted as JSON. If I ever wanted to replace my browser, I’d certainly need to figure out how to rewrite these scripts first.

NP: Aimee Mann—How I Am Different

Hey, it’s July and I decided to run a “Summer of Scripts”. Every day of this month I will present a shell script, a small program or a shell function and explain how I use it in my daily life.

I hope it will interest you and you can find something useful there.

NP: The Pretty Reckless—Everybody Wants Something from Me