Terry Jones

I like customizing my environment. I’ve spent lots and lots of time doing that over the decades.

Some examples: My emacs environment has about 6000 lines of elisp that I’ve written to help me edit. I have over 500 shell scripts in my bin directory (30K lines of code), and certainly hundreds of other scripts around the place to help with other specific tasks. My bash setup is about 2000 lines of shell script.

That’s about 40K lines of code all written just to help me edit and work in the shell.

As a computer user, I’m damned happy I’m a programmer. I don’t think I can imagine what it would be like to be a computer user and not a programmer.

As a non-programmer you’re at the mercy of others. When you run into a problem you don’t have a solution for, you’re either out of luck, you have to spend often huge amounts of time solving it in some contorted semi- or fully-manual way, you have to find someone else’s (likely partial) solution and maybe pay for it, or you ask or pay someone to solve your problem, or you wait for the thing you need to appear in some product, etc. And all the while you’ve got a perfectly good high-speed general-purpose machine sitting right in front of you, likely with all the programming tools you’d need already installed for you…. but you don’t know how to use it!!

How weird is that?

As a programmer when you run into a problem you don’t have a solution for, you can just write your own.

One thing that always surprises me is how little time most other programmers tend to spend customizing their environments. Given 1) that programmers probably spend a large percentage of each day in their editor, in email, and in the shell, 2) that those things can all be programmed (assuming you use emacs :-)), and 3) that programmers usually don’t like repeating themselves, doing unnecessary work or being inefficient, you’d think that programmers would all be spending vast amounts of time getting things set up just so.

FWIW, here’s a description of the email setup I’ve built up over the years.

But first some stats.

I’ve been saving all my incoming and outgoing emails since Sept 19, 1989. I don’t know why I didn’t start earlier – I wish I had. My first 7 years of emailing is lost, almost certainly forever. I’ve sent 125K emails in that time and received 425K. I’ve got all my incoming email split into files by sender, with some overlap, in 6700 files. The total disk usage of all mails is just under 4G. I have 1.1G (compressed) of saved spam. I have 1250 mail aliases in my .mailrc file.

1. I write mail in emacs, of course. Seeing as email is text, why would you use anything but your text editor to compose it? Not being able to use emacs to edit text is a show-stopper for me when it comes to using software products. Don’t try to make me use an inferior editor. Don’t ask me to edit text in my browser.
2. All my outgoing mails get dumped into a single file. I occasionally move these files when they get too big. I keep things this way as it’s then really fast to look at stuff I’ve sent, which I do frequently. I have shell commands called o, oo, ooo etc., to show me the last (second last, etc) of these files (starting at bottom) instantly.
3. I read mail in emacs (using VM). I could do that differently, but email is (usually) text and I want to copy it, paste it, edit it, reply to it, etc. I also use the emacs supercite package, smart paragraph filling, automatic alias expansion, etc. All that has been standard in emacs for at least 10 years, but it’s still not available in tons of “modern” email readers.
4. VM recognizes the 37 email addresses I’ve used over the years as indicating a mail is from me (and so doesn’t put that address in any followup line).
5. I do all my MIME decoding manually. VM knows how to handle most things, I just don’t let it do it until I want it done. That’s mainly a security thing – several years ago I predicted that PDF files would one day be used to trigger buffer overflows, as just happened. I don’t open any attachment of any form from anyone I don’t know (and don’t open them from some people I do know who like to pass along random crap from others).
6. I have VM figure out exactly where each mail should be saved, based on sending email address. So I never have to make a decision about where to save anything.
7. I have 154 virtual folders defined in VM. These let me dynamically make a mail folder based on fairly flexible rules (subject, sender, etc). They’re not folders on disk, but are composed from these on the fly. It’s a great feature of VM, highly useful. E.g., I have friends with multiple email addresses – my friend Emily has used 21 emails addresses in the last 15 years and I can see all her incoming mail in one virtual folder no matter where she sends it from. Virtual folders can be used for much more than that though.
8. I have an emacs function that detects if the person sending me mail also uses VM and, if so, lets me know if their version of VM is newer than mine. That way I don’t have to think about upgrading VM – when a friend does it, emacs tells me automatically.
9. I have VM keys set up to send messages to SpamBayes to teach it that things are spam or ham.
10. I have an emacs hook function that looks at the mail I’m currently looking at in VM and sets my email address accordingly. So if I’m reading mail from Cambridge it sets my address to be my Cambridge one, and similarly for Fluidinfo, for my jon.es domain and a couple of others. That means I pretty much never reply to an email using an address I didn’t want to use on that email. That’s all totally automatic and I never have to think about email identity, except when mailing someone for the first time.
11. VM also does a bunch of other things for me, like add attachments, encrypt and decrypt mail, etc. But that’s all fairly standard now.
12. I use a script I wrote to repeatedly use fetchmail to pull my incoming mails from half a dozen mailboxes.
13. I use grepmail to search for emails. It’s open source, so I was able to speed it up, fix some problems I ran into, and add some enhancements I wanted in versions 4.72 and 4.80.
14. In front of grepmail I run my own mail-to program which knows where I store my outgoing mail, parses command line from and to dates to figure out the relevant files to pass to grepmail, etc.
15. I use cron and some scripts to maintain a list of email addresses I’ve ever received/sent mail from/to (78500 of these) or just received from (40K of these). Cron updates these files nightly, using another program that knows how to pull things that look like emails out of mail files.
16. I have a shell script which looks in the received mail address file to find email addresses. So if I am wondering about what someone’s address from, e.g., Siemens might be, I can run emails-of siemens and see 140 Siemens email addresses. Yes, I used to send a lot of mail to Siemens.
17. I use procmail to filter my incoming mail. With procmail I do a bunch of things:
18. Procmail logs basic info on all my incoming mail to a file.
19. It looks for a special file in my home directory, and if it’s there it forwards mail to my mobile phone.
20. It also looks for mail from me with a special subject, and when, found either creates or removes the above file. This allows me to turn forwarding to my mobile phone on and off when I’m away from my machine.
21. It dumps some known spam addresses for me.
22. With procmail I run incoming mail through a script I wrote that looks at the above file of all known (received) mail addresses. This adds a header to the mail to tell me it’s from a known former sender. Those mails then get favorable treatment as they’re very likely not spam.
23. With procmail I run incoming mail through another program I wrote that looks at the From line and marks the mail as being something I want delivered immediately. If not it gets put aside for later viewing.
24. With procmail I run incoming mail through another program I wrote that looks at the overall MIME structure of the mail and flags it if it looks like image spam (hint: don’t send me a GIF image attachment).
25. Finally, I also use procmail to run incoming mail through both SpamBayes and Spam Assassin.
26. I used to use procmail to auto-reply to anything considered spam (and then auto-drop the many bounces to this). But I turned that off as it was making too many mistakes replying to forged mails from mailing lists.
27. I have a program that cron runs every night which goes through the day’s spam and summarizes the most interesting messages. It typically pulls out 15-20% of my spam into a summary mail which it sends me. The summary is sorted based on the mail address in the To line (my old mail addresses get scored very low). It also identifies common subjects (so I can kill them), and does some checks like tossing emails whose subjects are not composed of at least some recognizable words. This program is pretty severe – all these mails have already been classed as spam by one of the above programs, so this is just a safety check that I haven’t tossed anything I should keep. It generates a piece of emacs lisp for each message it pulls out so I can jump straight to the correct spam folder and message number in case I want to look at something. It also keeps a list of things to watch for that are definitely not spam. With this program in place I never go looking in my spam folders. I can also run this from the shell at any time.
28. I have a program that summarizes the mail I’ve put aside (not for immediate delivery). Cron runs that nightly and mails me the result. I can also run this from the shell at any time.
29. I have a simple program I use to grep for mail aliases in my .mailrc.
30. I have a script which lists my received email files in reverse order of last update. I can pipe the output of that program into xargs grep to quickly search all incoming mail, in new-to-old order (for speed), mentioning any term.
31. I have a script to send unrecorded mail (from the shell). That’s mail that doesn’t have my usual FCC line in it, in case I’m mailing out something large and don’t want a copy of it in my outgoing mail file.
32. I have an emacs function to visit my current outgoing mail folder with backups disabled (the folders are often large and I rarely want to edit them).
33. And I can’t resist pointing out that I wrote the Spamometer in 1997 to do probabilistic spam detection, and set up a Library of Spam (which attracted a hell of a lot of spam). This was 5 years before Paul Graham wrote his famous A Plan for Spam article about doing Bayesian filtering to detect spam. The Spam Assassin is very similar in approach and design.

You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.