I lost my “administrative hymen” last month when a server of mine was rooted due to the recent Exim exploit (CVE-2010-4345). This is a post-mortem of the incident.
For shame, it took me almost a month to even detect the break-in, when I wondered why I didn’t get any mail from that address.
The same intruder cracked many machines and installed rootkits on them. He was not very professional: he left traces in the “panic log” of exim when he tried to download a configuration file with wget like this:
wget ... >exim.conf
instead of
wget -O exim.conf ...
… which of course writes the wget status messages into the file,
too, which results in fatal parse errors:
# cat /var/log/exim4/paniclog
2010-12-16 01:09:54 string too large in smtp_notquit_exit()
2010-12-16 02:27:17 string too large in smtp_notquit_exit()
2010-12-16 05:12:11 string too large in smtp_notquit_exit()
2010-12-16 14:49:27 string too large in smtp_notquit_exit()
2010-12-16 18:25:20 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:25:20-- http://62.141.42.28/exim4.conf
2010-12-16 18:25:20 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:25:20-- http://62.141.42.28/exim4.conf
2010-12-16 18:45:03 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:25:20-- http://62.141.42.28/exim4.conf
2010-12-16 18:54:13 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:54:12-- http://62.141.42.28/exim4.conf
2010-12-16 18:55:02 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:54:12-- http://62.141.42.28/exim4.conf
2010-12-16 19:55:02 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
option setting expected: --2010-12-16 18:54:12-- http://62.141.42.28/exim4.conf
The installation of the rootkit also changed timestamps:
He added a key to root’s .ssh/authorized_keys and started another sshd
on a non-standard port (59997). He did not detect that this port
was blocked out-bound with iptables.
The exploit circulated around December 7, the patch was released December 10, and I got hacked December 16. And I actually read the news but thought I was not affected. Sigh.
Clearly, this should not happen. I admit the following mistakes:
The system ran a old (2007), un-updated version of GRML that I installed in a hurry with grml2hd without spending too much time thinking about the details.
It was not updated often because things used to break, being based in Debian unstable, and since it had many packages installed, updating often was a hassle.
It came with loads of stuff that I didn’t need, but they cluttered up logwatch and other intrusion detection tools.
As a result of this, I didn’t forward Cron messages to a mailbox I actually read.
It came with Exim preinstalled, so I used that for the single mailbox I had on that machine.
(Not that this would have saved much; the bug was also in the last version before the security update, and since the intruder (accidentally) shut down the mail system, I would not have gotten any messages about the break-in.)
Lessons learned for the future:
Set up minimal, but meaningful logging, so it will be read.
Send administrative messages to external hosts.
Update the system regularily. I now installed Debian squeeze, which is going to be stable “really soon now”, and then will serve its purpose for the next few years with comparatively few updates.
Do a minimal install. This time I used
grml-debootstrap, which only installed a core Debian system, on which I added exactly the packages I need.Use Postfix instead of Exim. Postfix not only is vastly easier to set up than exim (at least on Debian, Postfix default configuration fits on a single screen, while Exim has a configuration directory filled with almost 2000 lines of text), but also has a better security record. And I use Postfix on other servers too, so I will be more alert if I read about it being unsecure.
Considerations for the future that I probably won’t follow:
Make lots of partitions and sprinkle noexec, nosuid mount options. (Complicates setup and disk space planning.)
Setup remote syslog. (I don’t have the resources for this, not worth it for a single machine.)
Switch to a distro that doesn’t package antique stuff. (Which?)
Use SELinux or such stuff.
NP: Leonard Cohen—Avalanche