tailwatchd failed @ Tue Jul 12 09:35:01 2011. A restart was attempted automagically.

It turns out that I needed to install MySQL for tailwatchd to work properly again (whatever that daemon does).

What I need to do :

1. Remove /etc/mysqldisable
2. /scripts/mysqlup –force
3. /scripts/checkperlmodules
4. /etc/init.d/cpanel restart

I just want to configure my servers to sync time with a public NTP server. Using the default configuration (/etc/ntp.conf), ntpd starts correctly but never writes anything to /var/log/messages besides the first lines at startup.

What I have found is that the “restrict” lines in the default ntp.conf file do not seem to allow ntpd to connect to the servers that are listed there (*.centos.pool.ntp.org by default). Adding the following lines fix the problem :

restrict 0.centos.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 1.centos.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 2.centos.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
# the following is in the default ntp.conf file
server 0.centos.pool.ntp.org
server 1.centos.pool.ntp.org
server 2.centos.pool.ntp.org

Also (as an alternate fix) I had success with the following servers :

server ntp.isc.org
server clock.redhat.com

I’m not sure why, I suspect that’s because in the DNS they resolve to a single IP address, but I don’t really understand what would cause this.

Anyway, with these changes my NTP now sync properly, and I can see the “synchronized to 206.248.171.198, stratum 1″ lines in the /var/log/messages file.

The Plone display menu

With old skin templates, you could change the title just by adding a .metadata file or by editing it directly in the ZMI. With Zope 3 views, however, I had no idea how to do this.

So I searched and did not find much information about it, but thankfully I found this thread in the Core Developers mailing list, which is about adding this exact feature to Plone!

Here how it works : 

   <browser:page
        for="OFS.interfaces.IFolder"
        name="myview_view"
        class=".myview.View"
        template="templates/myview.pt"
        permission="zope.Public"
        />

   <browser:menuItems
        for="*"
        menu="plone_displayviews">
        <browser:menuItem
            title="My View Title"
            action="myview_view"
            description="Display something"
            />
    </browser:menuItems>

Here I’m changing the title for myview_view to “My View Title”. The first XML tag is just the view registration (nothing special here). The browser:menuItems and browser:menuItem is what’s important. plone_displayviews is the name of the menu you need to change. You can add as much browser:menuItem as you want in the menuItems tag. The action parameter is what is actually making the connection between the view and the menuItem.

Thanks to Martijn Pieters, Martin Aspeli and others involved in adding this feature!

I first tried doing that manually with GNU parted, but that turned out to be hard to do, it seems I could never get the partitions to have exactly the same size…

So I searched a bit on Google, and found a way to make a backup of the partition table of one disk, and copy it to another disk very easily. As usual, the dd command is always a great tool for that kind of job.

First, here what you need to do to create a backup of the MBR (which contains the partition table), and save it to a file.

# dd if=/dev/sda of=mbr.bin bs=512 count=1

This will copy the first 512 bytes of your disk (/dev/sda), which happen to be where the MBR is located. The “of=” (output file) parameter means that the MBR will be saved in the file mbr.bin of the current directory. Then, this is what you need to do to restore it, this time on /dev/sdc : 

# dd if=mbr.bin of=/dev/sdc bs=512 count=1

You can also replace only the partition table, but not the code area, which contains the boot loader such as GRUB. The code area uses the first 446 bytes at most, while the partition table is in the next 66 bytes of the hard disk (64 bytes + 2 bytes signature). So by skipping the first 446 bytes, you will only get the partition table from your backup, and keep the rest intact : 

# dd if=mbr.bin of=/dev/sdc bs=1 count=64 skip=446 seek=446

That’s all! See the Wikipedia article on Master boot record for more information about what is in the MBR.

Re-read partition table 

After altering the MBR, you will probably need to tell Linux to reload the partition table in order to see the changes. GNU parted provides a simple tool for that, partprobe. Just execute “partprobe” as root and you should be OK.

Verifying your partition table

To be sure you partition table is OK and exactly the same between /dev/sda and /dev/sdc, just look at the special file /proc/partitions

# cat /proc/partitions

major minor  #blocks  name
   8     0  488386584 sda
   8     1     514048 sda1
   8     2  487869952 sda2
   8    16  488386584 sdb
   8    17     514048 sdb1
   8    18  487869952 sdb2
   8    32  488386584 sdc
   8    33     514048 sdc1
   8    34  487869952 sdc2

The grub manual provides some information about fallback options : http://www.gnu.org/software/grub/manual/html_node/Booting-fallback-systems.html . Unfortunately, I didn’t manage to make this work (one command it refers, grub-set-default, doesn’t even exist in Fedora/CentOS). The options described in the manual are also a bit too complicated for my needs.

What I really need is the ability to boot into a new kernel, and in case it’s doesn’t boot correctly, I should be able to reset the server (using a remote power switch), and the old (working) kernel should come up instead of the new one. Well, the “savedefault” command of grub does just that!

All you have to do is enter the grub console by typing “grub” at the command line. Then enter the following command :

savedefault --default=1 --once
quit

The default option specify which kernel should be loaded (1 is the second kernel listed in grub.conf). The –once parameter is important. It tells grub to use this settings just one time (otherwise, it won’t work, grub will just boot the default kernel from grub.conf).

Be sure to set the “default=0″ option in /etc/grub.conf to a “safe” kernel. (Of course, if your old kernel is not the first entry, change the “0″ accordingly)

That’s it! Just reboot the server, it will boot on kernel “1″ (the one from the savedefault command). If it works, edit grub.conf to set this kernel as the default. If it doesn’t work, reset the server, and grub will boot on the default kernel.

The script was made for a system running Exim, MySQL 4 and Python 2.3 on Redhat Linux, so it should be easy to install for similar configurations. Of course, it should be possible to run it on any environnement and MTA (if your MTA gives you the possibility of refusing mail based on the result of an external command).

Here is a list of features of my project :

• Whitelisting of single IP addresses
• Whitelisting of IP networks (CIDR)
• Support for SPF (if enabled, messages that pass SPF test will not be greylisted)
• You can enable greylisting by email adress and/or by domain.
• Configurable parameters, such as initial delay and expire delay.

You can download greylisting-py (yes, I did not spent much time trying to find a name for it from the Google code page : http://code.google.com/p/greylisting-py/. There’s also instructions on how to install it and use it with Exim.
I hope it will be useful for some people!

But what is greylisting?

Greylisting is a technique to help fight spam, very useful as a complement to other filters like SpamAssassin. Basically, it works this way :

1. MTA 1 receives a message from MTA 2. It dectects a new “triplet” [sender ip, sender address, recipient address], and since it’s the first time it sees that triplet, it refuses the message with a temporary error code (4xx).
2. Since the error code was temporary, MTA 2 keep the message in queue to try sending it again later (usually after a delay between 1 and 60 minutes)
3. MTA 1 receives the second delivery attempt. It knows that the same triplet has tried sending a message previously, so it accepts the message.

If MTA 2 was a bot sending spam, chances are that it would never bother trying again, so the spam will never come trough. In my experience, greylisting really helps with spam filtering, but of course with the drawback that some messages are delayed for several minutes.

Greylisting drawbacks

There are some drawbacks to using greylist :

• Message will be delayed from about 15 minutes to 1 hour, depending on the greylisting implementation and the sending MTA. This delay will only happen the first time a server with a given IP address send a message with a given sender and recipient address (MAIL FROM/RCPT TO).
• Greylisting may cause problem with some big networks (like gmail, hotmail, …). These networks sometimes send emails from a different IP at each new try, so the greylisting delay will always start back from zero, until the same IP is used again. This may cause messages to be delayed longer than intended, or ultimately fail. This problem can be solved by whitelisting these networks and using SPF to accept message without going trough the greylist (if a server passes SPF test, it will most likely also pass the greylist).
• Some non-compliant servers could treat the temporary error as an permanent error and drop the message, but I don’t think any important MTA have this problem.

For more details, look at the Wikipedia article.

However, when updating CentOS to the new 4.6 version on a Virtuozzo VPS, you may encounter a dependency error. On my VPS, I got the following message :

Missing Dependency: glibc-common = 2.3.4-2.36 is needed by package glibc-dummy-centos-4

The problem is with the glibc-dummy-centos-4 package (provided by SWsoft), wich seems to not accept the new glibc-common version.

A solution I have found after searching on the web, is to simply remove the glibc-dummy-centos-4 package. Apparently, this package doesn’t do anything (well, the name does say it’s dummy , so removing it does not harm.

So the solution is :

rpm -e glibc-dummy-centos-4

You could also add exclude=glibc* to you yum.conf file, which is another solution I have found on the CentOS forum.

After that, you can resume your update with “yum update” without any problem!

Simple rsync/SSH backups

rsync is a powerful tool that, combined with SSH, can be used to easily transfer a large number of files over a network connection. What’s more, it will also save bandwidth and time by only transferring files that have actually changed since the last backup. You can do a backup from one server to another with a command as simple as this one :

rsync -a –delete user@source.com:/var /destinationdir

Put this command in a crontab, and you have a basic backup system working!

SSH keys

A problem you may have with this solution is that the rsync command will ask for a password when run (the password for the remote user, since it uses SSH). This doesn’t work well with a backup script that you want to run each night at 00:00, for example.

There is a solution to that problem, of course! You can generate an SSH key that will allow you to login to the remote system without any password. I won’t go into details here (you can search about SSH keys on the web for more information), but here are the basics steps :

1. Generate a private/public keypair on the “source” server, using the “source” user.
2. Copy the public key to the “target” system, and copy it’s content in the “.ssh/authorized_keys” file of the “target” user.
3. Login from the source server/user, to the target server/user, SSH will login using the private key and won’t ask any password!

How to make differential backups…

It is the power of hard links that enables us to do an incremental backup. The command “cp -la dir1 dir2″ will create a copy of the dir1 directory to dir2, using hard links for files (it doesn’t make links for directories, that wouldn’t make sense if you think about it…). The copy will take almost no space on disk, and will behave exactly like it was a real copy.

The idea of the rsync incremental backup is that you make a “cp -la” copy of your backup (made previously via rsync), which gives you two exactly same directories. Then, you can run rsync again to update one of the backups, which will give you an up-to-date backup, while the previous one stays exactly the same. You now have what seems like two complete backups, but it takes only the space of the full backup plus the changes between the two. The ease of use of the full backup strategy with the benefit of small disk space used by the incremental backup strategy!

A simple backup script

Here is a simple shell script that would create backups using this method. It will keep the latest 10 backups in directories from 0 to 9. 0 will always be the latest backup and 9 the oldest. Only the “0″ backup is updated with rsync, after the rotations and hard-link copy have been made.

#!/bin/sh
# rotate backups
rm -fr 9
mv 8 9
mv 7 8
mv 6 7
mv 5 6
mv 4 5
mv 3 4
mv 2 1
mv 1 2
# now make a copy using links of the latest backup
cp -la 0 1
# update the "0" backup with fresh files
rsync -av --delete backup@myserver.net:/ /storage/backups

That’s all! Of course, this script doesn’t do any error checking, so I wouldn’t suggest using it without understanding and modifying it to your needs.

A Python backup script

I have written a Python script that I use on my own servers. You can specify several backups to make, and options for each backup. Two backup modes are supported : rsync and mysql, which uses mysqldump to dump the database, then transfer it with SSH.

You can see or download the script here.

Again, I recommend reading the script to understand it before using it. That’s never good to use something you don’t understand! Especially in this case as the script may not be full-proof or completely adapted to your situation.

For more information on rsync backups see : http://www.mikerubel.org/computers/rsync_snapshots/

These are problems I had with CentOS5 host/guests setup using LVM without a virtual disk, and without using the Redhat’s virt-manager tool, but of course it may also apply to other kinds of setup.

Error: (22, ‘Invalid argument’)

Possible causes :

• There is something wrong with the kernel referenced by the “kernel” parameter of the domU configuration file. Perhaps that’s not a valid kernel, or the file doesn’t exist.
• There is something wrong with the partitions referenced by the “disk” parameter of the domU configuration file. Perhaps the device it points to doesn’t exist, or it is not a valid device.

Error: destroyDevice() takes exactly 3 arguments (2 given)

Possible cause :

• The domU root partition is mounted. It must be unmounted before booting the domU.

switchroot: mount failed: No such file or directory

Possible cause :

• The kernel you’re trying to boot does not support booting as a Xen domU. If you try to boot the kernel-xen that comes with CentOS, it will NOT work (at least, it doesn’t work without using a virtual disk like the virt-manager tool does). You should try using the kernel that comes with the official Xen distribution. You can download the tarball here, and use the kernel located in the dist/install/boot directory.

No console, boot process seems to hang up after “Starting SSHD”

Possible cause :

• I have noticed that the images from jailtime.org have this problem, and images that you made yourself may also have this problem. You need to configure a console for Xen in the /etc/initab file of the domU. Add this line to the domU’s inittab file :

  co:2345:respawn:/sbin/mingetty console

Warning: unable to open an initial console.
Restarting system.

Possible cause :

• I had this problem when using the kernels from XenSource to boot a CentOS 5 image (this worked previously, but now fails). I fixed the problem by using the kernel that comes with CentOS 5.1 instead, and building an initrd for the Xen guest. For more information, see this comment.

Yum update problem

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=os error was
[Errno 4] IOError: <urlopen error (-3, 'Temporary failure in name resolution')>
Error: Cannot find a valid baseurl for repo: base

• You problably forgot to configure nameservers. Edit /etc/resolf.conf and add “nameserver” lines. For example :
  nameserver 4.2.2.1
  nameserver 4.2.2.2

Iptables problem

Trying to start iptables gives :

iptables v1.3.5: can't initialize iptables table `filter': iptables who? (do you need to insmod?)

Perhaps iptables or your kernel needs to be upgraded.

Trying to “modprobe iptables” also gives an error message :

FATAL: Could not load /lib/modules/2.6.18-53.1.6.el5xen/modules.dep: No such file or directory

Here what I did to fix the problem:

• yum install kernel-xen kernel-xen-devel
• /sbin/depmod

After that, /sbin/iptables -L now works correctly!

Setup the dom0 (host OS)

Of course, Xen should be installed on the host OS. I use CentOS 5, so I just selected Xen during the installation. It will install a xen kernel that you should use to boot the host OS (dom0). I won’t go into details here, because that’s really easy to do with CentOS or Redhat Enterprise Linux. With other distributions, you could have to install distribution-specific packages, or use the official Xen package from xen.org.

Setup the target partition or logical volume

You should create a filesystem for the “root” partition and the swap. You could use simple files, but you will have better performance using real partitions or LVM volumes. LVM volumes also has other advantages, like the ability to create snapshots for backing up data, and easy resizing.

The following commands will create a 5GB root logical volume (LV) and 1GB swap in the /dev/vg0 volume group (VG). For more information about LVM, search for LVM howto in a search engine.

# root
lvcreate -L 5000M -n mailroot /dev/vg0
mkfs.ext3 /dev/vg0/mailroot
# swap
lvcreate -L 1000M -n mailswap /dev/vg0
mkswap /dev/vg0/mailswap

You can then mount the root partition and copy the base system (either an image from jailtime or an image you made yourself) on it.

mkdir /mnt/mailroot
mount /dev/vg0/mailroot /mnt/mailroot
# if you image contents is located in /centos...
cp -R /centos/* /mnt/mailroot/

Don’t forget to unmount the root partition when you’re done! Xen will not boot the domain if the partition is already mounted.

Download a kernel for the domU

The kernel that we will need to boot the domU has be to located in the dom0.

You can use the standard xen kernel that comes with CentOS to do that (e.g. vmlinuz-2.6.18-53.1.13.el5xen) , but you’ll also need an initrd, or the kernel won’t boot. To make the initrd, use the following command :

/sbin/mkinitrd --with=xennet --preload=xenblk /boot/initrd-centos5-xen.img 2.6.18-53.1.13.el5xen

This makes an initrd image with the required modules to boot a domU. The last parameter is the version of your kernel (the one you will use to boot the domU). You can get this number by typing “uname -r” on the command line. This will result in a /boot/initrd-centos5-xen.img image file.

Note (2008-02-14) : in a previous version of this blog post, I recommended to use a kernel from the official Xen distribution at xen.org. It worked, but it doesn’t seem to work anymore.

Create the configuration file

The configuration of the Xen guest is controlled by a simple text file. Create it as /etc/xen/yourdomUname, and move (or symlink) it in /etc/xen/auto if you want to start it automatically on boot.

Most basic parameters in this file are easy to understand. You should make sure “kernel” points to the kernel you copied from the xen tarball. “memory” is the amount of RAM allocated to the guest. “name” will be the name of the guest that you will use when connecting to it or shutting it down using the “xm” command.

“vif” contains information about network interfaces. One important thing in that line is the MAC address. If you don’t specify it here, a random MAC will be assigned at each boot, and that may not give good results. Edit the last 3 numbers (put anything, it just has to be unique across your network).

Finally, ‘disk’ is the parameter that tells Xen what partitions to use and what device name it will assign them. The last line, ‘root’, will tell the kernel what is the root device.

kernel = "/boot/vmlinuz-2.6.18-53.1.13.el5xen"
ramdisk = "/boot/initrd-centos5-xen.img"
memory = 512
name = "mail"
vif = [ 'mac=00:16:3e:21:f1:31,bridge=xenbr0' ]
dhcp = "dhcp"
disk = ['phy:/dev/vg0/mailroot,sda1,w', 'phy:/dev/vg0/mailswap,sda2,w' ]
# The next line would be useful if you want to use an simple file instead of a partition/LV
#disk = ['file:/root/test.img,sda1,w', 'file:/root/centos.swap,sda2,w' ]
# We don't use pygrub, we boot the kernel directly from dom0
#bootloader="/usr/bin/pygrub"
root = "/dev/sda1 ro"

Boot the domain!

OK, you’re ready to boot the guest domain! Just issue the following command to “create” (which means boot, really) the domU.

xm create /etc/xen/YOUR_CONFIG_FILE -c

The -c parameter tells xm to connect to the domain’s console. You can disconnect from it by pressing CTRL+], and connect to it again with “xm connect NAME”.

If everything works right, you should see the login prompt appearing, and you will be ready to use the new guest domain!

Fix SSH

If you made the guest image yourself as I explained in my previous post (part 1), you need to create the random device to fix SSH (and probably other services that requires generating keys). Issue the following commands on the guest’s console :

/sbin/MAKEDEV generic
/etc/init.d/sshd start