Zach’s WordPress Blog
(Static COSI Archive)

I thought as a sort of semester wrap-up, I’d post the current status of my projects and my plans/to-do list.

Current Status:

• The production blades seem to be up and running stably with the bugs worked out.
• Animal disk I/O is still running slowly.
• Righteous seems to still be stable.
• The wiki seems to have everything worked out except for some last few policy decisions.

Future Plans/To-do list:

• Increase size of the cluster image
• Troubleshoot Animal’s disk I/O issues
• Roll out the ldap image
• Start work on a Google Android project
• Refine documentation (e.g. AoE Setup Process, add ip/mac/distro to vm template)
• Clean up scripts on Righteous and Animal (specifically with regards to Error Checking + Validation)

I thought I’d make some notes of some things I’ve changed/done over the past day or so before I forget to write them down somewhere.

• Removing the network udev rule on several virtual machines (cluster, time, xen) experiencing networking issues
• Updating the sudoers file on the primary machines (righteous, animal, hydrogen, helium, lithium) (no changes to people, just to options, such as enabling logging)
• Updating the banner text on hydrogen, helium, and lithium ( smslant font, http://network-science.de/ascii/)
• Updated the wiki creation how-to
• Recreated comm as a fresh (64 bit) image
• Created an image for ldap
• Created an image for zimbra

Because Jeremy was kind enough to put together a 64-bit blade build, I was able to get all production blades (formerly 0001-0011, now hydrogen-lithium) running 64-bit installations, allowing for 64-bit domUs, which should increase overall performance (yay!).

Because backups are always nice, I also backed up the initial installation as a tar file (with setup instructions in the tar in /root/scrips/afterclone.txt). The backup is on raidA in the appropriately named location.

Someone mentioned I should probably make a post about the progress I made last workday, so I suppose I will.

• Lithium imaged with Berylium (and a script to back up (for dom0s) and instructions what to change after restoring the backup)
• Production Services to Lithium (except mirror and cosi)
• COSI image resized
• Old Nagios image renamed and New (64 bit) image creates
• Modification of Animal’s scripts to allow cluster to just be on the external network
• Updates to the wiki (updated creation instructions, added resize & rename instructions, updated running status on the images that changed dom0)
• Taking steps to quantify the I/O issues on animal (ran some basic tests, will be running iozone in the next few days)
• Fixing an issue involving things not being able to access /dev/null on etch-based images

It looks like there seems to be a fairly consistent problem on our virtual machines running debian etch. Jeremy noticed it on cluster, then I noticed the same issue on time and docs. The symptom: /dev/null and /dev/false aren’t
accessible to non-root users and not having udev installed. The fix: installing udev. I’m not exactly sure what the problem is, but it certainly seems to cause various programs (e.g. bash) to throw a “few” errors about not being able to write to /dev/null.

Because of the recent issues with 0001 and 0010 and the fact that the only proposed fix was to update the image (which would involve recompiling the kernel), it was decided that it would be easier (and make more sense) to speed along the development of a 64-bit dom0 (which was already in development on berylium [formerly 0100] by Jeremy), and get it to the point where it could be copied onto other blades to get everything back up (instead of fixing the current dom0s and switching to 64-bit in a week or two). A “rough draft” has been put on lithium [formerly 0011], and most of the VMs are up and running there. The game plan is as follows:

• Get berylium cleaned up and everything installed and working (e.g. nagios plugin and live migration setup)
• Boot into a live cd and tar it up (tar -cvvpf backups/tarnamehere.tar . –exclude=backups/* –exclude=dev/* –exclude=media/* –exclude=mnt/* –exclude=proc/* –exclude=tmp/* –exclude=xen/*)
• Boot 0001 and 0010 into live cds and untar it there
  • Change dhclient.conf
  • Change hostname
  • Change hosts
  • Change sshd_config
  • Change motd
  • Change fstab UUIDs
    Change macid in network-bladecenter-bridges
  • Remove udev rules for networking
• Reboot the blades (which would, at that point, be hydrogen and helium)
• Hope everything works (troubleshoot as necessary)

Because the blades now (or will shortly) have 64-bit support, I changed the name of the current Nagios image to nagios32 and created a new 64-bit nagios image for Matt to move stuff over to. The rename process is kind of long (not nearly as bad as the resize process), so I thought I’d document it here (although it’s already on the wiki).

• Shutdown the image
• Kill the instance of vblade for the image
• Move the physical file
• Mount the filesystem
• Edit the config files
  • /etc/hostname
  • /etc/dhcp3/dhclient.conf
  • /etc/ssh/sshd_config
• Edit the masterlist on animal
• Start the vblade daemon
• Edit the hostsfile on righteous

The COSI image ran into an interesting issue; it ran out of space. For future reference, the resize process is as follows:

• Shutdown Image
• Create Backup Copy (cp /mnt/raidB/xenlib/images/imagename.disk /mnt/raidB/xenlib/temp/)
• Turn Image back on if desired
• Create image of new size
  • “dd if=/dev/zero of=/mnt/raidB/xenlib/temp/new_imagename.disk bs=1024k count=1 seek=X”
• Partition the new image, mount the images, and copy the files, preserving permissions
  • “losetup -f” to get next open loopback device. (the following commands assume it’s loop0)
  • “losetup /dev/loop0 /mnt/raidB/xenlib/temp/new_imagename.disk”
  • “losetup /dev/loop1 /mnt/raidB/xenlib/temp/imagename.disk”
  • “fdisk /dev/loop0″ (Create swap and root partitions)
  • “kpartx -av /dev/loop0″
  • “kpartx -av /dev/loop1″
  • “mkswap /dev/mapper/loop0p1″ (assumes swap partition is 1)
  • “mkfs.ext3 /dev/mapper/loop0p2″ (assumes root partition is 2)
  • “mkdir /mnt/{old,new}”
  • “mount /dev/mapper/loop0p2 /mnt/new” (assumes root partition is 2)
  • “mount /dev/mapper/loop1p2 /mnt/old” (assumes root partition is 2)
  • “cp -pr /mnt/new/* /mnt/old”
  • “umount /mnt/new”
  • “umount /mnt/old”
  • “kpartx -d /dev/loop0″
  • “kpartx -d /dev/loop1″
  • “losetup -d /dev/loop0″
  • “losetup -d /dev/loop1″

  Shutdown the current image

• Shutdown the vblade export
• Put the new image in place (mv /mnt/raidB/xenlib/images/{,old}imagename.disk /mnt/raidB/xenlib/{temp/new_,images/}imagename.disk)
• Start the vblade export
• Start the image
• If everything works, clean up after yourself (rm /mnt/raidB/xenlib/temp /mnt/raidB/xenlib/images/old_imagename.disk)

An issue I ran into recently (and was finally able to confirm was solved) is that when having a virtual machine on the COSI setup mount an AoE device, some special considerations need to be made. Because the modules directory is on a separate partition (shared between the images), modules listed in /etc/modules wont be added because the files haven’t yet been mounted. When the devices in fstab are finally mounted, the modules are now accessible, but because the AoE modules weren’t added, the mounting of those devices fails.

The solution? Add the modprobe commands, followed by “mount -a” to the rc.local (which is run after the modules directory has been mounted).

If anyone knows of a better solution, I’d certainly be interested, but this appears to work for now.

I recently tried to set up a fedora Xen image with little success. Maybe I’ll be able to find something when I look around again, but I thought I’d at least make a post so I remembered where I ran into issues.

In theory the setup is basically the same as creating a debian/ubuntu image except that you use rpmstrap (apt-get/yum install rpmstrap). In theory, you should then be able to run a command such as “rpmstrap –arch i386 fedora7 /mnt/tmp/ http://path.to/fedora/mirror”, but rpmstrap doesn’t seem to come with scripts for Fedora (it comes with quite a few, but sadly the Fedora ones were missing). I tried installing rpmstrap on a few platforms (debian etch, ubuntu fiesty, fedora 7) to see if the scripts were included on one or them, but I couldn’t find them if they were.

Update: It looks like rpmstrap supports up to Fedora Core 4, but since I’ve only used Core 6 and 7, I didn’t recognize the nicknames for the earlier versions. Sadly Core 4 is so out of date that there is no easy way to install it and upgrade from there, but now that I have something to work with, I might work with someone more familiar with Fedora and hammer out a script for 7 (I can’t be the only person wanting to use rpmstrap to install Fedora).

I’ve finally gotten around to making it happen. I thought it was going to be painful, but it actually wasn’t bad. It was just a matter modprobing aoe and dm-multipath (and adding them /etc/modules), changing the masterlist on animal, and exporting them using vblade daemon.

I only ran into one issue: in the process of changing over to AoE, I started two images (comm and storage) on a blade (0010) that only had 896MB of memory for the dom0, bringing the memory for the dom0 to 640MB (each image used 128MB). I then shut down the two images, and started a different image (cosi). I thought the other two images had fully shutdown, but apparently I jumped the gun. This lowered the memory for the dom0 to 128MB (cosi uses 512MB). You’d think xen would give the dom0 the extra 256MB of memory (from the shutdown of comm and storage), but it looks like it just sets it aside. Since I don’t know enough about xen to know if there was a command to get the memory back and the dom0 was running dangerously slowly, I decided that it might be a good idea to get the memory back before the network (and network mounted images for the vms) timed out, so I did the only thing I knew would work: a reboot. It’s probably not the best solution, but it worked.

Now everything is set (including AoE on 0001) and I was able to reproduce the ‘issue’ of xen taking memory away from the dom0 when a vm was started, but just setting it aside (for use by other images if you create them) when the vm is shut down. If anyone knows anything about getting the memory back (or why the memory can’t be returned), I’d be interested in hearing about it.

The old NFS-mounted files on mirror have been replaced with the up-to-date, shinny, AoE-mounted files (which supports files >2GB). The NFS mount has been removed from mirror’s fstab and unmounted. It’s been unexported. It hasn’t yet been deleted (that’ll probably happen in ~ a week). If anyone notices anything missing, let Chris (and me) know.

I made a few changes to make the documentation website a bit more usable. First, I made only the wiki subfolder block outside connections. Second, I added a more helpful 403 Forbidden document. I’m going to be looking into adding user-based authentication for outside the campus, but I’ve never combined ip-based authentication and user/password based authentication so it may be a bit buggy at first.

Todd requested an image for Xen related materials, so now we have one. Just a basic apache setup. Didn’t run into anything worth mentioning during the creation process except that the hostname seems to be having some issues resolving.

There’s been talk of setting up a wiki for internal COSI documentation for a while, but nothing really happened with it, so I decided to make it happen while I have some free time over break. It was just a matter of making an image, installing Apache, MySQL, and PHP. Downloading and installing MediaWiki, running through MediaWiki’s configuration program, and setting apache up with the following rule to limit it to only show up for Clarkson IPs:


Order deny,allow
Deny from all
Allow from 128.153.



My latest project is to get user storage up and running.

The first step was to get the image some space. A terabyte seemed like a nice round amount. It was just a matter of DDing an image file on animal, adding it to the rc.local for export (using vblade), configuring storage for AoE (adding aoe and dm-multipath to /etc/modules), mounting the image on storage, formating it, copying the contents of /home, and adding it to /etc/fstab.

The next step is to set up quota. The final step will be to develop a management interface for accounts and quotas.

In celebration of Software Freedom Day, the COSI held an open house style event. Student from Clarkson showed up, learned a little about the labs and about open source and had a little fun playing open source games such as Tremulous and looking at open source alternative to proprietary programs. Overall, it was a nice low-key celebration, although it’d be nice to see a bigger celebration next year.

As an act of support for the open source project, I set up a tremulous server in a VM on 0011. It seems to be working alright. If anyone is interested in maintaining it, let me know, and I’ll set you up with an account.

I was working on setting up a tremulous server and ran into an issue involving libc6. Apparently etch doesn’t include the right version, but fiesty does. To use debootstrap to put fiesty on the images, I needed to first install the scripts on the blade:
$ cd /tmp
$ wget http://mirrors.kernel.org/ubuntu/pool/main/d/debootstrap/debootstrap_0.3.3.2ubuntu3_all.deb
$ dpkg --extract debootstrap_0.3.3.2ubuntu3_all.deb debootstrap-feisty
$ sudo cp debootstrap-feisty/usr/lib/debootstrap/scripts/{warty,warty.buildd,hoary,hoary.buildd,breezy,dapper,edgy,edgy.fakechroot,feisty,feisty.fakechroot} /usr/lib/debootstrap/scripts/

Then, it was just a matter of running the command to debootstrap the image (debootstrap --arch i386 fiesty /mnt/tmp http://archive.ubuntu.com/ubuntu)

Thanks to our resident (well, long distance) detective, Cyrus, the mystery seems to have been solved.

It turns out that the issue on the blades was being caused by them getting dhcp from Clarkson. It’s been an issue all along, but we didn’t notice it because of the 5 minute lease time (it would only effect us from the time Clarkson’s lease renewed one a day-ish until the time that Righteous’ lease renewed a maximum of 5 minutes later). When we changed our lease to be longer than Clarkson’s, our access became the one that only applied for the period between renewals.

The solution? Adding a prepend statement to the dhclient.conf to make sure our dns (Righteous) is always listed first in the resolv.conf. It turns out that this issue will effect anything that uses dhcp to get an address from Clarkson (the reason we weren’t getting the error on the virtual machines was that they were either statically configured or, in the case of the new ones, networking hadn’t been restarted since modification of the configuration files).

So far, so good. If this wasn’t the only issue, we should know in the next 24 hours.

Edit (Saturday Noon) - Still no issues; looking good so far.

I (after consulting with Jeremy and Jake, as I’ve never before used perl) managed to update Cyrus’s script on animal to support images being exported over aoe (including exporting a script to be run on boot to export them). Adding images is now even easier!

I had some free time, so I added a few more images (no, not photographs). Notable ones include one image ready to go for nagios (which Matt has already gotten up and running, at least on some level) and one ready to go for time (maybe a new member interested in getting involved could tackle this one).

I ran into some issues with getting a steady stream of error messages about peth sent to tty0 on the production blades. This is yet another issue Cyrus already addressed (like the ssh issue on righteous). He even submitted a bug fix to xen for this one. It was just a matter of applying and configuring the bug fix and after everything was restarted, the error disappeared.

This did not fix the apparently unrelated issue of ssh on the blades “breaking”. The temporary fix for that is to log into the blade in the server room and restart networking. Still working on this one. Thanks to some help from Jeremy, we’ve determined that after a while, the server room network entry disappears from the resolve.conf file. This seems to coincide with ssh “breaking”

A few days ago, I took some steps to automate the ATA over Ethernet (AoE) startup process. I thought it’d be best to stick them in writing here:

• Adding the necessary modules /etc/modules on the three machines currently running AoE (0010, 0011, and mirror)
• Adding the necessary vblade exports to animal’s /etc/rc.local

What remains is to test the AoE environment to eliminate any bugs before moving the rest of the production images and machines over.

Ever wonder what would happen if some moron were to try to boot Animal with the power supplies on the RAID units on “off”?

Well, it turns out that the raid card on animal flips out and makes the most annoying noise possible. In this case, what it took to make it stop was to: start the raid units, restart animal, go into the setup menu, confirm the settings with 4 LVMs (1 small one, 3 ~6 TB), and the change each hard drive from “failed” to “online” (except the hotswap ones), exit the config, reboot animal, and viola.

It sounds easy, but with the beeping noise that makes it seem like the world is going to end if you don’t hurry, it’s a bit more difficult.

Righteous decided to stop accepting ssh. After a bit of poking around, we decided to run /root/scripts/iptables.sh, and it started working again. Yay for easy solutions.

The first step was to get another blade running xen. To do this, I took a backup of a production blade, unzipped it on blade 3, and changed the hostname information (in /etc/hostname, /etc/dhcp3/dhclient.conf, and /etc/init.d/sshd.conf), removed a pesky file causing some issues with the networking, set up aoe (add aoe and dm-multipath to /etc/modules), and was set for the dom0. At that point, I created a new xen image and config file (changing the drive locations, the name, and the mac addresses) on animal, exported them using vblade (vbladed 1 4 eth0 /mnt/raidB/xenlib/images/cluster.disk), and , on 0011, partitioned (using fdisk) and set it up with etch (using debootstrap). At that point, I mounted it, edited some config files (/etc/hostname, /etc/dhcp3/dhclient.conf, /etc/ssh/sshd_config, /etc/fstab, and /etc/networking/interfaces), and then started the image. I then ran into the error message “4gb seg fixup, process ___ (pid ___), cs:ip _______”, which popped up regularly for the majority of running processes. It was an easy fix; all I had to do was install libc6-xen, but it wasn’t very easy to find. I then worked with Ryan Lewis to set up the cluster account and key-based authentication to the lab machines. Ryan then configured the lab machines to allow ssh for only key-based authentication.