02/5/2007:

Final post for this semester (This year actually!). I threw my scripts in here. And here is a link to a miniscule xen bug I "fixed". See you in 8 months.

24/4/2007:

As usual I have neglected this page, and now create an excessive attempting to summarize the last month of work:

Let's start with the storage layout. Pretty Picture . Most of the physical device stuff (Raid layout) is explained in the previous post. We have RaidA which contains backups (we still need to create an automated backup script for important stuffs). RaidC is pretty much entirely mirror, served over NFS. RaidB is pretty much everything else, including our images' root file systems.

As we make the move to a virtualization infrastructure, network file systems become important, after a fair amount of research it was decided that the best choice for managing virtualized file systems was going to be a Network Block Device protocol. We started with AoE (ATA over Ethernet), which is a non-routable protocol (Network Layer of OSI model), and thus has significant speed advantages over iSCSI, which has the heavy overhead of the TCP stack (seriously, who want's to route a block device across the Internet?). But, when we started we did not have a separate SAN network (we were running everything over the CU network), AoE has no security (pretty much relies on the idea that you have an isolate SAN), and the AoE packages on Debian had no nice init.d scripts for managing multiple block devices. So iSCSI ensued (with password protection). Now, we do have an isolated SAN, and word on the street is the AoE package on Debian has been updated. Switching back to AoE would be highly recommended. Another reason for making this switch is that dynamic additions/removals of block devices with IETD (iSCSI enterprise target daemon) is a real pain. In a matter of fact, I never got it to work properly, so every time I want a new block device, I have to shutdown all the images, shutdown all the iSCSI targets, edit the ietd.conf, and then restart ietd, then bring everything back up. Not cool. With AoE, each block device has a separate process that manages it. Cool.

storage.cosi.clarkson.edu will eventually be a way of distributing/sharing our storage. Currently it is just serving SVN, but soon user accounts will be made available. With these user accounts, you will not be able to get a shell on image (that is not their purpose) but you will have a home directory that you can scp/ftp/sshfs to. I think the estimated quota limit was going to be 500MB per account.

A couple notes about Animal: Unfortunately since we don't have the hardware to set up clustered SAN, Animal (our SAN server) is our single point of failure for most services. A power loss or network interruption to Animal, would be somewhat equivalent to ripping out the hard drive cables serving the root file system for each of the Xen images. Note that Animal's root file system is mirrored on the storage units, so we are somewhat protected against disk failure. That being said, lets try to keep Animal alive, and on the network. Oh, and because Animal serves NBD to the images, it imposes a shutdown ordering: If you want to take animal down, first stop all the Xen images, then stop iSCSI/AoE, then you can mess with Animal.

Networking stuff: Pretty Picture We have three networks. Green is the OIT/CU [128.153.144/145.*], it provides Internet. Blue is our "internal" network, running on our super sweet new Linksys switches, it will be used primarily for ghosting. Originally the plan was for just these two networks, but while drawing up this picture, I decided to add the third (Recently it has gone from being a picture, to actually existing). The red network is our server room network, it isolates the SAN and SSH access. It also seemed like a good idea to not do ghosting on the same network as the SAN.

A couple things about the switch setup:
Pretty Picture. Ports 21-24 on the top Linksys switch, are connected to 21-24 on the bottom switch. They are configured as a LAG (Link Aggregation), thus allowing 4 separate 1gbit/s connections between the top and bottom switch ports. On the bottom switch, ports 37-48 are a separate vLAN that extends the server room network, which originates at the second Blade Center switch. Ports 37 and 38 are connected to ports 5 and 6 on Blade Center switch 2, and are configured as a LAG. Port 48 on the top Linksys switch is its own vLAN, and is connected to 48 on the bottom switch (extending the server room network), and is used so that the tops switches IP (for management) can sit on the server room network).

So now all our services are virtualized. This means COSI (as in cosi.clarkson.edu, and planetcosi) got turned into an image. This left Righteous open for use, which worked out perfectly because I wanted a physical box for the DHCP server (and gateway for server room network only hosts such as Animal, and DNS for .sr.cs.clarkson.edu stuff). I was originally going to make the DHCP server a Xen guest, but this causes an issue when you are starting dom0 networking with DHCP, and THEN starting the DHCP server as a guest. The requirements for this physical box were that it had redundant disks (Righteous has SCSI card, and we had a Winchester RAID lying around) , and three network cards (Righteous has two built-in, and I ordered a third that will pop into the PCI-Express slot). Speaking of PCI-Express, after opening Righteous I was annoyed to find that we actually had a computer with a PCI-Express slot. Had I known this earlier, we could be running our storage off a PCI-Express controller card, which would probably been oodles faster. Oh well. Other good news is that righteous has a PCI-X slot as well. This means that if Animal ever bites the dust, you "should" be able to just pull the controller card out throw it in Righteous, tell the BIOS to boot of that card, and then you have effectively turned Righteous into Animal! But we will just hope it never comes to this.

Thats all I can think of now... more later.

12/3/2007:

It's midterm time, I have not updated this in a while, so I may have forgotten some of the random things I have been doing around the labs, but here is what I can remember:

-Setup networking on the blades (More of this to come once the internal network has arrived)
-Produced a "production" worthy blade for xen images. Currently this blade hosts various neat images such as cluster,cosi-svn,comm,mirror, and it will temporarily host the cosi website as we transition it to a xen image. Todd is donating me blade 1 so I can create a second such production blade. (Live migration coming soon to a blade center near you!)
-Provided much xen-image-setup/council/moral-support/you-did-that-wrong-do-it-again to Matt and Jacob on their subversion and mirror projects respectively.
-Dissected an IRC botnet perl script that was caught in our sweet (no pun intended) honey pot. I sent out an email explaining it's contents. For the most part the bot is much like any other botnet bot (provides shell execution, tcp/udp flooding etc...). The neat feature of this things was that it was using google/yahoo to search for vulnerable websites, and in effect proliferating itself almost auto-magically. I do not think the contents of the script are appropriate for web posting, so you will just have to take my word on this one. After noticing my interest in the subject Michael sent me a couple binary virii he had caught, I disassembled them and messed with them a bit. One takes advantage of an old bug in 2.4 kernel [do_brk() function], the other was something related to modprobe.
-Got the storage situation straightened out, this involved learning a bit about the current SCSI hardware. Now that things are running smoothly, I am documenting what I have done, so I no longer have to be responsible for it :P . For archival purposes I will mention some tidbits here:

-We have 2 IBM EXP400 storage units. They each contain fourteen 300GB Ultra-320 SCSI hard drives and an "adapter" on a single SCSI bus, that makes 15 devices. A Single Ultra320 SCSI bus supports 16 devices (the adapter in animal takes the 16th spot). Thus I requested we order a dual channel Ultra320 card that support RAID5. And now we own a LSI Logic
-The current RAID setup is as follows; CH0 DR1&2 are mirrored (RAID 1, 300GB) and contain Animals root file system (Animals internal HD is not being used). CH0 DR3-10 is raidA (RAID 5, 2.1TB) and will probably contain no network exported files, only static backups (for example a monthly backup of the website). CH0 DR11-14 + CH1 DR1-4 is raidB (RAID 5, 2.1TB) and will probably contain the file systems used for the image repositories. The verdict is not in yet on the preferred protocol (iSCSI,ATA over Ethernet, GNDB). CH1 DR5-12 is raidC (RAID 5, 2.1TB) and will contain NFS (or similar) exports for production use. (Mirror and the website are being hosted off this raid). The last two drives on CH1 are fail over drives, if any of the arrays have a single drive failure, the rebuild process will start automatically with one of theese drives.

-Jim and I are planning to draft a letter to [Place Bureaucracy Here] so that we can get cs.clarkson.edu DNS requests forwarded to our DNS server (coming soon).
-I haven't done much with cluster, but now that storage is up, and NFS is running, I intend too.

1/27/2007:

I am back! I got networking set up on the Blades, currently the first switch is attached to the Clarkson network. Once the internal network is up I think I will set up a VLAN on both switches so that the IOS/HMC IP's sit on the internal network, and all blades have access to both nets. Once storage is up I will attempt to tackle cluster/etherbooting again.

4/25/2006:

The semester roundup:
Project 1) Lab build etherbooting

-After learning about PXE booting and DHCP methods, I attempted to solve a lab administration problem. My goal was to automate the boot process of the lab, so that a specified OS, or third party image (ghost) would booted when computers were restarted.
-Deliverables: Grub Patch, Scripts and Config Files, XEN image (itldhcp.sclab.clarkson.edu) [A README from the XEN image]
-Learned: Intricacies of PXE booting and DHCP configuration, Refined C skills, Created my first XEN image, refined bash scripting skills
-Status: Completed successfully. The XEN image now controls the boot sequence of ITL machines.

-Preformed a significant amount of research into bleeding edge cluster, queuing, and cluster administration technologies.
-Met with various professors (Fowler, Hesse.) to get an idea of their needs.
-Deliverables: MPIqd (A backend queuing system for LAM/MPI distributed jobs and cluster management), XEN image (cluster.sclab.clarkson.edu, a README)
-Learned: Tons about clustering technologies, refined java skills (first time seriously using threads), refined MySQL skills (first serious SQL project), and refined XEN skills.
-Status: Finished, somewhat. Held together with duct tape and popsicle sticks. Needs significant improvement. I will be gone next semester, so I will pass the code off to Jeff. I intend to continue on this project upon my return to Clarkson.

4/16/2006:

Code, lots of it. MPIqd (MPI Queue Daemon) is roughly 1000 lines by now. Its a nifty little sucker. Jeff's frontend downloads source code (jobs) and compiles them, and places relevent info in a SQL database. MPIqd queries this database, finds active lab machines (nodes) runs lamboot on them, passes them the job, starts the job, generates statistics, cleans the job, and repeats (And a whole lot more than I don't care to mention). Sometime next week me and Jeff will set up an NFS server for the source files, and once then next COSI lab build gets ghosted we should be in buisness. The code is still sitting on my repos (COSI subversion server bit the dust?), but I will at least hand the code of to Jeff at the end of the semester.

4/6/2006:

Not much to report. I have been attending honey-pot meetings occasionally. Helped solve some snort issues, hopefully we will have a working VM honey-pot soon. I have also started writing the back-end for the COSI cluster. This is a challenging task, seeing as I have never written a daemon before (Fear of Threads). No guarantees as to whether I will finish it this semester, but at least I will leave behind a framework. The daemon will be written in JAVA (Simple threads, robust SQL libraries, and most importantly..) using the jakarta-commons-daemon package. Once I have decent framework I will move my subversion repos to the COSI repos. Out.

3/20/2006:

Yarr. It has been many moons since I updated this, so here we go:
Before break, COSI/ITL decided it was high time this university had a cluster. And much reading/research ensued. Personally I learned a ton about some bleeding edge cluster technologies. At a COSI meeting I volunteered to head the cluster project, but since then it has left my hands, and a cluster manifested itself nearly overnight (props to McCabe). It's not the coolest (feature-full) of clusters (Really it's just mpi on all the linux partitions), but it fulfills our basic requirements for the time being. (I am still waiting for Condor to become friendly with the linux kernel). So the next steps should involve informing professors of the existence of this cluster, creating a tard-proof front-end for queuing, and maybe even learning some MPI ourselves. (Because we have so-many computationally hard problems to solve :P). More later.

2/22/2006:

Lab etherbooting going swimmingly. I refined the dhcp server, so it now runs a few scripts, and generates the neaded DHCP/grub config files.
This simplifies the process of adding new boot configurations and changing the grub config.
Stay tuned... I have ideas about a supercomputer boot configuration. (insert evil laugh)

2/12/2006:

Todd gave me access to one of the xen machines sitting in the ITL. After some fidgeting i got the DHCP server set up on an image. The image has 2 virtual NIC's (One sits on a Clarkson subnet for ssh connections, the other for the random subnet that bootclients are assigned to [TFTP serves of this NIC]). A quick dhcp init.d script edit now lets lab maintainers change the default boot on lab machines via a single command ("/etc/init.d/dhcp3-server restart preferedOShere").
I also submitted my patch to GNU GRUB.

2/1/2006:

I gave a brief demo at the ITL meeting on etherbooting. Implementing PXE booting techniques will help with lab maintenance (udpcast a partition via pxe).
Part of getting this to work smoothly involved patching GNU GRUB.

11/9/2005:

A brief overview of Sangoma WAN technologies and a look at some traces

11/7/2005:

I have been having troubles with the Sangoma source, but in the mean time, i have written a patch for Ethereal, that supports reading the text output of Sangoma traces. Screenshots here.

9/12/2005:

This is a test of our broadcasting systems.

The first project I intend to work on this semester will involve changing Sangoma's WAN card tracing utilities to support pcap format.

Stay tuned for more.