COSI lab build/workshops

Purpose: Proof of concept, set up all lab machines to boot off of the network. This is acheived through using PXE (pixie) to pull the kernel and initramfs off of mirror. The arguments to the ramfs set up an NBD share drive as the filesystem for "/", and completes the boot using the NBD filesystem. The NBD server is set up "copy on write"(COW), so that any changes made to the image are stored in RAM (and a .diff file on the server), instead of written to the image. This allows us to have a build that refreshes on reboot, so the user is given root access (through sudo) at all times. Updating the lab build is as simple as booting one machine with a special image and making changes once.

Server (Tinkerbell)

Running Debian Wheezy, headless. (NOTE: Archlinux doesn't play nice with these particular SCSI drives! It's possible that the newer kernel is to blame, so be extremely careful when updating.).

NBD setup was detailed in a wiki page about Diskless Systems.

RAID 10 - 6x 300G 10K drives = ~840G of space and ~280MB/s read speed, this is more that enough for our purposes. Booting from RAID10 with GRUB2 was detailed in this article.

Persistent data on "/home/csguest/Documents", provided by an NFS share that gets mounted on boot. This is a temporary solution for persistant storage, as any user can see and delete any files on the share. This is also mountable read-only from anywhere on campus, or mounted read-write inside the COSI labs.

Set up three images to facilitate seamless updates: arch0, arch1, and arch-testing. At any one time, mirror will only point at either arch0 or arch1 for the recommended build, and arch-testing is a special NBD share that is not COW. Update process is as follows:

Copy CURRENT build image (arch0, arch1) to arch-testing:

root@tinkerbell ~$ cd /build/; cp --sparse=always {arch0.img, arch1.img} arch-testing.img
Copy CURRENT build initramfs files (arch0, arch1) to arch-testing:

root@mirror ~$ cd /storage/pxe/; cp {arch0, arch1}/* arch-testing/*
Boot a lab machine into Archlinux -> Arch Testing and apply desired changes. Suggestions:

csguest@cslabs ~$ yaourt -Syu [--noconfirm]
csguest@cslabs ~$ yaourt -Syu --devel --aur [--noconfirm]
csguest@cslabs ~$ yaourt -S <new_packages>
Reboot to confirm changes, and copy initramfs and vlinuz to mirror (NOTE: only needed after kernel update, but since this is Arch, it will most likely be a necessary step):

csguest@cslabs ~$ scp /boot/* <username>@mirror.cslabs
Log out of all users, switch to a virtual terminal, and clear caches and temp files. A script (/root/cleanup.sh) is provided to facilitate this, but make sure you read what it does before running it:

root@cslabs ~$ ./cleanup.sh; poweroff
Move arch-testing to whichever image is NOT currently in use (i.e. if arch0 was used in step 1, use arch1 in this step.):

root@tinkerbell ~$ cd /build/; mv arch-testing.img {arch0.img, arch1.img}
Prepare mirror files for switch by moving the ramdisks to the folder of the image used in step 6:

root@mirror ~$ mv initramfs-* vmlinuz-linux /storage/pxe/{arch0, arch1}

Update pxe cfg (/storage/pxe/boot-screens/arch.cfg) to point to new arch image:

label Arch_Build
	Menu LABEL Arch Build (Recommended)
	KERNEL arch0/vmlinuz-linux
	APPEND ip=dhcp nbd_host=128.153.145.69 nbd_port=6969 nbd_name=arch0 root=/dev/nbd0
	INITRD arch0/initramfs-linux.img

menu begin
	menu title Archlinux
	label Arch Testing
		Menu LABEL Arch Testing
		KERNEL arch-testing/vmlinuz-linux
		APPEND ip=dhcp nbd_host=128.153.145.69 nbd_port=6969 nbd_name=arch-testing root=/dev/nbd0
		INITRD arch-testing/initramfs-linux.img

Or simply just:

root@mirror ~$ sed -i -e 's/{arch0, arch1}/{arch1, arch0}/g' /storage/pxe/boot-screens/arch.cfg

Any machines that are restarted will now be using the new build, without restarting the NBD server so current sessions are kept alive!
Once the new build is working without issues, copy the new image file to a backup (becuase paranoia!):

root@tinkerbell ~$ cp --sparse=always /build/{arch0.img,arch1.img} /backup/current.img

Planned changes:

The .diff files are not removed when clients shut down, which is a known and confirmed bug reported here. While this is not expected to change immediately, a cronjob that removeds /build/*.diff every night (or hour, or minute) is an acceptable solution, and the bug report should be checked every so often for changes.
Set up overlayfs on storage for authenticated persistant storage.

Clients

Running ArchLinux - Standard build from the Beginner's guide.

An image is a sparse file with a working operating system on it. They can be grown, but not shrunk (without creating a different file, mounting both as loop devices, and copying all of the data over).

Any image can be added as a potential lab build, and can be chosen at boot (Debian-based systems, Arch-based systems, and potentially others that support NBD).

All changes are currently stored in RAM and are temporary, so CsGuest is given full passwordless sudo access. This allows anyone to experiment and learn with a lab build, something that is very helpful.

Planned changes:

Updates and new packages!
Persistant storage - utilize hard disk on the lab machines to mount on /storage. There will be symlinks from /home/csguest/* to places on /storage so that user data persists over reboot (but not configs). A startup script will mount the disk and verify it's integrity (creating folders if necessary), and if not /storage will still exist on the system (albeit not persistant) so the home directory is seamless.

Workshop (in progress)

Archlinux install workshop - base/base-devel to DE.

Tuesday, 4/23, 5:00 PM
Thursday, 4/25, 5:00 PM

Spring 2013 - Andrew Potter

COSI lab build/workshops