Clarkson University Internet Teaching Lab
MP252 (1 Credit)
This semester I am enrolled in MP252 for one credit.
Mid Semester Status Report
So far this semester I have accomplished quite a bit. Most of my work has been with the Windows lab build and the networking in the labs. I have also done some minor work on the Ubuntu lab build and on other various things in the lab.
This semester I produced a new Windows build for most of the computers in the lab except for the VR-Alienware and the COSI only computers. This process was easier this semester since I kept a log of all the problems on the lab build in my email last semester. It was also easier because I had a ghost image that had a base Windows install with all of the programs I install from OIT on it which required very little updating since there were no newer versions of the programs. This process was also much simpler because Active Directory was already set up and I was not trying to get that to work at the same time as the lab build. So far the lab build has been stable with the exception of group policy changes I have made to make similar to the recommended Federal Desktop Core Configurations. Hopefully everything will remain stable and I will only have to ghost out an updated Windows build once before the end of the semester if needed.
This semester I also helped make numerous network changes. Early in the semester, Benedict and I finished wiring all of the free ports in the labs to be on the Clarkson network. I also made some changes to the internal network switches in the labs. One of these changes included removing the management link between the two switches and configuring them to be managed on the internal network rather than the server room network. The two switches can now be reached on any of the lab computers through https or ssh at 10.0.0.2 and 10.0.0.3 respectively. Another change that both Cyrus and I made was fixing the server room expansion vlan. This stopped working early last semester for unknown reasons but was fixed through removing the management link and removing one of the two network connections that were run from the Blade Center half of the server room network to the expansion vlan on the internal switches. Another network change that Cyrus and I made was getting righteous to serve DHCP on the internal network again.
One of my other projects this semester that has been on hold lately is the network monitoring server. Early in the semester I made some upgrades to Nagios and produced some new documentation for upgrading plugins and such for Nagios; however, I finally got Nagios working through SNMP rather than through addons and plugins. One issue I ran into though is getting SNMP to reject connections from everything but Nagios. Hopefully in the next few weeks I will get a chance to begin working on this again and get everything stable before the end of the semester.
There are other numerous things I did in the labs this semester so far. I have updated the CS Labs homepage as needed and am hoping to either create a new design for the ITL website or use the same design as the CS Labs homepage. I also moved what use to be the ITL build computer into the COSI / VR side of the lab to be used as a computer for the projector. I also updated the four machines that both Wenjin and Madhu used for over a year in the ITL to be used for classes again. I also moved around the VR lab computers once the second door to the COSI / VR lab was blocked off. At the end of last semester, I helped migrate the old mailing lists over the new mailing list server. I have also labeled lots of things in the labs, helped clean the lab when needed, moved the Ubuntu build over the VR XPS computers, have been attempting to set up Crux as a gateway server, and updated the grub configuration on all of lab machines to have a password.
End of Semester Status Report
This semester I accomplished quite a bit. Earlier in the semester, as noted in my mid-semester report, I created the new Windows lab build and changed a lot with our network setup. For the most part that is what I continued to do for the remainder of the semester.
Specific Project Updates
Nagios
Well I haven't really done much with Nagios since the beginning of the semester. I did however create a new VMware image to install the stable 3.0 release that came out in March and I have begun setting that up. One thing that I did recently discover was that while SNMP can make some management aspects easier in the sense that I can edit all the configuration files for monitoring machines, it requires me to develop some crazy snmp-walk commands to pull information like disk usage. Thus I will most likely continue to use the add-ons that I used in my release candidate installation of Nagios to monitor machines in the labs and use snmp when it doesn't require substantial amounts of testing and development. Hopefully within the first week or two after finals I will finish setting this up so that everyone can receive alerts over the summer about any issues in the labs. I also am planning to move this over to the new righteous (which I will talk about later) due to the ability to monitor things on all three networks in the labs.
Windows Build
So this semester the Windows Build has been very stable. It is probably the best one I have produced so far and it didn't require any updating during the semester; however, I did reghost it out to all the machines at one point so that it would be fresh for the rest of the semester. The bad part about that decision was that the networking in the labs was really screwed up that night thus I broke whole lab and had to stay up till 3 in the morning getting all the machines in the ITL back up and working.
Active Directory
This semester I have continued to refine Active Directory in the labs. I changed the group policy on the lab machines, Windows servers, and user group policy a few times during the semester to make them simpler. I also restructured everything in the domain so that it had a clear order and so group policy was applied correctly to everything. One new thing I did try was deploying printers with Active Directory. This turned out to be trivial with Server 2003 R2 since you can attach printers to a group policy; however, it took me a week or two to sort out the issue of it getting set as the default printer which may have been due to network issues in the labs at the time. A few other recent things I did was I created a network home folder for every user in the domain and I created a network drive called "Shared" which every user automatically gets in My Computer upon loggingn.
Networking / Righteous
So this semester we added some more roles to Righteous. One of them was having righteous take over DHCP once again on the internal network. Another was having it answer DNS requests on the Clarkson Network for the cslabs.clarkson.edu domain. This was fairly straight forward since I only made the changes for the DNS on Clarkson's network and it just required adding some additional configuration and modification of our hosts generation script. I also made some minor modifications to our physical machines on the server room network so that they used static IPs versus DHCP since this would allow all of our virtual machines to stay running even if righteous was down for some reason. This change also required some changes to /etc/hosts since many things on our servers use host names and it required changing some NFS mounting stuff on the blades. One other change I made was to statically configure the blades on the Clarkson network as well for future changes we might implement.
One major problem this semester was the internal lab network was very unstable. One of the biggest problems was DHCP. Sometime righteous would give you a lease in about 5 seconds like it should and other times it would not give you a lease at all or it would about a minute later after you requested one. This caused quite a few issues since the Ubuntu lab build required an NFS share to be mounted almost all the time, ghosting often failed as I noted above, and Active Directory propagation and login was often slower than it should have been.
In order to isolate the internal network problem and to better utilize some of our hardware, I chose to virtualize righteous using Xen. This process ended up being fairly straight forward since I had an early version of the Xen book to use as a reference and I used CentOS which allowed me to choose to install Xen upon installing the distro. The install and configuration went fairly well in exception to my stupidity of not checking the wiki and setting righteous to a static IP which also happened to be one of Jim's honeypots. Once this was done I proceeded to create the four new VMs which would act as our DHCP/DNS server and one of them as a new gateway server. My main reason for choosing 4 VMs versus 2 (1 for a gateway, and 1 for DHCP/DNS on all 3 networks) was so we could isolate things and isolate problems. That way if someone changed something on the server room DHCP/DNS server and it broke, then the internal network would remain working correctly as well as the DNS for the cslabs.clarkson.edu domain. In the end, I ended up discovering that the issue with our internal network was due to something wrong on the switches. I ended up resetting them and reconfiguring them only with the necessary settings needed. Now, everything works great and hopefully it will remain that way. The best thing now is that everyone no longer has direct access to one of our most critical machines in the labs. I have also posted all of my setup steps on the wiki and have linked to them below.
Various Others
I also did other various things in the lab this semester. Some of them included unpacking and setting up the new printer, moving the VR area around, running networking for ACR2 in the server room, helping determine the correct start up and shutdown procedures for the server room, and hooking up the new UPS that arrived a few weeks ago.
Modified Documention
- Nagios VM Description Page
- Monitor a Remove Linux Server with Nagios
- Nagios Setup
- Network Port Layout
- Righteous
New Documentation Produced
- Upgrade Nagios NRPE Addon
- Upgrade Nagios Plugins
- Internal DHCP/DNS Server
- Internal DHCP/DNS Server Setup Process
- Server Room DHCP/DNS Server
- Server Room DHCP/DNS Server Setup Process
- CS Labs DNS Server
- CS Labs DNS Server Setup Process
- Isengard Server
- Isengard Server Setup Process
- Righteous Setup Process