REVISION: 1
This document provides an introduction to networks and security, describes COSI's network infrastructure as of 2015-08-16, and proposes a new architecture centered around a firewall named "Ziltoid".
Security is important for any organization, and COSI is no exception. We have both physical and digital considerations, but here I will consider only computer security. While COSI does not have many secrets it needs to keep, securing our infrastructure has many benefits:
More professional appearance, especially to OIT and visiting recruiters.
Exposes members to the principles and practice of network security.
Protects our time investment by avoiding unintended modifications to our systems, which could cause painful rebuilds.
Gives us a system capable of handing out honeypots or other intentionally-vulnerable systems to students for security research without exposing our network to attackers.
Networks are built in a layered fashion, which each layer handling one particular aspect of getting data from one host to another. The bottom layers are said to "encapsulate" the data passed from the upper layers.
COSI uses 1000BASE-T (an electrical standard) over Category 5 cable (often just called "ethernet cable") to transport Ethernet traffic between hosts.
On top of Ethernet we use mostly the Internet Protocol, version 4.
When a host on an Ethernet network wants to send an IPv4 datagram to another host, it first consults the local Ethernet. If the host is not there, the routing table is consulted to see if the host knows where to send the packet. There is a "default route" to a gateway that will be used if no more specific route is found.
The gateway does the same thing, and has its own gateway. There's usually a route to the host you're looking for. There's a lot more complexity here, and in IPv6 things are both simpler and more complicated.
On top of the unreliable datagram-oriented IP, there is the Transmission Control Protocol (TCP), which provides a reliable stream-oriented interface. To allow multiple concurrent ongoing TCP connections between two hosts, a "port" is utilized. Every TCP packet (data encapsulated in the IPv4 datagram) specifies which port the datagram is for. The tuple of (local address, local port, remote address, remote port) uniquely determines a TCP connection.
The gateway is in a unique position to control access to the hosts that use it as a gateway. When used for access control, it is called a "screening router". The gateway can accept or reject IP datagrams on any basis it chooses.
First, let me define "authentication" and "authorization". Authentication is all about verifying identity. One traditional method for authenticating to a system is the username/password pair, with the password being a secret that only the entity who should be authenticating knows. There are plenty of other ways to do authentication, most involving cryptography. Authorization is all about determining whether an authenticated entity is "allowed" to perform a certain action.
No security policy can be evaluated without taking into account what attacks it is protecting against, and is being protected. There are three general properties worth protecting: confidentiality, integrity, and availability. Confidentiality means that only authorized entities can access certain information or systems. Integrity means that systems are only modified in authorized ways, and can not be tampered with. Availability means that systems are available for their intended use.
For COSI, confidentiality is usually not very important. The exception is mostly private keys and other such sensitive cryptographic information. Most of our systems have no secret data whatsoever. As mentioned in the intro, integrity and availability are critically important. These are the properties I will focus on.
There are a few general principles that guide secure systems. The most important is that of "least privilege", which ensures that any system is only able to access what it needs to fulfill its purpose. Enforcing least privilege requires participation at both the host and network level. "Defense in Depth" is another important concept. This is using multiple layers of protections to avoid the entire system falling if a single piece is exploited. As we'll see later, some of the tools for this can make certain classes of attacks practically impossible. Since we don't have the resources to carefully audit all of the software we use, this is an important technique.
this section needs to be fleshed out
Scans for insecure configurations of software (eg common usernames and passwords).
Automated exploitation of common software flaws to gain control over either the software or the whole system for possibly nefarious purposes. As an example, the Shellshock vulnerability was incredibly easy to exploit and would grant local code execution trivially.
Targeted, focused network attacks specifically on our infrastructure.
Threats 1 and 2 are the most likely to occur (our systems undergo automated scans constantly), and thankfully also the easiest to prevent. Threat 3 is rather unlikely, but still possible. We should strive to make our systems challenging for attackers to break into, without imposing undue burden on the labs.
Our security infrastructure is completely ad-hoc. All security is currently host based, with no network security in place (beyond what OIT provides, which as I understand it is minimal to nonexistent). All hosts on our network are globally routable, exposing them to the raw, unfettered wasteland that is the internet.
The rest of this document describes the new architecture and recommendations for improving security.
I (cmr) have been working on getting ansible up and working for giving "executable documentation" for how our systems are configured and set-up. This helps with lab continuity, but also provides a straightforward way for newcomers to learn how our systems are setup. It provides a framework for replicating and understanding our services for the purposes of audit and evaluating the impact of changes outside of production. Using ansible, all relevant host-based measures can be specified and tweaked in one place, and applied to all hosts equally, with minimal effort.
Compared to writing shell scripts or documenting every little step on the wiki, ansible is a huge time saver. Configuration management and automation is the principle domain of "developer operations" (DevOps) and having knowledge and experience with these tools is an excellent resume bullet point, and sure to impress recruiters.
We should set up automated security scans to be performed regularly, and fix any problems that arise. We should review mailing lists such as oss-security for recent vulnerabilities that may impact our particular configurations.
This would be during the main phase of the meeting, probably immediately after lightning talks, and should never exceed 5 minutes.
Automated systems can help a lot with this, but we still need to monitor the automated systems for any alarms they raise!
It's possible we could pony up the money (or find a volunteer) to do professional penetration testing on our networks. Honestly, though, we can probably do a lot of it ourselves. If a pentest came back with anything that isn't subtle as hell, this plan has failed.
This section describes some specific recommendations. There are roughly four categories of hosts: high-security, internal services, external services, and user. High-security hosts are probably only our authentication server (Talos, running LDAP/Kerberos etc) and the firewall (Ziltoid, described later). Internal services should only be accessible from inside the lab network, and have minimal host security requirements. External services require careful consideration to secure, though need not be as sandboxed as the high-security hosts. User hosts are the VMs we hand out like candy, and by default should be behind the firewall but untrusted (the antics of CS students are indistinguishable from malice). At request, we should be able to lift any and all restrictions in place, so as to not impede experimentation or utility.
If you don't know, Linux is an open source kernel, which is the bit of an operating system that implements privilege separation and (usually) communication with the hardware. All of our systems so far have been a Linux of some flavor, and our high-security hosts will be no exception. One reasonable alternative would be the OpenBSD project, which the patches we'll be applying are often inspired from. OpenSSH is also an OpenBSD project.
Grsecurity is a long-running project to harden the Linux kernel with exploit mitigation features. It is provided as a "patch" (list of which lines of code to change) to the source code. It also integrates PaX, which is also a patch). A quick overview of some of the features these patches provide:
It's important to note that none of these prevent vulnerabilities. Instead, they make successfully exploiting existing vulnerabilities much more difficult. This is the "defense in depth" principle in action. Unfortunately, Debian does not have an official package for a kernel patched with Grsecurity. We (I) will have to maintain our own kernel configuration as an ansible script. On the upside, we will benefit more from some of Grsecurity's structure randomization as none of our systems will share the same randomized structure layout, preventing a successful attack on one system from carrying over to another system.
Grsecurity includes a Mandatory Access-Control subsystem called RBAC. The classic Unix permission system is entirely filesystem-oriented and is called "discretionary" - it's up to root whether or not the permissions are respected, as root can read/write/etc every file on the system. Linux adds a number of tools for fine-grained permission handling, including a capability system, and namespacing. While these are fine tools for sandboxing applications, for enforcing system policy they are quite impotent.
RBAC allows configuring at a very fine-grained level exactly system resources what each process, user, etc can access. The Grsecurity wikibook has extensive documentation about configuring RBAC. It is quite easy, especially compared to creating policies for SELinux.
SELinux is only an access control system, so we would need to combine it with Grsecurity anyways for the benefits it provides. We'd also have to either import policies or write our own. SELinux is much more complex than RBAC, both to understand but also to deploy and write policies for. Note that it is possible to combine RBAC and SELinux, but that would largely be pointless.
The Secure Shell protocol is a widely implemented protocol for securely connecting to remote systems. SSH will be the only way for users to log in to our systems. It will be able to do authentication through Kerberos/LDAP. It is currently the simplest and most secure (and securely implemented) widely available protocol. The configuration we will use is currently on the wiki. The cryptographic primitives used is severely constrained to those that are simple to understand, implement, and are most secure.
OpenSSH can also be used as a proxy/tunnel for other IP traffic via SOCKS. This will be our "VPN", instead of the more traditional OpenVPN. We can add OpenVPN later if absolutely needed, but I suspect it will not be needed and omitting it reduces our attack surface. I also hold the TLS protocol in extremely low regard. It is overly complex, has insecure configurations, and even without having to depend on the Certificate Authority infrastructure, the certificates themselves are too complex. Additionally, OpenSSL itself which OpenVPN uses is not held in high esteem either.
Key management is the trickiest part of using SSH, and I don't expect to take any particular precautions for most user accounts beyond the basic security training given at the start of each semester. Administrator accounts, however, will be very tightly controlled. The most important account is the ansible account, and it will have its own key which is tightly controlled and rotated every semester.
Clang is a relatively new C compiler based on the excellent LLVM infrastructure. It is used by Sony, Apple, Google, IBM, NVIDIA, and others in their production compilers. It is also used as the compiler on FreeBSD. It has a number of "sanitizers" which can help harden userspace applications. The three most important for our purposes are the "address", "undefined", and "integer" sanitizers. The address sanitizer (ASAN) will abort a program if it access non-allocated memory, use-after-free, and double/invalid frees. This behavior is never exhibited in a correct program, and indicates either a bug or an active exploit. The undefined behavior sanitizer (UBSAN) will catch a laundry list of behavior that is considered "undefined" by the C or C++ standards. No correct program is allowed to cause undefined behavior, so if a program is killed by UBSAN it is because of a bug or an active exploit. The integer checker will check for integer overflow. While this is not always indicative of an exploit or a bug, it frequently is. As an example, the recent Stagefright vulnerability in Android is caused by integer overflows.
This hardening is the first line of defense against 0days.
Gentoo is essentially a meta-distro. Since we want to take advantage of compile-time hardening as much as possible, and the HSHs are severely stripped down compared to an average Linux install, it may make sense to use a very limited, custom Gentoo install. We'd use our own PID 1 instead of a bunch of scripts, etc. With ansible tying it all together, the amount of manual work compared to Debian is minimal.
Ziltoid the Omniscient is the name of the firewall. It will be our gateway and a screening router. It will control which hosts are allowed to communicate with which other hosts, and on what ports/protocols. The exact details are to be ironed out, but some general policies are outlined below.
The only HSHs we are likely to have are Talos (the authentication server, which has sensitive cryptographic data), Ziltoid (the firewall proposed later, directly exposed to the internet), and our VM hosts (which have a quite large attack surface, since they have to run untrusted code in the VMs, and accept untrusted network traffic to feed into the VMs). These will have full Grsecurity with the most restrictive RBAC policies possible. As Kerberos will sometimes be non-functional when Talos needs to be accessed, it will have its own set of SSH keys for management.
External services should ideally be in the same configuration as HSHs, but with more users able to log-in and use sudo. Additionally, the RBAC policies would only restrict the services that are accessible, and not everything on the system. The firewall should restrict incoming connections to only be allowed to these external services, and no others.
Internal services will be completely inaccessible outside the firewall. Ideally, they would be in the same configuration as external services besides this differences.
Users hosts are outside of our direct control. The default VM template we give out should have Grsecurity, but no RBAC. They should be completely behind the firewall, but users can reasonably request to be placed outside the firewall. At that point, the host is completely untrusted and cannot be let back inside the firewall without being rebuilt.
A honeypot would be a neat thing to have, and with a firewall in place able to do eg QoS, would be much less risky than in the past (especially for netsec students).
An intrusion detection system would be nice to get some insight into the attacks we receive.
Let's not get our shit owned this year, ok? We've had three notable incidents and one of them involved OIT telling us our systems were broke. That isn't ok. Let's do better.