This semester, I'm teaching my class on System Administration / Internet Operations once again.
The syllabus and all course materials are available here:
https://stevens.netmeister.org/615/
All videos for the lectures and exercises are public and available for free on YouTube:
https://www.youtube.com/c/cs615asa/videos
If you want to follow along, I'll be posting lecture videos and related links in this thread throughout the semester.
System Administration, Week 1: Introduction
In this video, we cover a number of administrative issues relating to our course: we discuss why and how System Administration is covered in an academic Computer Science curriculum and outline the course syllabus.
System Administration, Week 1: The Job of a System Administrator
In this video, we try to capture the job of a System Administrator. We show what things SysAdmins may encounter in their day to day routine, ranging from blade servers and routers to cable ties and power tools and everything in between. As we try to define the job, we find out it's not quite that easy...
It's duct tape and WD40 all the way down.
System Administration, Week 1: Core Principles
In this video, we present a few core principles that will guide us throughout the semester: Scalability, Security, and Simplicity. We'll also get to know a few basic "laws", well known by any System Administrator. If you're wondering what all this has to do with Legos, please tune in...
System Administration, Week 1: UNIX History
We're borrowing this video from our "Advanced Programming in the UNIX Environment" class to give a brief summary of the history of the UNIX family of operating systems.
System Administration, Week 1: Warming up to EC2
In this short video, we prepare for our first homework assignment and demonstrate how to launch a #NetBSD instance in AWS EC2.
https://www.youtube.com/watch?v=cA_pgRH0IDw
Note: the AMI in the video is outdated; I have up to date images listed here:
https://stevens.netmeister.org/615/netbsd-amis.html
Or you can create your own:
https://www.netmeister.org/blog/creating-netbsd-ec2-amis.html
System Administration, Week 1: AWS Aliases
System Administrators are notoriously lazy, and AWS commands a notoriously lengthy to type. In this video, we demonstrate the use of shell aliases and functions to save ourselves some typing whenever we run AWS EC2 commands.
The aliases and shell functions we use are available here:
https://github.com/jschauma/cloud-functions/blob/main/awsfuncs
System Administration, Week 1: Warmup Exercise 1 - No Space Left On Device
In this video, we try to find out what happens when we run out of disk space as well as how the system behaves when use up all inodes. This is intended as a warmup exercise for our week 2 topic, introducing the concept of disk storage and filesystem behavior.
System Administration, Week 2: Storage Models and Disks
In this video, we'll introduce the larger topic of filesystems and storage. In particular, we'll discuss the conceptual storage models, such as Direct Attached Storage (DAS), Network Attached Storage (NAS), Storage Area Networks (SANs), and Cloud Storage.
System Administration, Week 2: Devices and Interfaces
This segment discusses common storage device interfaces, including SCSI, ATA, SSD, Fibre Channel, and hinting at storage configurations like JBOD and RAID, which we'll get back to in the next video. At this point, it feels a bit dated, and I may skip it going forward and perhaps expand more on enterprise storage, but then again, it's only 10 minutes of your time.
System Administration, Week 2: Storage Virtualization
In this video, we cover the concept of storage virtualization -- combining individual disks into larger storage pools and utilizing resources from such a pool. This includes a discussion of RAID and some of the different supported levels as well as Logical Volume Management (LVM). We further illustrate some of these properties by example of ZFS.
System Administration, Week 2: Physical Disk Structure
We'll take a quick look at what a hard disk drive actually looks like. This helps us understand addressing schemes such as CHS and LBA, what physical aspects affect hard disk performance, as well as partitioning requirements. While a lot of this is tied to old magentic-spinning-platters drives, it explains a lot of assumptions partitions and file systems make even if using SSDs.
System Administration, Week 2: Partitions
In this video, we talk about how to divide a single disk -- physical or virtual -- and how the partitions relate to the physical structure of the disk. We show examples partitioning disks on NetBSD, OmniOS, and Linux using the disklabel, fdisk, and format tools.
System Administration, Week 2: Moving EC2 Volumes
We've talked about EC2 Elastic Block Storage volumes, and how we can treat them as if they were hard drives plugged into an instance. In this video, we run through one of our recommended exercises for Week 2 and show how to move an EBS volume across instances and operating systems from a NetBSD EC2 instance to one running Ubuntu Linux.
System Administration, Week 3: The Boot Process & the MBR
In this video, we discuss the boot process on a high level as well as take a fairly detailed look at the MBR. We'll create a suitable NetBSD BIOS partition by hand, utilizing the dd(1) command because using fdisk(8) would be just too easy. In the process, we learn a fair bit about the structure of the boot sector.
System Administration: Week 3: File systems
In this video, we pretend to be a file system, trying to store all our cat photos in a reasonable manner on a raw disk. By manually writing data and metadata, we begin to understand what a file system has to do. We also show how the tar(1) utility creates output that very much resembles a filesystem format.
System Administration: Week 3: Files go hier(7)
In this video, we're wrapping up our discussion of filesystems and partitions with a look at file types and partitions and filesystems mounted by default on #NetBSD, #FreeBSD, #OmniOS, and Fedora Linux. We close with a look at the filesystem hierarchy as defined in the hier(7) manual page.
System Administration: Week 3: Resizing a file system
In these two videos, we show how to resize an existing filesystem. First on #NetBSD using the resize_ffs(8) tool, where we first increase the size of a 512MB partition to 1GB, then shrink it down to 256MB. Next we repeat the same exercise on #Debian Linux, using the resize2fs(8) tool.
System Administration: Week 4: Types of Software
With this video, we begin our Week 04 topic of "software": what types of software there are, how they fit together, how to install software, and how to manage dependencies. We try to draw a terrible analogy to - what else - cars, and quickly realize that the distinctions between firmware, operating system, system software, add-on software are difficult to make.
System Administration: Week 4: OS Installation
In this video, we perform a step-by-step manual installation of #NetBSD onto a virtual machine to illustrate the details of the process, including partitioning, boot loader installation, OS set extraction etc.
We also discuss planning of the OS installation by looking at data classification into shareable/non-shareable and static/variable data and think about how to scale this process.
System Administration: Week 4: Package Management
In this video, we continue our discussion of the difference and relationship between the operating system and so-called "add-on software". We conclude that in order to install and maintain all such software, we want to use a package manager, and illustrate common features by example of the 'dpkg', 'rpm', and #NetBSD's #pkgsrc tools.
System Administration: Week 4: Package Management Pitfalls
In this video, we discuss some of the problems with package managers, native language packaging solutions, and the implications of their use on dependency resolution, package integrity, and trust. We revisit "left-pad" and "dependency confusion" to illustrate some of these problems.
System Administration: Week 5: Networking I: Layers
In this video, we begin our longer discussion on the topic of "networking". We're using tcpdump(8) to capture a single TCP SYN packet and start looking at the MAC and IP information, teasing out each individual byte. And don't worry, we dunk on the OSI stack model as needed and popular these days.
System Administration: Week 5: Networking I: IPv4 Basics & CIDR subnetting
In this video, we cover the basics of the 32-bit IPv4 address and how we organize networks using Classless Inter-Domain Routing or CIDR subnetting. (Don't worry, we'll get to #IPv6 in the next video.)
System Administration: Week 5: Networking I: IPv6 Basics
In this video, we get familiar with our Big Hero IPv6, looking at the structure of the IPv6 header and IPv6 address representations.
Since we're using AWS #EC2, remember that we have instructions for how to set up a dual-stack VPC and subnet to launch your instances in:
System Administration: Week 5: Networking I: IP Allocation & IPv4 Exhaustion
Mommy, where do IP addresses come from? In this video, we discuss how IANA allocates IP addresses to the Regional Internet Registries and try to illustrate just how large the #IPv6 address space is.
System Administration: Week 5: Networking I:The Physical Internet
In this video, we look at the physical structure of the internet, with a focus on submarine internet communications cables. Jumping from the bottom of the OSI stack all the way to Layer 9 ("political"), we then discuss how different countries use their political power to enforce internet blocks on their citizens, leading us to warrantless wiretapping in AT&T's room 641A.
System Administration: Week 5: Networking I: A Network of Networks
In this video, we look at how independent networks connect to one another, how Autonomous Systems numbers allow us to identify network operators, and how peering between independent ASs works.
System Administration: Week 6: Networking II: A Simple Request
In this video, we trace a simple HTTP request made via telnet to find out just how exactly our application knows how to connect to the remote server. In the process we learn about the ktrace(1) utility, as well as the nsswitch.conf(5), hosts(5), and resolv.conf(5) configuration files.
System Administration: Week 6: Networking II: ARP and NDP
In this video, we illustrate the functionality of the Address Resolution Protocol (ARP) and it's IPv6 equivalent, the Neighbor Discovery Protocol (NDP).
System Administration: Week 6: Networking II: ICMP
In this video, we demonstrate the use of the Internet Control Message Protocol or ICMP by tracing and analyzing ping(1) and traceroute(1) invocations.
And in case you're wondering: http://shouldiblockicmp.com
System Administration: Week 7: DNS, Part I
In this video, we are beginning our discussion of the #DNS. We go back to the early days of the internet when copying /etc/hosts from system to system was the way to resolve hosts...
(Hosts file from 1983: https://rscott.org/OldInternetFiles/hosts.19831104.txt)
...and we cover the structure of the domain name space and the creation of the top-level domains.
(Second-level domain inventory from 1987: https://rscott.org/OldInternetFiles/domain-info.19871215.txt)
System Administration: Week 7: DNS, Part II
In this video, we dissect DNS lookups performed on our EC2 instance, then discuss just how a caching resolver performs the lookup, moving from "magic happens here" to the below visualization.
System Administration: Week 7: DNS, Part III
In this video, we try to wrap up our discussion of the Domain Name System by addressing the nature of the root nameservers, looking at various different resource record types, observing reverse lookups, and thinking about how we can have assurance of authenticity and integrity of the DNS results returned to us via DNSSEC.
System Administration: Week 7: DNS & HTTP
We're moving from the DNS on to HTTP. Sorry, no videos for this subtopic, but here are the slides from our last class. We discuss how to get your domain registered and then move on to summarize briefly HTTP the protocol and look at what CDNs do.
System Administration: Week 8: HTTPS
We sketched out the basics of HTTP in our last class, so necessarily we'll have to cover #TLS now. These slides illustrate the benefits (transport encryption, endpoint authentication) as well as some of the pitfalls (complexity, cert revocation, endpoint "authentication").
System Administration: Week 8: E-Mail, Part I
In this video, we begin our discussion of E-Mail by looking at the components of the larger mail system (the Mail User Agent, the Mail Transfer Agent, the Mail Delivery Agent, an Access Agent), observing the packets involved in a simple SMTP exchange, and track an email from one system to the other.
System Administration: Week 8: E-Mail, Part II
In this video, we observe the incoming mail on our MTA, look at how STARTTLS can help protect information in transit, how MTA-STS can help defeat a MitM performing a STARTTLS-stripping attack, and how DANE can be used to verify the authenticity of the mail server's certificate.
System Administration: Week 8: E-Mail, Part III
In this video, we look at ways to combat Spam. In the process, we learn about email headers, the Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication, Reporting and Conformance (DMARC). SMTP doesn't seem quite so simple any more...
yeah. the sad thing is also that the *most* likely to be "correct" for SPF/DKIM/DMARC/rDNS are the bigger spammers. SMB are least likely.
then you get into choices in RBLs, having to have logins established to get blocks removed for the biggies like google/outlook and check for your spam rating.
definitely not impossible but also not for the faint of heart.
yeah... BGP is not for the faint of heart...
it's kind of like politics. you can claim anything you want and more careful folks have to do research or explicitly void your claims. sadly, in BGP and politics, way too many folks just blindly believe, then are surprised at the consequences. :)
i don't mind in a networking course (i mean i do, but that's another story), but in a security course, where i have two or three lectures to explain how networks work, or jan's systems course, BGP is just a bridge too far
yup. it's definitely a rabbit hole if that's not what you're there for.
even when i was doing a 5 day intensive DNS class, i would figure 30 minutes to try to explain enough of routing and anycast to explain why it applies to DNS.
suspect folks teaching CDN have to take a similar hit.
Jan Schaumann
Paul_IPv6
S.P.Zeidler
Daniel
shx
peter honeyman