Ksplice is system for automatically patching a Linux kernel without reboots. Jeff Arnold '07, MEng '08 developed this as his master's thesis, and today released the system to the Linux Kernel Mailing List (LKML).

Ksplice requires nothing more than the currently running kernel source and the configuration settings, combined with a patch to the source code, to generate a kernel module which, when loaded, can be used to immediately patch vulnerabilities or introduce new behavior. As a maintainer of linux.mit.edu, the SIPB Linux dialup server, Jeff has in the past used Ksplice to keep this server up, which many people on campus rely on.

This really has the potential to revolutionize systems administration for high-reliability systems. But...don't take it from me. Take it from Ted Ts'o '90. Ted is an active kernel hacker who, among other things, developed the ext2 filesystem, the basis for the ext3 filesystem, which is one of the major filesystems used on Linux today.

Here's what Ted says in ZDNet's article:

Top kernel developer and Linux Foundation fellow Ted Ts’o said the Ksplice software is much needed by telecommunications providers and anyone who hates downtime. “It allows you to hot patch the Linux kernel with a security update without rebooting the computer. It’s a binary patch capability that is highly automated,” said Ts’o. “Users in the carrier grade linux space have been clamoring for this for a while. If you are a carrier in telephony and don’t want downtime, this stuff is pure gold.”

The best part? It doesn’t require any kernel modifications, Ts’o said.

(Here's the LKML posting: http://permalink.gmane.org/gmane.linux.kernel/669951)

Yes, the weeks before Spring Break tend to suck. Like, not just your average suck. They tend to really suck. And with that much suck, I should have spent all weekend tooling and doing nothing else.

But of course that's never how it works.

Last weekend, I took the Zone Cell Challenge.

Now, this is going to require a little background. Actually, there's quite a bit of background. Back In The Day, the MIT I/S Department (now IS&T) hired student developers to basically help build Athena. They were called the Watchmakers, based on a book by Niven & Pournell (there's a page that explains the metaphor). The Watchmakers worked in the Watchmaker Zone, and to help them test, debug, and generally try to break the technologies they were developing, I/S donated three servers, which formed the ZONE Kerberos realm and the zone AFS cell.

Ok...I guess I should take another step back and quickly explain Kerberos and AFS. Kerberos, named after the three-headed dog guard of Hades (usually seen as the Latin Cerberus), is a protocol for mutual authentication of a client and server through a trusted third party. Basically, you can prove who you are to other computers on the network, and your password never has to cross the network unencrypted. This works because there's a single machine that all other computers on the network trust, called the KDC. Kerberos is broken up into "realms," which represent distinct sets of users and servers. The normal realm here on campus is the ATHENA.MIT.EDU realm, but there are also the CSAIL.MIT.EDU, MEDIA-LAB.MIT.EDU, and NUMENOR.MIT.EDU realms.

AFS is the Andrew File System. Developed at Carnegie Mellon, AFS is a network accessible file system. It's a way of accessing your files from any computer on the Athena network. AFS is broken up into "cells," and there are many more AFS cells than there are Kerberos realms. Like Kerberos, there is an athena.mit.edu cell. There are also csail.mit.edu and numenor.mit.edu cells. However, there is additionally, for example, the sipb.mit.edu cell, run by the SIPB. This allows SIPB to maintain its own servers and software and allocate quotas without having to go through the Athena maintainers.

So...in addition to all of those cells and realms, there's the lesser known Zone Cell, made up of three servers (remember - they're for the Watchmakers): casio, seiko, and timex. If you look in the Zone Cell, you'll see a series of very explicit rules. I'll copy some of the better ones:

1. The Zone Cell is not for reliable data storage.
2. The Zone Cell is not for reliable data storage. (in case you missed it the first time)
9. ASO reserves the right to remove someone's zone cell bits. Reasons this may be done include: [...] gratuitously compromising the security of the zone cell or zone kerberos realm, or intentionally causing Rule 1, 2, or 10 to become relevant. [...]
10. The zone cell is not for reliable data storage.
11. The zone cell is not for reliable data access.

So...now that we've established the reliability of the Zone Cell, what is the Zone Cell Challenge?

The Zone Cell Challenge is an event that is usually for people interested in becoming SIPB AFS administrators. It's designed to help people become more familiar with how AFS and Kerberos work.

In the Zone Cell Challenge, you start with root access to one of the three Zone Cell servers, and you have to accomplish three things. First, you have to give yourself root access to the other two Zone Cell servers. Second, you have to make yourself an administrator of the Zone AFS Cell (which is different from having root access on the servers!). Finally, you have to make yourself an admin of the Zone Kerberos Realm, which gives you permission to change other people's passwords or create new accounts.

It actually turns out that none of these tasks are particularly challenging if you read the right documentation. And since I was interested in learning more about how to administer AFS, I signed up for the Challenge on Friday night. And at about 3:45 AM, I actually started trying to hack the Zone Cell in earnest.

The first part is easy, once you find the right command. It took me about 30 minutes to gain root access on all three servers.

And then, almost immediately after I had logged in to all three to make sure it worked, all of them froze. And they stopped pinging.

Well...crud. What did I screw up?

Well, fortunately, I didn't screw anything up. The Zone Cell servers had been moved to virtual machines, and all of the virtual machines on one server froze, including casio, seiko, and timex. Fortunately, there were other VMs on that machine that people cared about, and one of the Athena server maintainers brought the servers back up.

One task down, two to go.

Before about 6 AM I managed to finish the second task - making myself an administrator of the Kerberos realm. I went to sleep, and when I woke up, I spent a couple of hours on the last task. The last task was also a little entertaining. I had a theory of how to make myself an AFS administrator, but I was worried that I would screw up the AFS users database. So I asked on the zone-cell zephyr class (kind of like a chatroom)

Me: Ok. I think I know what I want to do [...], but I'd like to check and make sure I'm not going to screw it up before I try. Anyone around?

Someone from Athena server ops: I think you should just risk breaking it. At worse, you'll just have to fix it after the fact

Well...ok? I guess I'll just back up the files I'm modifying...

Anyway, I can't really say much more, because where's the fun in saying what I did? But in any case, hacking other people's servers with permission is a really fun way to spend a weekend.

But I guess I should go back to tooling now...

6.033 is Computer Systems Engineering. It has lots of different components to it - so far we've talked about the Therac-25, the X Windows System, multi-threading, and a couple of other things. It moves quickly through different topics, all of which are very interesting.

In particular, though, 6.033 has two design projects, where we're presented with a fairly real world problem, and we have to try to solve it. This year, we have to create a controller for a NAND flash device.

The controller sits between the filesystem and the physical device itself, and has to make the flash look like a traditional magnetic disk, in spite of the fact that flash memory is distinctly unlike a magnetic disk.

Here's the description of the problem: http://web.mit.edu/6.033/www/dp1/.

Feel free to submit solutions in the comments. My one recommendation is that you should make sure that you actually have enough RAM to store what you want. My friends and I spent multiple days coming up with and throwing out lots of different options, and none of us have a solution that we're truly satisfied with, so don't feel bad if you have a hard time coming up with something that works.

You can not solve this problem without making significant sacrifices in either the available storage space, the read and write performance, or the ability of the controller to balance wear over time.

I'll post my solution after I give you guys time to play with it. Also - feel free to let me know if you have any questions - I know it's kind of hard to follow at first. (And as usual, questions about other MIT-related things are also OK)

I keep a list in my head of what topics I want to blog about. But I think of it as a stack, as opposed to a queue - so I cover things that happened more recently first.

So obviously, I need to cover what I did over IAP, and then maybe I can make it as far back as last semester.

Fortunately for me (and those of you who want to hear about 6.111 and my other adventures in EE), I didn't do much this IAP, so I should be able to just knock this out and move on. Last IAP I spent almost all of the month working on 6.270. It was fun - the robots are really cool. But it was also stresful. I didn't get much sleep during the last week of IAP (or during the rest of IAP, for that matter).

So I decided that this year I was going to be on campus, but not have any formalized activities taking up all of IAP. I was going to rest and recuperate so that I could go into the spring term well rested and just generally enjoy myself.

I think I more or less accomplished that, with a few exceptions. So, "what I did for IAP" is a fairly short list.

One of the bigger things I did was teaching an introductory class in Python for SIPB. SIPB (the Student Information Processing Board) is a student group that focuses on improving computing, both in general and specifically at MIT. They're responsible for things like debathena, which is forming the basis for the new release of the Athena system, and scripts.mit.edu, a web hosting service for the MIT community.

Additionally, they run a series of IAP classes, covering everything from how to hack Facebook Javascript to the "Caffeinated Crash Courses," which attempt to cover everything you really need to know about a topic (usually a programming language) in 3 hours. They also have a series of decaffeinated introductory courses in a whole host of languages. I taught the one in Python. It was 4 classes that were 3 hours each, and covered both the basics of the Python language and some applications such as web development.

It was a good experience, and I had 40-50 students. That being said, I learned some very valuable lessons:

1. Teaching is hard
2. Teaching for 3 hours straight is really hard
3. It's hard to prepare enough material to teach for 3 hours straight
4. If you do have enough material, it probably took you at least 6 hours to get it together

I certainly have a lot more respect for the work that good teachers must put into their preparations, because I know that I'm certainly not a great teacher, and the preparation took me a really long time. On the other hand, it was a really good experience, and I really hope that I get a chance to teach this again next year.

Other than that, I basically spent all of IAP doing unimportant things - cleaning my room (although not too much), setting up a better backup solution (although it's still not there yet), and lots and lots of sleeping.

Anyway, that's all for now, but Monday is the Ring Premiere, when all of the sophomores get to see what our Brass Rat looks like for the first time. Lots of people think that Ring Premiere isn't that interesting, but I'm personally excited.

And thus began the second of the dreariest, grayest, rainiest first two days of school I can remember.

Fortunately, I was intentionally getting up well before my classes started and the breakfast was tasty, so it all works out in the end.

Anyway! Enough of that. I think of my "list of potential blog entries" as a stack, not a queue - I do the most recent things first. So today I'm exploiting that property to put off writing about last semester even longer in favor of the classic what-classes-am-I-taking-this-term post.

I'm signed up for 66 units of classwork this year. In theory, one unit corresponds to one hour of work per week, be that in the classroom, in lab, or at home. In practice, the correspondence usually isn't so great, but it's still a good indicator of just how hosed you're going to be. To put this in perspective, for the last two terms I've signed up for 51 units. This should be an interesting term.

(Additionally, courses are broken down by how the expected distribution of hours, to give you a sense of how you will be spending your time. These are usually written A-B-C, which means A hours in the classroom, B hours in lab, and C hours doing out of class work. I'll go ahead and put those in, too.)

6.003 Signals and Systems (4-2-9): This class covers basic signals analysis, how to build filters, and analyzing things in the frequency domain, instead of the time domain by using Fourier, Laplace, and z-transforms. Lots of cool concepts, but the class has a reputation for being "grungy math part II" (to 6.002's "grungy math part I").

6.004 Computation Structures (4-3-8): Alternatively titled by the instructors as "The Way Digital Things Work". This class starts at the level of MOSFETs (basic electronic switching devices) and moves up through gates, modules, integrated circuits, processors, and a basic operating system. By showing you how to take some small thing, build something bigger, and then stop caring about the smaller thing, you learn how to build very, very complex components with millions or billions of individual MOSFETs. And in the process, you do actually build each of these pieces in simulation, using software that was designed for the course. 6.004 has a reputation for being one of the best classes in Course VI, and many people (Jessie included, I believe) count it as their favorite class at the 'tvte. This term, 6.004 is being co-taught by Steve Ward and Chris Terman. I'm really excited about having another class with Chris, who was the professor for 6.111 last term.

6.033 Computer System Engineering (5-0-7): According to the Underground Guide Course Evaluations, covers "everything you ever wanted to know about computers, but didn't know to ask." Things from how the X Windows System works to why Worse-Is-Better is a better development philosophy than Do-The-Right-Thing (the jury's still out on that one, for the record, or at least I think it is). It should be a good class, and the lecturers are supposed to be excellent. This class is also a CI-M, a "communications intensive in major" course. I have to have two to graduate, so this will finish off that requirement (6.111 was one as well).

18.440 Probability and Random Variables (3-0-9): Many of the EE-related classes require a probability class, either 6.041 or 18.440, which is supposed to be a little more mathematically rigorous. A friend of mine just changed his major to 18, and has to take 18.440 as one of the requirements, so I figured I'd take that, because classes are always more fun when you're taking them with people you know.

(Ok - I wrote that blurb this morning, but having actually been to today's lecture, I'm actually really excited about it, because while I usually hate probability, Professor Kelner seems to be a really good teacher who gives very well prepared lectures. He's a recent MIT graduate student, so he also seems to be fairly understanding of students' needs - for example, he's planning to hold one set of office hours the night before the psets are due.)

17.42 Causes and Prevention of War (4-0-8): The only reason I signed up for this class is that lots of my friends were as well. I've never taken a political science class before, so this should be interesting. It's a CI-H ("communications intensive humanities"). I need two of these as well, but I already have 2, so I'm actually taking this one because it counts towards the HASS-D requirement (you have to take classes in a certain variety of disciplines. Please don't ask me to explain the various HASS requirements. I honestly don't think anyone really understands them).

Hmm...well, I wrote just about everything up above this morning. Then classes happened, I got distracted, and am only just now finishing.

It turns out that the end of my day made the perfect counterpart to the start of my day.

First, some background. My room doesn't get much circulation, so as a general rule, I leave my window opened a few inches when I leave in the morning so that it's not uncomfortably hot when I get back from classes. This is normally fine - even when it rains - because Burton-Conner has the interesting property that it always seems to be at a higher pressure than the outside world. That, combined with the screen in the window, seems to keep my room cool and dry.

Of course, today I leave W20, where I had been doing homework, to walk home. After a day of fairly clear weather, it's raining again, and harder than before. So I get back to my dorm soaked, and walk up to my room to find the third of my bed closest to the window nicely doused by the rain.

But that's ok. It was a good day. My classes are interesting. My professors are good. It should be a good term.

It's awesome. It's lots of fun. You should do it when you get here. Or keep working on those puzzles that Snively and Brian posted—we still don't have right answers to those!

Anyway, I'm only posting just now, because I've been Operating. Death From Above is a HUGE team (our mailing list tells me that there are 115 people!), and probably 1/4 or 1/3 of those are hunting remotely, instead of from on campus. Smaller teams can coordinate by just yelling across the room. Since we can't really get away with that, we have an Operator (inspired by the Matrix). It's the Operator's job to keep track of who's working on which puzzle, where they are, how they're doing, whether they've slept, and when they're hitting their heads against the wall so hard their brains are going to fall out.

In addition, the Operator is responsible for "calling in" answers. When you think you have an answer to a puzzle, you fill out a little form on the official Hunt website, and they call you to check your answer. As the operator, I take other people's answers and call them in, so I also get to enjoy all of the excitement when we get an answer right (which happens pretty rarely, for the record—we're averaging less than one an hour).

So I've got this awesome little command and control center in the corner of our headquarters. There's my laptop in the middle, where I watch what answers have been submitted, an extra computer on the left, where I have Skype, AIM, and Jabber open to talk to remoters, and the VOIP phone on the right that we borrowed from IS&T (because cell reception here is practically non-existent).

I realized last year that I'm a lot better at trying to organize people than I am at actually solving puzzles, so I've been focusing on doing that. Every day since Friday, I've shown up at noon, sat behind the desk for about 14-16 hours (with occasional interruptions to run over to the other rooms), and then gone home and slept for a few hours.

It can be boring, but it gets really exciting every few hours because you're talking with 5 or 6 remoters, and trying to handle 3 different people who all have answers at the same time, and the phone just starts ringing off the hook, and...it's just awesome. So much fun.

Anyway, that's my Mystery Hunt story. Anyway, back to the phones—while I was writing this, we just solved 3 more puzzles! (although we're still at less than 1 per hour)

This particular post, though, is about a particularly well-known class around the Institvte.

As many of you know, 6.001 was a class invented at MIT to teach the fundamentals of how to program. The first class was taught in 1980 by Hal Abelson and Gerry Sussman, the authors of its companion book Structure and Interpretation of Computer Programs, which also happens to be the official name of the course. The class has been offered constantly since then, and it's been taught roughly the same way.

The class was structured around Scheme, a programming language which Sussman invented with Guy Steele in the 70's here at MIT. It's an unusual language with a syntax that's radically different from most other programming languages (at least ones that aren't LISP), so most people aren't familiar with it coming in. And because of this, the class has gotten a lot of flak in recent years, especially from people who know "more normal" programming languages.

In spite of this, 6.001 had a very strong reputation for teaching people not only how to program, but how to think about programming, and how to take big problems and deal with them. There are a lot of classes that claim to do this same thing, and probably many of them do, but 6.001 seems to be unique in just how it manages to pull it off.

Starting this year, though, the Course VI department is beginning to migrate incoming freshmen to the new curriculum. And 6.001 doesn't really have a place in the new curriculum, so this is the last term that it was offered. Several years ago, Sussman said that he wanted to be the last person to teach 6.001, and so he taught it this term, taking it back from (guest blogger) Eric Grimson, the head of the department, who has taught the class for as long as any of my friends can remember.

Because 6.001 is "early" in the day (10 AM - which is very early by MIT standards), frequently the students will opt for other means of learning the material, but for this last lecture ever, current students of 6.001, past students of 6.001, and even people who had never taken 6.001 came to see the class off.

And just to put their own stamp of approval on the class, some hackers apparently covered the original room number (32-123) with a new number for the day:

For the last lecture, Sussman spent about 30 minutes talking about the halting problem - the idea that a computer can't determine whether or not it's possible to calculate something in finite time. At the end of his lecture, though, one of the TA's came up to say something about the class's passing. In particular, he noted that it was "remarkable that a course invented then [the 80's] is still so dead on in what matters." He pointed out the remarkable insight of Abelson and Sussman to see into the core of computation, "keeping it fresh for more than a quarter of a century." And afterwards, the entire room stood up and applauded the class.

Now that 6.001 is gone, it's being replaced by 6.00, 6.01, and 6.02 in the new curriculum. There are a lot of students that don't really like the new curriculum, but for the most part, it seems to me that they are just being resistant to the change. It's definitely true that 6.01 hasn't quite settled into being a really solid class yet, but 6.02 looks like it's amazing. Among other things Chris Terman, my 6.111 professor, is teaching part of it, and he's just generally awesome.

It think that the new curriculum isn't quite there yet, but it's definitely close.

I promise that I'll try to post more during IAP and this next term! Really! I've definitely got a long list of things to talk about.

But when you move to Boston, it's really hard not to be a Red Sox fan - you just kind of get sucked into it. And it's even worse when you have a lot of friends from around here - you wander into a suite lounge or a floor lounge to find 10+ people anxiously watching everything, and screaming whenever anything happens - good or bad. It's exciting, and it's even more exciting when the team is winning.

The night of the seventh game in the ALCS, I was on Burton Third starting to plan out our 6.111 final project with Chris '09. Of course, we got a little distracted. And when the Sox won, a massive crowd of people left Burton 3rd and went down to Burton 1, cheering as soon as the elevator doors opened. They were immediately joined by Kip (who, remember, has a voice "only a few decibels louder than a sledgehammer"), and JKim, among others.

We then went floor by floor up Burton-side, cheering every time the elevator doors opened, and every time, people would come out of their floor lounges cheering with is, until we couldn't all fit in the elevator.

But that's not what this post is really about.

It's not even about the fact that the Red Sox completely dominated the game last night.

It's about the hack that went up in the middle of the night to support the Sox.

I got an e-mail this morning from a friend at Next House saying that she saw this above New House on her way to classes:

This, of course, is the wrong view of the banner. A better version looks something like this:

When I went to look at it, it looked like it was strung between Tang (a graduate dormitory) and MacGregor. Absolutely insane. Over 1000 feet, apparently.

You can see Tang on the far left, barely in the frame, and MacGregor is on the right.

Go Sox!

I should also warn people at the beginning that this is a long post. I wrote it to show people the way that I chose to tackle an electronics problem that was presented to me. It's not something that I learned directly from any class, and I don't have any clever terms to describe how I approach things. But it still shows something that I've learned from here, because before I got to MIT, I never would have been able to make it all the way through a project like this. Those of you that know basic electronics will find parts of it boring, and those of you that don't will find parts of it inaccessible. You should read this anyway, because it's just a cool project.

In my advising seminar last year (6.070 - it's a great class - I highly recommend it, not just as an advising seminar), the instructor said (and I'm paraphrasing here) that "all of electronics can be divided into blinky lights, funny sounds, and weird smells." I'm not sure why the smells were included though - we usually try to avoid those.

I'm a little disappointed that nobody has asked me about the breadboard I'm working with in my banner photo, because that actually happens to be a prototype for the project I want to talk about. It falls into the blinky lights division.

My friend Chris '09 and I rented a Zipcar for a day at the beginning of the summer, because he was helping me move into my summer room at EC, and he also needed to run to Target, Costco, and a few other places. And once you have a Zipcar for more than 4 or 5 hours, it becomes cheaper to get it for the whole day, so we threw in a trip to IKEA just for good measure.

While we were there, we discovered these awesome glass shelves with fluorescent lights on the inside. They were called NIANs, in IKEA's typical we're-going-to-pretend-to-be-exoticly-Swedish-while-still-using-words-that-sound-like-English fashion. They were also on clearance, and thus marked down from $70 to $10. Now, granted, it is IKEA, so it's hard to tell whether it was originally $70, but regardless, at $10, that was an awesome deal, so we bought two of them each. They look something like this:

So we're two EE-inclined guys, and we look at these lit shelves, and we come to the natural conclusion that the only possible thing to do with them is to replace the fluorescent bulb with RGB LEDs, and then make that computer controlled.

And because Chris was a lighting techie in high school, he suggests that we should make both ends communicate using DMX512, a standard protocol for communicating between stage lighting devices.

So...I have a project. Well, more of a problem (problem as in problem set, not personal problem). And the way that I solve problems like that is by breaking it down into a series of simpler problems.

But before I do that, I'd like to give a quick preview of the finished project, or at least the project as it is now. We don't have the LEDs in shelves yet, but we do have a demo board that drives one LED:

Ok. Back to the design steps. The first step for me is figuring out what the signal path is. I've already established that I'm using DMX to drive RGB LEDs, so it looks something like this:

Ok. The next step is to figure out exactly what's coming in and exactly what needs to go out. On the one end, DMX travels on top of a protocol RS-485. That's a protocol with 2 wires, and the signal is either a 1 or a 0 based on which of the two wires has a higher voltage than the other (called a differential signal).

On the other end, the LEDs will be powered with a PWM signal. PWM is a way of varying a signal from completely on to completely off. Since the red, green, and blue components of the LED should each have a variable brightness, we can use PWM to change it. Basically, you turn the output on and off really quickly, and the longer it's "on," the brighter the LED seems, until it's always on, which is full brightness. For those of you who are math inclined, the brightness level that our eye perceives is the average voltage over time. Here's a photo showing what the signal looks like when it's 1/3 of the full brightness:

^[1]

Just for comparison, here's another signal that is 2/3 of full brightness:

Ok. So we know what signals we're dealing with now. Let's go ahead and summarize again:

Now that the problem is pretty well defined, I try to start from each side and work my way inwards until I have a series of processing "blocks" that I know I can implement individually.

For example, the two wires in the RS-485 (the protocol that DMX uses) signal can as high as +12V or as low as -7V, but most of the chips that I work with operate with signals that are +5V for a "1" and 0V for a "0" (These are known as "TTL logic levels"). Feeding something with the levels of RS-485 will almost immediately fry the chips, so I need to convert those levels to TTL logic levels.

Now, it just so happens that I know of a component that can take in TTL level DMX signals and convert those to PWMs: a microcontroller (often abbreviated μC - remember that μ is the Metric abbreviation for micro). A microcontroller is sort of like a miniature computer: you can write programs for it (usually in C), download them onto the chip, and either read inputs from the pins or make the pins high or low. Their processing power is roughly equivalent (or maybe slightly better) than the earliest computer that used to take up whole rooms. Since I've turned the problem into TTL signals coming in, and TTL signals going out, a microcontroller can probably take the place of that void of "Processing."

So, the final flowchart looks sort of like this:

And that's the signal path. But that's a whole lot to digest for one post, so I think I'm going to stop there, and deal with the questions that I'm sure people have (because I'm terrible at explaining these sorts of things). Sometime soon I'll do another post about how you go from that flowchart to a circuit, and then to an actual working PCB, and also deal with some of the problems I had - because it's never as simple as the flowchart makes it look!

[1] (This photo was adapted from the Wikipedia article on PWM, and so I am obliged to release it under the GFDL. Therefore, I also provide the source code used to generate the images - 1 and 2)