The Operating System is Dead

As an infrastructure provider we have been involved in maintaining and running hundreds of thousands of operating system installations over the years. Yet we believe that the operating system as we know it is slowly going extinct.

In my view there are two main reasons for this:

1. The competent sysadmin is disappearing from the human resources market
2. A single server doesn’t cut it anymore.

The competent sysadmin is almost extinct

If you think developers are hard to find look at sysadmins. System administration as a discipline requires in-depth knowledge of operating system behaviour and tools. Many sysadmins are specialised in either Linux or Windows much in the same way programmers are specialised in Java or Ruby but maybe with less variety.

All good sysadmin we know have grown up tinkering with Windows 95 or Slackware on their custom built PCs. Back in the day things broke down often and needed things like manual registry change or kernel recompile every month or so just to keep them afloat. Inevitably you end up picking up a thing or two and before you know it you are a sysadmin. It was like in the early days of the automobile industry.  A few passionate visionaries built cars in their backyards, ran them without a helmet a blazing (back then) 40Mph across a dirt track. Everybody knew what a carburettor was or how to watch for oil pressure. Just a handful of cars were in existence back then.

In recent years though, things got complicated as they got more and more popular. The average sysadmin now has to deal with farms of servers or VMs and their hosts. It’s a miracle they’re keeping up. Things have been invented to make this process easier. Things like Puppet, Chef and our beloved Ansible have been invented to help them manage this pleiad of assets.

Unfortunately,  to be able to use any of these a sysadmin must be able to write code. You could say that it’s not real programming but rather some specialised recipe editing but I would argue that things are rarely  that smooth and you can’t find recipies for everything and sooner or later you end up having to write your own.

And so, this requires more and more involvement by the development team in what was ran traditionally by the sysadmins.  This is now called DevOps.

The issue with DevOps is that it is now even harder to find good engineers to serve this role because if an engineer is good enough to write code she will usually become a developer rather than a sysadmin as the horizon is simply wider for them.  I’m not saying people are not drawn to this field anymore or that being a developer is better.  What I’m saying is that statistically you get many more developers to a sysadmin and it’s becoming such a huge problem many organisations are now simply using some developers as sysadmins.

These developers are approaching the problem differently. They don’t have the huge in-depth knowledge of how an operating system works nor do they want to have. They simply want to get over with it as quickly as possible and so many of them just see this part as a nuisance. They prefer working with APIs and some user interface.

This is why platforms are gaining momentum.

Distributed Operating Systems

I would argue that the OS is pretty much on the verge of becoming just a piece of the puzzle, moving away from the very central role it always had. As developers, we still need configuration data to be stored somewhere, we still need locking and coordinating processes. These things were traditionally offered to us by the operating system and a single server and are now being offered by distributed operating systems and a collection of many servers.

There have been some attempts in the past to offer this but they failed to gain momentum. In the most recent years Docker has helped the adoption of distributed operating systems because of the simple deployment method  (the Dockerfile) along with the efficiency of Linux Containers. It isn’t enough by itself though. One needs to manage many different containers at the same time to really make sense to use it instead of VMs. You simply cannot do it manually as there are too many of them. The microservices paradigm in itself promotes the usage of swarms of specialised services so it is now even harder to keep up, even with things like Ansible or Puppet because you need to make sense of the resources available and find a “space ” where to stuff a container in a big cluster.

And so, what is really needed is what the operating system calls a scheduler or a “process manager”. This is what Mesos and Yarn are bringing to the table. And so we are getting closer and closer to an operating system model, but at a different scale. Here is a comparison:

The platform services out there such as AWS, Azure and GCE and Bigstep provide methods to develop these new “Applications” but in a portable way. This means you can take your “Application’s Docker Image files and simply “Execute” them on another compatible cloud, running the same “operating system” and you’re set.

Unfortunately things get complicated when you need to do debugging or performance tuning. I think this is where best practices as well as the involvement of the platform vendors comes in.  Things like the THP heavily affect performance and tuning and tweaking will mean a lot of saved dollars but will also require specialised people and things will quickly get out of the range that a regular developer will be able to mess around with.

Think about these new components like you do when thinking about a modern BMW engine. You can look at it, perhaps add some oil to it but truly messing with it is beyond many of us.

Popping the hood is no longer necessary nor recommended unless you are a professional specialised in that particular piece of the puzzle.

At the same time bigger, safer, faster and more affordable cars are now available.  We have traded flexibility and transparency for efficiency.  This is why I believe the only way forward for the industry is through high quality platform services and the operating system as we know it will have its place only as a specialised part of the stack. Things like Ubuntu Atomic already show this trend to become a reality. Soon very few people will ssh into operating systems and it’s for the best. We can finally get the grease off our hands and start driving.

What are your thoughts on this subject? Let us know in comments.