Agile Sysadmin

London DevOps Meetup

2010-02-23T23:03:44Z

Since the inaugral DevopsDays conference in Ghent, Belgium, a few of us London-based devops have been gathering once a month to eat, chat, swap stories and generally encourage one another. Since we’ve sustained this for four months, we concluded that we have a kernel of people who would turn out for a monthly meeting, and decided that this month we’d throw it open to the world!

If you don’t know what this whole Devops thing is, I suggest you take a look at my article ‘What is this Devops thing anyway?’, and also James Turnbull’s (excellent) ‘What Devops means to me’. In short, we’re trying to build a community, a movement even, of cross disciplinary people - people who can program, and manage systems, people who understand that in the software industry, coders, testers, analysts, and sysadmins are all on the same team. If this sounds like you - if you want to make a difference, and would like to meet similarly motivated people, you’d be very welcome.

There’s no particular technological bias - we generally like Puppet or Chef, we largely run our infrastructures on Linux, and there’s a tendency to program in Ruby or Python, but I think this is just representative of the places we work and the people we know. So, if you’re interested in bridging the gap between devs and ops, and if you’re interested in how to build infrastructures in an agile way, we’d love to see you.

The intended format is straightforward - every other month we’ll have a social, drinks-based event, in a hostelry of choice, and interleave this with a talk-based event, kindly sponsored by The Guardian at their awesome new premises in Kings Cross.

Tomorrow’s event is at The Priory Bar in Clerkenwelld, starting at about 1800.

Don’t worry if you can’t make it that early - it’ll be totally ok to turn up at any time - just depending on what you have on, and how far you have to travel. There’s no fixed agenda, just a chance to meet some cool people, chat and drink.

We’ll be following this up with a technical meeting on the 2nd of March, at the London ThoughtWorks office in Holborn. We’ve not yet decided what we’ll do there, so if you have any ideas, let us know.

Please let others know about the event - get tweeting and blogging - the hashtag is #ldndevops. In the meantime, take a look at our London Devops planet, and if you like irc, pop in and see us on freenode in the ##infra-talk channel.

See you on Wednesday!

A short introduction to Chrome OS

2010-02-22T05:03:44Z

Google are everywhere these days. The number one search engine, a suite of hugely powerful mail and business applications, shopping, maps… but last year they seemed to move from the Software as a Service model to something rather different, with the release of their browser, and their operating system, Chrome. Richard Cohen explores what it’s all about.

We’ll start with ‘cloud computing’, which is actually an appropriately nebulous term… It’s been traditional for a long long time, particularly when teaching computer networking, to use a cloud in a diagram to represent “all the stuff which happens on the network between computer A and computer B”:

We don’t care how the communication happens between A and B, we just assume they can talk to each other. This usage has been changed a bit, in the past 5 years or so, to add the idea of services which live in ‘the cloud’ - we don’t care where they are, as long as we can communicate with them over the Internet, using standard tools from anywhere. A simple example would be using gmail/hotmail/yahoo for your email, rather than an email system supplied by your employer, school or ISP. Another simple example would be Facebook - you don’t know anything about how it works, you don’t have any relation with them other than being a user of the service, you can access it from anywhere…

So ‘cloud computing’ is the idea of taking things which previously ran either on your computer, or at least on your local network, and shoving them out into the cloud. It relies on a combination of 2 or 3 technologies, which are just about available now, but weren’t 5 years ago - ubiquitous broadband, standard network protocols, and, to a slightly lesser extent, cheap/ubiquitous computing.

Chrome OS is what falls out of all that, if you’re Google. It’s an OS designed to run on cheap portable computers (i.e. netbooks) which are always on-line (wifi or 3G) and running a standard platform called a ‘web browser’. It assumes that you use Gmail in your web browser, rather than Outlook connected to an Exchange server. It assumes that you use Google Docs rather than Microsoft Office. It assumes that you stream music and even video (TV, etc.) from the web, rather than storing and playing it locally. It assumes that you’re using Google talk through your browser, rather than running a local chat program. It assumes that you’re doing all of this on a ‘secondary’ computer.

They’re not claiming that you can or should run your life like this yet. What they are saying - correctly, in my opinion - is that if you are willing to trust the ‘cloud services’, you can move a lot of what you do online without losing very much, and gaining ubiquitous access in return. You’ve got a big heavy laptop… I’ve got a little handheld phone… they both access the same online data, to a great extent, and my phone has a longer battery life, built-in 3G, and lacks the potential for spraining my back by carrying it.

As for running Chrome OS to take a look at it, there are a couple of options. You can’t run it directly on your Mac, because it only runs on netbooks, and a small specific list at that. What you can do is run it under virtualisation - same same way I, for those rare occasions when I need it - run ‘Windows in a window’. VirtualBox is a virtualisation tool, which will let you run Chrome OS in a window on top of your Linux, Windows or Mac OS machine. Alternatively, and much more simply, just load up the Chrome browser - technically, the chromium open-source browser, which is the core of Chrome, since Chrome is still Windows-only. Chrome OS is, in essence, a modified form of the Chrome browser running on streamlined Linux OS, so if you want to know what Chrome OS looks like today, running Chrome (or chromium) will get you a lot of the way there…

Chromium is pretty usable as a browser - I’ve been using it as my main browser for a few weeks now, with little trouble. Chrome OS, however, is not usable yet. It’s expected to be released to hardware manufacturers sometime around the spring, and be on the market latish next year.

About the author

Richard Cohen is a Linux expert and Python developer, living in Hong Kong. A long time authority on handheld devices, he also holds the rare distinction of having worked for both Sun Microsystems and SCO.

How to build 100 web servers in a day

2010-01-21T20:03:44Z

Strategies for configuration management: imaging and automation

“Great news,” says your boss. “The server you built yesterday is perfect. Now we need a hundred more just like it. And when you’ve done that, could you take a look at my laptop before lunch? I can’t seem to get into my email.”

It’s common nowadays for sysadmins to manage very large numbers of servers - often virtual servers created on-demand in cloud platforms like Amazon EC2. How do you build so many servers and keep them updated - automatically, reliably, and quickly?

One common strategy is the golden image pattern: set up the server the way you want it, and then copy the disk image and use that to boot as many clone machines as you need. Another is the automation pattern: start with a generic server, and then apply automatic configuration tools such as Puppet to bring the machine into a specified state where it is ready for production.

The golden image

The golden image is attractive to many because it’s easy and cheap. It’s quick to create images, and you just need a big disk to store them.

Imaged machines also have low latency: they’re ready for production in just a few minutes from startup. Finally, it’s platform agnostic: you can image virtually any kind of machine, even Windows (which most config automation tools can’t handle very well).

So what’s the problem with the golden image? Luke Kanies, creator of Puppet, prefers to call it the “foil ball” approach: your images can quickly become an unmanageable, crumpled mess. Firstly, how many images do you need? Well, as many as you have different types of machines: web servers, databases, load balancers, DNS servers, and so on.

The number of images multiplies again when you take history into account. What happens when you need to recreate the state of the database server last month to troubleshoot a problem? You need to keep old images around, which means a new image every time you make a change.

And here’s a thing: if you do need to make a change (for example a security upgrade for a critical package), you’ll have to rebuild the golden image and then re-image and reboot all affected machines - which means lots of bandwidth and possibly downtime. And, unfortunately, even imaged machines usually need some final tweaking (eg IP addresses) before they’re ready for use.

Automation tools

By comparison, building your machines with an automatic configuration tool such as Puppet, Chef or even cfengine gives you many advantages. The configuration is self-documenting; it’s easy to see what should be on each machine, and more importantly, why it’s there. It will also stay in sync with reality.

Although your server types will be different, they will also likely have a lot in common. With images, these supposedly common settings could easily drift apart over time. With automation tools, you can put these settings into modules or classes, just as a software developer re-uses a module or library in another project.

In fact, all the advantages of software modules are available: you can share code with others, download public recipes and manifests, and maintain a complete version history of your network.

Automated configuration is intrinsically multi-platform. Puppet and Chef can abstract away some of the differences between operating systems, and you can add minor tweaks of your own. For example, you might need a slightly different MySQL config setting for MySQL 4.x as opposed to 5.x. A one-line switch in the configuration can address this problem, instead of creating two disk images.

Of course, everything comes with a downside. It is more difficult and usually more time-consuming to build an automated configuration, especially from scratch and when you’re still learning to use the tools. And some things are hard to automate. Some software just wasn’t written with automated installation and management in mind.

Also, not all software will be packaged for your platform (or you need special compilation options). In these cases you will need to package the software yourself.

Finally, configuration management doesn’t always mix well with human intervention. In particular, someone may make a config change on the machine which is then automatically reversed by Puppet. At best this can be a nasty surprise, at worst it can result in a site outage. Managing desktops automatically can create similar problems. This needs to be addressed at a business level with a proper change control process and making it clear to everyone where the boundaries of automatic management lie.

Combined strategies

So both imaging and automation have their place; neither is a magic bullet. The best strategy for scaling your deployments is often a judicious combination of these two approaches. You can, for example, start with a ‘stem cell’ image containing an approved base build, security fixes and settings which don’t often change, which is then configured by Puppet or Chef. Or you could use provisioning tools such as Kickstart and Cobbler and then hand over to Puppet. Or even build the machine with Puppet and then create a golden image from this.

One thing is certain: ignoring the problem is not an option. Moving to an automated network might seem like a big project, but the key is to start small. Set aside a block of time every week to spend on improving your infrastructure, and automate one small thing at a time as part of a gradual continuous process. The more you invest in this, the more benefits you’ll see - and the more time you’ll have for fixing your boss’s email.

About the author

John Arundel is a consultant, systems integrator and automated infrastructure dude who specialises in building performant, highly-reliable systems with Puppet and other open-source tools. He writes about these and other issues at the Bitfield Consulting blog.

Advanced Dependency Management with Yum Shell

2010-01-15T12:03:44Z

Many years ago, Red Hat had a (rather unfair) reputation of being a Linux distrubution forever crippled by a painful and clumsy package management system. “Dependency Hell” was the name used to describe the situation one could get into where circular dependencies arose, and the user became stuck. Dependency hell (which was always a bit of a myth) is a thing of the past now, especially since RHEL 5 where Red Hat adopted the powerful Yellowdog Updater, Modified (yum) which CentOS and Fedora had used for some time. However, just occasionally, problems arise which require some black-belt dependency solving.

For example, last week, a Linux user was attempting to replace the default sysklogd with rsyslog. He had a package which provided a newer version than the one provided in the distro, and was trying to install it. Yum wouldn’t allow this, owing to a conflict of ownership of the logrotate syslog entry. He was going to try uninstalling syklogd, but a quick check suggested this might not be a good idea:

[root@yarg ~]# rpm -e --test sysklogd
error: Failed dependencies:
    	syslog is needed by (installed) initscripts-8.45.30-2.el5.centos.x86_64
        syslog is needed by (installed) vixie-cron-4.1-76.el5.x86_64

Aha - yum will remove initscripts, upon which most of the system depends, so that’s not a great idea.

So - we have a catch 22. We want to install rsyslog, but we can’t because of a conflict with sysklogd. We can’t uninstall sysklogd, because that will remove most of the system. Now, one could trust that rsyslog is a drop in replacement for rsysklogd, and remove sysklogd with rpm -e –nodeps, but that feels a bit hacky. No, this is exactly the sort of situation in which we can use yum shell.

Yum shell is a very nifty feature of yum which allows the user to stack commands to resolve various types of packaging issues. This is exactly what we need to do with our rsyslog / sysklogd issue, and was a godsend on earlier versions of Redhat when replacing sendmail with exim or postfix provided similar challenges.

So, we fire up yum shell:

[root@yarg stephen]# yum shell
Loaded plugins: fastestmirror
Setting up Yum Shell
> install rsyslog
> remove sysklogd
> run 
> exit
Leaving Shell

You’ll see some feedback during the process - but what’s basically happening is that all the operations are being run in a single transaction (and, incidentally with only one call of python, and the yum libraries, repo checks etc). This means that we get around the issues yum was complaining about, because the operations are carried out in a single shot.

I think yum shell is a powerful feature, and it’s great to have the capability to script sets of commands and bundle them up into a single operation, and one I think you’ll find will be a potententially valuable addition to your toolkit.

How to print every nth line of a file

2009-11-23T12:03:44Z

Sometimes a scripting language isn’t the right tool for the job. I’ve been working on a piece of code that parses huge logfiles. The test data needed to be representative across 24 hours - a simple slice wouldn’t do. I spent a few minutes knocking up a python script to print every 100th line when I stopped and thought ‘There’s a better way’!

Of all the Unix command line utilities that I’ve used, I think I’d highlight sed as the most useful and powerful. You can really accomplish quite a terrifying amount with it. To meet my needs, the sed ‘script’ is seriously trivial.

sed -n '0~100p'

We invoke sed with the -n option, which means only include explicitly printed lines in the output. 0~100 means select every 100th line, beginning at 0. Since 0 doesn’t exist, it starts at line 100. It then just prints the line.

I didn’t pursue my python script to do the same, but I wouldn’t mind betting the sed version is much faster, and given the size of these logs, that a very good thing indeed!

The Atalanta Systems Guide to Open Source Operating Systems

2009-11-19T05:23:44Z

One of the most common questions for newcomers to Linux is “Which Linux version should I use?” or “Which Linux version is best?”. I’m also asked for my professional recommendations about which OS, or which distribution people should run, especially by people new to the idea of using an open-source operating system. I had this conversation again very recently, and felt it would be worth publishing my own experiences and recommendations.

Broadly speaking, computer use tends to be divided between workstation/desktop and server. In the review that follows, I’ll try to approach the discussion from both perspectives.

Red Hat

For many people, especially in the commercial business world, Red Hat is Linux. If you work with Linux professionally, my experience is that the vast majority of machines you will meet will be running Red Hat.

Back in 2003 Red Hat changed the world. Until then Red Hat (by this stage at version 9) was entirely free. You could buy support from Red Hat, but all the software, all the binary packages, all the patches were freely available. Red Hat changed the rules - they announced that henceforth Red Hat would be producing two versions - an enterprise grade, commercially supported, paid-for ‘Enterprise Linux’ (these days referred to as RHEL), and a ‘community edition’ - called Fedora Core. The idea was to use Fedora as a community driven, fast-moving, entirely free and open test bed, from which technology would trickle down to the more stable enterprise edition.

So how does Red Hat shape up in 2009?

On the server, Red Hat is unquestionably the industry leader. I’ve deployed and managed many hundreds of RHEL 2, 3, 4 and 5 boxes, and can say without a doubt that it is very highly reliable and stable. Documentation is generally of high quality, and on account of its status as the industry standard, there’s also a vast amount of information available on the internet. Red Hat commercial support has generally been good, on the few occasions I have had to use it.

One big advantage of RHEL, since version 5, is the adoption of the yum package management and dependency solving tool. Coupled with a decent understanding of RPM, and when best practices are followed (i.e. not mixing repositories and randomly installing third party packages), this provides what is, in my view, the ultimate *nix package management solution.

A further benefit of Red Hat is they have a large number of incredibly smart people working on some very clever tools - a quick glance at the suite of stuff coming out of Red Hat’s Emerging Technology (ET) labs reveals a huge depth of well-written and carefully thought out utilities. Most of these have good documentation and active mailing lists and IRC channels. They’re also frequently written in Python, and are very open to patches and contributions.

On the down side, the implication of a very stable platform is a very slow release cycle. This means certain packages can get really quite significantly out of date - Ruby and Python are obvious examples, but similarly the Apache webserver, Tomcat, PHP, MySQL - these have been patched for security but few if any upstream features or improvements make it into the Red Hat repositories. This can be mitigated in some ways - Red Hat sell an Application Stack, which provide an updated LAMP platform, and there are reliable community projects which provide newer version of the stock packages.

Another disadvantage is that Red Hat’s Licensing structure means that cost can quickly get expensive if there is a requirement for anything not in the ‘base’ channel - so clustering support, virtualisation, the application stack and other add-ons can be quite a budget breaker.

On the desktop, RHEL ships with Open Office and Firefox, and for the very simplest of tasks will be adequate. However, manual intervention will be required to get java and flash plugins working, and the default music and video playback capabilities are primitive. Notionally Red Hat is suitable for the desktop, but it’s rarely used in this environment.

CentOS

As soon as Red Hat announced their plans for commercial linux, groups of people sprung into action. Being an open-source company from the very beginning, Red Hat agreed they would continue to provide source rpms for all their software (to subscribers). Thus immediately it became possible to rebuild a free-as-in-beer RHEL from the sources, having removed logos and not freely distributable content. Thus the CentOS project was born.

CentOS has most of the advantages of advantages of Red Hat - being 100% binary compatible, it’s exactly as reliable and stable, only it’s absolutely free.

You don’t get official commercial support, and there is a delay between the release of the latest Red Hat point release, and the CentOS equivalent. This can be several weeks, or more.

CentOS users tend to be seasoned Red Hat users, and frequently seem to occupy the higher strata of professional Linux users. This means that for the clueful, the community support is generally excellent. I know many of the CentOS core developers personally - they’re a super bunch.

Although it is perfectly possible to run CentOS as a workstation distribution, taking advantage of its reliability and stability, this is a somewhat manual undertaking. Not a problem if you’re a seasoned Linux administrator, don’t need the latest and greatest desktop toys, and don’t mind a bit of work and research to get things up and running. However, the same comments for Red Hat apply here: CentOS on the desktop is possible, but unusual.

Fedora

If you want to know the future of Linux, track Fedora. It’s fast moving, pretty and powerful.

Everything you would expect in RHEL is in Fedora, but much newer. RHEL is ultimately built from Fedora, so in effect you get a very good idea of what’a going to be in future releases.

Releases are frequent, but generally of high quality. There is no easy upgrade path between editions - the principle is simple: You keep backups? Fine, just reinstall periodically. It is possible to upgrade with yum, but I can’t recommend it. Maybe the process has improved since I last tried it, but if you want an upgrade that is guaranteed to work, just reinstall.

On the desktop, Fedora is cutting-edge - there is plenty of good quality, community-contributed documentation for getting the usual requirements set up.

Debian

Debian, for me, just gets it right. I’ve used Debian consistently since ‘Potato’, and have found it reliable, easy to keep up-to-date, logical, and secure. It’s very portable - running faithfully on eleven platforms, and quick and easy to install.

At any time there are three Debian branches - stable, testing and unstable. Stable is as its name suggests - solid, reliable and slow to change. Testing is generally pretty up-to-date, and pretty fast-moving - it represents what the next Debian stable release will be, and periodically is frozen, and then becomes stable. Unstable (also called Sid) is bleeding edge, very fast moving, and occasionally broken.

A frequent criticism levelled at Debian is that it can get out of date. Once stable is released, only security updates will be applied, meaning the latest and greatest features never make it in, and as time elapses, it begins to look a bit long in the tooth. In fairness, this is no different for Red Hat or CentOS. However, Debian has a well-designed backports system, which enables newer packages to be built using the stable tool chain. These naturally carry the risk of compromising the stability of the system, but it’s a reasonable compromise. Furthermore, Debian Testing is usually every bit as stable as Fedora or Ubuntu, and pretty much equivalent in freshness.

One huge win is the breathtaking number of exceptionally well-maintained packages that are available out of the box. At last count there were something like 15000 packages available across all platforms. Debian maintainers have earned a (rather unfair) reputation as somewhat grumpy, and perfectionist and pedantic - precisely the people you want to ensure consistently high quality of software in your community operating system.

For a server OS, Debian is ideal. The package maintainers sometimes have their own peculiar ways to manage things, but they’re generally well documented and well thought out. Most packages also come with a curses-based configuration tool which sets up the most commonly used setups out of the box. Many times when I’ve needed to get something up and running very quickly, in a way I can trust and rely on, I’ve turned to Debian.

Debian’s ‘political’ credentials are unmatched. By default they ship only software released under a respected open source license. This extends to the Firefox browser, which is shipped as ‘Iceweasel’ - with the non-free components removed. The Debian maintainers also take a firm line on the linux filesystem hierarchy - rubygems, for example, are forced to be installed in /var/lib/gems, and are not in the default shell path.

This can make getting drivers for proprietary hardware a challenge, but again, there are good repositories available which provide these packages. Use on your own conscience, and at your own risk.

Ubuntu

Ubuntu is a funny beast. Based on Debian sources, it ought to share all the benefits of Debian, but with the added bonus of commercial support and marketing from an increasingly recognised brand.

Ubuntu has invested heavily in making stuff work automatically - installation is via a live cd, with almost no user interaction. Most of the system is automatically configured, and auto-discovery of hardware requiring third party drivers, together with easy tools to obtain and install the drivers means that an Ubuntu system is pretty much the best guarantee you have to get a system that ‘just works’ out of the box.

As a result of this ease of use, Ubuntu has made big strides towards making Linux publicly available and easy. Coups such as preinstallation on consumer hardware - both netbooks and Dell workstations, have made Ubuntu the name most non-geeks think of when Linux is mentioned. Within the system, a good job has been done making everything look the same, and pleasant to use.

The downside to all this is that the general quality of Ubuntu’s six month releases has been seriously questioned. Several releases, including the most recent, have been characterised by really quite dreadful reports of instability and unreliability. Ubuntu’s commitment to making third-party drivers available has also been a two-edged sword. They can’t guarantee their stability or quality, yet they’re shipped with Ubuntu. There’s been some interesting debate around this subject in the Linux press recently - read more about it here.

Ubuntu is also, of course, run entirely for profit. Unlike the Debian project, which enjoys a sort of democratic process, the decision as to what makes it into each Ubuntu release is largely made by Canonical.

The blessed antidote to all this is that Ubuntu does offer a Long Term Support (LTS) distribution, every two years. While the initial release stands every chance of being as buggy as any other, after a few months these are mostly ironed out, and by the time of the next release, it’s generally pretty reliable. The great thing is it’s then available and supported for a further 30 months. A predictable two-year release cycle, with commercial support, offering a general package level which is, on average, slightly newer than Debian stable, is actually a pretty good recipe for success. By way of a reference, I would say that of all the laptops I have used, my favourite has been one running Hardy Heron (8/04 LTS).

Slackware

Slackware was my first Linux distribution, and for many years was my favourite. Its philosophy is simple and old-fashioned. Provide a stable, up-to-date platform, using the latest stable releases of userland tools, with a vanilla kernel from kernel.org. Packages are distributed in the form of tarballs, or software is installed from source.

Slackware makes the assumption that you aren’t stupid, you know what you’re doing, and you can read manual pages.

I still have a soft spot for Slackware - it is always pretty up-to-date, has a simple, manual, text-based installer, and reliably provides a lean, fast system.

It’s a good way to learn about Linux - almost nothing is provided for you - you have to work it out from the fine documentation.

The obvious drawback is the lack of a dependency-solving package manager. This makes it tricky to install software if you made a mistake in the installation process, or if you want to install something not provided by the distribution. Third-party package tools exist which go some way to addressing these issues, but these are not officially supported. For more information see Slapt-get.

Suse

Suse is another Linux company that has changed. Based in Germany, and originally named SuSE, they were early innovators, putting a huge amount of effort into Live CDs, the KDE desktop, and their integrated system tool, YAST. They were also one of the early supporters of a 64 bit kernel.

Novell purchased SuSE GMBH in 2003, and rebranded to SUSE. Shortly after, in 2005, there was the release of the community edition (opensuse) with the paid-for enterprise editions continuing (Suse Linux Enterprise Desktop/Server).

Suse is very well regarded, particularly on the European continent, and in the world of high performance computing. It is also RPM based, however, my impression is that provision for non-distro packages is not as great as for the Fedora/CentOS/Red Hat family.

The big weakness is that Suse lacks a CentOS equivalent - there is no freely available Enterprise Suse, which to me leaves them marginalised. I’m very rarely called upon to manage a Suse machine.

On the desktop, Suse has always been excellent. Their paid-for Enterprise Desktop is slick, clean and has all the third-party features built in as standard.

Mandriva

In the earlier days of Linux, Mandrake was a popular option for users for whom installing Debian or Slackware was a bit challenging. Originally based in Paris, it had quite a cult following, after the first release, forked from an early Red Hat version.

Between 2003 and 2004 Mandrake operated under bankruptcy protection, emerging in 2005 with a merger with Brazilian Linux provider Connectiva, and forming a new entity ’Mandriva’.

Years ago I used Mandrake for a few clients, and found it to be very good - easy to use, and reliable. I’ve used Mandriva once or twice since, and have observed that it has quite a flowing amongst new users who haven’t gone down the Ubuntu route. However, I’ve not got much experience of it. It’s probably worth playing with.

Gentoo

I’ve never used Gentoo in anger. Years ago, when Gentoo was in its infancy, and still had a cute cow as its logo, I gave it a try. The concept is simple enough - take the BSD approach of maintaining a source tree and scripts which enable systematic building of packages and tools, and deploy it on Linux. A further motivation is the ability to customise everything, and tweak and tune everything, down to the compile options. One of the claims is that this allows the user to build a very highly tuned system. In many way’s it’s the perfect geek system, and as it celebrates its tenth birthday, it’s clear it’s doing something right!

I’ve also found Gentoo documentation to be amongst the best on the internet, and the gentoo users and maintainers I’ve met at conferences are invariably highly skilled and pleasant.

I’d say if you have the time to fiddle, Gentoo could be a lot of fun. I’m a pragmatist, and like to be able to install and remove packages in seconds, so a binary based distribution suits my purposes better.

OpenSolaris

I really like Solaris. In many ways it can deliver on its claim to be “The most advanced OS in the world”, boasting the world’s best and most advanced filesystem (ZFS), one of the finest observability tools in the industry (dtrace), cutting edge OS features (predictive self-healing and secure execution) and a range of three powerful virtualisation technologies (Zones, Xen and Virtualbox). At Atalanta Systems we readily deploy and support Solaris on both intel and Sparc, with a high degree of success.

Unfortunately, Solaris lacks a number of features we come to expect as standard, for example there’s no CIFS/SMB support, and it entirely lacks an integrated, modern, networked package management system. There are ways around the packaging problem - I’ve used NetBSD’s pgksrc on Solaris, and there are a few projects which use an apt-like tool to provide some of the packages you’d find in the Linux world. I’ve also recently discovered Portaris - a port of the Gentoo portage system to the Solaris system.

However, strictly speaking, standard Solaris isn’t quite open source. It’s freely available, but you need to buy a support contract to get patches, and there’s no CentOS equivalent.

But, over the last five years or so, Sun has been manoeuvring to open source most of (and ultimately all of) their code base. Beginning with dtrace, and accelerating in 2007 with the successful poaching of Debian founder Ian Murdock, and the release of OpenSolaris in 2008, they’ve taken huge steps.

Open solaris is fully open source, it has up-to-date tools, and an excellent packaging system. It works very well on the desktop, with all the usual requirements readily available. However, like Fedora and Ubuntu it’s on a fast, regular release cycle, and can’t really be considered stable or fit for production.

If you want to find out more, there are OpenSolaris user groups in most big cities. I’ve been to several of the London ones - they’re full of interesting people, and Sun provides free food and drink!

I’ll be continuing to keep my eye on OpenSolaris - when and if they provide a stable LTS-like edition, it could well be a force to reckon with.

*BSD

Of course Linux isn’t the only free and open source operating system - the *BSD family have actually been around even longer, but just aren’t as fashionable or as well-known. With its heritage in the earliest years of Unix, the *BSD family, are, as you might expect, highly mature and stable. There’s a perception that they’re harder to use, and more old-fashioned, but I find this to be a bit of a myth.

FreeBSD

FreeBSD is probably the most popular and easiest to get going. It’s the most commercially recognised, and has a fine body of documentation and a responsive online community.

Traditionally the system is small, lean and very fast. Packages can be installed as pre-built binaries, or using the standard BSD-style ports system.

FreeBSD is a much unappreciated backbone of the internet. I’ve run a number of sites on it, and it’s absolutely rock solid, reliable, secure and performant.

In addition to these benefits, FreeBSD also supports Sun’s ZFS and Dtrace, and with Jails has an equivalently lightweight and powerful container-based virtualisation solution.

Last time I used FreeBSD on the desktop, it felt like it was lagging behind Linux. Setting up X was a fairly manual task, and the process of getting the usual flash and java plugins was tedious. Likewise, support for proprietary equipment - particularly wireless chipsets and graphics cards is not as pervasive. I have no idea of the state of 3d desktop experience.

NetBSD

NetBSD aims to be really quite incredibly portable. I’ve run NetBSD on Alpha, SGI, Sparc, Zaurus, and 32 and 64 bit intel platforms. For me this has largely been for entertainment, but I could conceive of a situation in which a person wanted to make use of some esoteric hardware and run a first rate Unix on it. Particularly on the Sparc platform I’ve also found it to be blisteringly fast.

A big advantage of NetBSD is its excellent pkgsrc system. Similar in concept to the FreeBSD ports system, it again is highly portable - I’ve run it on Solaris and Slackware, and also use it on my macbook to provide up-to-date opensource packages.

OpenBSD

I have very little experience of OpenBSD. It has a reputation for being highly secure, and is popular for routers and firewalls on account of its excellent built-in PF firewall/routing software.

Most of the advantages and disadvantages of FreeBSD apply, only there are fewer packages, and the system is not optimised for speed in the same way.

Conclusion

I hope you’ve derived some benefit from this little run down of my experiences and, to an extent, prejudices, with the wide range of open source operating system options. In general I can make the following recommendations:

If you require commercial support, I would use Red Hat every time. Most of the time, I would use CentOS in an environment where there is already some Red Hat adoption. I also tend to use CentOS in new, large environments, because my puppet modules and packaging expertise become reusable.

I can also recommend Debian for servers, especially for single server deployments, and where you want to throw something together quickly. Debian is also well supported by puppet.

I think FreeBSD is definitely worth playing with, especially if you don’t already have significant Linux experience, and are prepared to try something a little different. I’ve not deployed it in a large scale environment, and while it is supported by puppet, I have no experience of doing so. Keep your eyes on OpenSolaris - the potential is there for this to be a world beater.

For a workstation, I would chose between Ubuntu LTS and Debian, with Fedora bringing up the rear.

While I do adore Debian, for the every day user, Ubuntu has the benefit of being entirely painless to set up - out of the box, pretty much everything works. In a development environment, I would then run virtual machines using KVM, VMware or VirtualBox, matching the OS of my deployment platform. It also has the benefit of being reasonably up-to-date, and predictably stable.

While Fedora is excellent, I don’t feel the need to stay on the bleeding edge, and I don’t want to have to rebuild my machine every six months.

For those who are interested, at the time of writing, the Atalanta Systems infrastructure runs on a mixture of Debian stable, Solaris 10 and CentOS 5. My Linux laptop runs Ubuntu LTS, and my workstation runs CentOS 5. Most of my daily work is carried out on MacBook Pros with VWware Fusion virtual machines running Debian and CentOS. CentOS and Red Hat are the most represented amongst our clients, with Solaris, Debian and FreeBSD also in use in a number of places.

DNS Zone File Fun with Python and Emacs

2009-11-06T18:03:44Z

Sometimes we’re faced with a boring, manual, labourious job which really needs to be done, will take a fairly long time, and be pretty unpleasant.

Whenever I’m faced with something like this, especially if it involves text, I try to make it interesting by setting myself the challenge of writing a script and/or using my editor to do the job faster than had I done it manually.

In this case, I had two problems. Firstly. I had just made about 50 entries into a reverse DNS zone.

These look like this:

1 IN PTR router.nso

Having done this manually, and rather slowly, I then had to make correpsonding forward entries, of the form:

router in A 192.168.1.1

My first thought was to do it in emacs with re-search-forward and replace-match, but this wasn’t easy to do, so my next plan was to write a python script that does it.

for entry in open("reverselist","r").readlines():
  tokens=entry.strip().split()
  ip=tokens[0]
  name=tokens[3]
  hostname=name.split(".")[0]
  print "%s IN A 192.168.1.%s"%(ip, hostname)

This takes an input file of a list of reverse zone entries, and spits out forward entries. It’s very simple - first we tokenise the input file:

'1  IN PTR router.nso'.strip().split()
['1', 'IN', 'PTR', 'router.nso']

We then take the first and fourth (python counts from 0) tokens - the last part of the IP, and the hostname, and take off only the short name:

>>> ['1', 'IN', 'PTR', 'router.nso'][0]
'1'
>>> ['1', 'IN', 'PTR', 'router.nso'][3]
'router.nso'
>>> ['1', 'IN', 'PTR', 'router.nso'][3].split('.')[0]
'router'

Then we just print them out again in the order we want.

I was then faced with my second problem. The (large) forward zone file is horribly and inconsistently formatted. Adding entries is always a pain, because if you just hit tab a few times you’ll never line up with the rest of the entries. Tidying it up has been on my todo list for a while, so I decided now really was the time to use the power of emacs!

So my task was how to convert various lines such as:

nst-xp-ie8	IN	A	192.168.20.236
svn		IN	CNAME	cap-svn

with their various spacing issues into a consistently spaced file.

In order to do this, we make use of the emacs regexp-replace function. This enables us to describe a pattern to match, and then defines how to match it.

Emacs regular expressions are a lot like sed’s, so I got there pretty quickly.

The first pattern to match is:

\(\w+\)\s-*\(IN\)\s-*\(A\)\s-*\(\S-+\)

\w+ means: match any word-constituent character
\s- means: match any white space
\S- means: match anything that is NOT whitespace
\(…\) is a construct which stores the match inside the brackets, so we can use it again later.

So this says:

Match the hostname, IN, A and the IP, and all the space inbetween; store the hostname, IN, A and the IP in variables.

The replacement is much simpler:

\1^I^I\2^I\3^I\4

\1, \2 etc are the variables, holding the matched hostname etc, and ^I is the tab character.

Simple really, but more fun, and done in less time than doing it manually.

Building puppet and facter RPMs for CentOS or RHEL

2009-11-06T17:03:44Z

In the first part of his excellent Puppet tutorial, John Arundel suggests that, given the speed with which Puppet develops and changes, and to keep things simple, Puppet should be installed from source. While for one box, this may be true, I tend to the view that you should use the native package manager wherever possible. In the case of CentOS or Redhat (the most common platforms for Puppet users, our survey says) , this means building RPMs. This article shows you how to build reusable puppet and facter RPMs, to make your life easier when you graduate from managing puppet on one server, to controlling a whole datacentre.

Set up an rpmbuilding environment

There are two things to consider here. Firstly, it’s not considered good practice to build RPMs as root, and secondly, RPM by default will place built RPMs in /usr/src/redhat. For this reason, it’s a good idea to create a dedicated user, and an area for building packages.

# useradd -m rpmbuilder
# su - rpmbuilder
$ for dir in RPMS SRPMS SPECS SOURCES BUILD; do mkdir -p redhat/$dir; done
$ mkdir redhat/RPMS/noarch
$ mkdir redhat/RPMS/x86_64
$ for x in 3 4 5 6; do mkdir redhat/RPMS/i$x\86; done
$ echo "%_topdir /home/rpmbuilder/redhat" > .rpmmacros

Get the facter and puppet sources

$ cd redhat/SOURCES
$ wget http://reductivelabs.com/downloads/puppet/puppet-0.25.1.tar.gz
$ wget http://reductivelabs.com/downloads/facter/facter-1.5.7.tar.gz

Stage the spec files and any patches

The heart of the RPM building process is the spec (short for specification) file. This is a text file which provides instructions to the rpmbuild command on exactly how to build the package. This includes names, versions, compile options, dependency information and any other information you might like to know.

Both facter and puppet tarballs ship with spec files for building RPMs - we need to get them and put them in the SPECS directory.

$ tar xzvf puppet-0.25.1.tar.gz
$ tar xzvf facter-1.5.7.tar.gz

The spec files live under the conf/redhat directory. At the time of writing, you would type:

$ cp facter-1.5.7/conf/redhat/puppet.spec ../SPECS
$ cp puppet-0.25.1/conf/redhat/puppet.spec ../SPECS

It’s a principle of RPM development that we always use pristine sources. However, sometimes, usually for distrubution-specific reasons, we need to apply a patch to the pristine code. The spec file will contain details of whether this is needed.

Check if there are any patches required:

$ grep ^Patch ../SPECS/facter.spec
$ grep ^Patch ../SPECS/puppet.spec
Patch0:         rundir-perms.patch
$ find . -name rundir-perms.patch | grep redhat
./puppet-0.25.1/conf/redhat/rundir-perms.patch

The patch file needs to be placed in the SOURCES directory, alongside the pristine source tarball.

$ cp ./puppet-0.25.1/conf/redhat/rundir-perms.patch .

At the time of writing, the facter spec file is out of date - the version number is for an old version. You can verify this:

$ grep ^Version ../SPECS/facter.spec
Version: 1.5.5

We need to up this to 1.5.7:

$ sed -i '/^Version/s/\.5\.5/\.5\.7/' ../SPECS/facter.spec

Strictly speaking, we should also make an addition to the changelog. If you feel so inclined, open the file in an editor, and find the %changelog section, and add an entry a bit like this:

* Fri Nov 6 2009 Stephen Nelson-Smith <me@sekrit.com> - 1.5.7
- Update to 1.5.7

Save the file, and we’re done.

Make sure we have the requirements

In order to build an RPM we need to be sure we have the prerequiste tools and libraries availale on our system. For something as small and simple as puppet and facter, this is trivial. If you start building bigger packages, with dependencies that in turn have dependencies, you’re probably going to find benefit from using Mock - the RPM building tool used by the CentOS project to maingtain the entire repository.

$ grep ^BuildRequires ../SPECS/facter.spec
BuildRequires: ruby >= 1.8.1
$ grep ^BuildRequires ../SPECS/puppet.spec
BuildRequires:  facter >= 1.5
BuildRequires:  ruby >= 1.8.1
$ rpm -q ruby
ruby-1.8.5-5.el5_1.1

OK, we’re going to have to build facter, then install it so we can build puppet.

Build facter

$ cd ../SPECS
$ rpmbuild -bb facter.spec

Look for the line that says something like:

Wrote: /home/rpmbuilder/redhat/RPMS/x86_64/facter-1.5.7-1.x86_64.rpm

Now, become root, and install facter:

# rpm -Uvh /home/rpmbuilder/redhat/RPMS/x86_64/facter-1.5.7-1.x86_64.rpm

Build puppet

# su - rpmbuilder
$ cd redhat/SPECS
$ rpmbuild -bb puppet.spec

You should see something like:

Wrote: /home/rpmbuilder/redhat/RPMS/x86_64/puppet-0.25.1-1.x86_64.rpm
Wrote: /home/rpmbuilder/redhat/RPMS/x86_64/puppet-server-0.25.1-1.x86_64.rpm

NB when you come to install these rpms, you also need augeas-libs and ruby-augeas. Both of these are available in EPEL. This is a new dependency since 0.25.x.

Congratulations, you’ve built facter and puppet rpms, which you can now distrubute across your envirionment.

Setting the MySQL client as a user's shell

2009-11-05T15:03:44Z

Sometimes corporate firewalls can be a real headache for developers. We had a situation today in which we needed to provide read-only access to a MySQL database, but had only ssh to the machine. We have a very strict user policy, so weren’t prepared to provide a shell account.

One quick and cunning way to provide MySQL monitor access without opening up the firewall, or creating shell accounts is to create a user with /usr/bin/mysql as their shell.

Create a user with /usr/bin/mysql as the shell:

# useradd -m -s /usr/bin/mysql dev

# grep dev /etc/passwd

dev:x:500:500::/home/dev:/usr/bin/mysql

Drop a .my.cnf into the new user’s home directory

[client]

user=dev

password=m3g4s3cr3t

Create a user and grants in the mysql database

mysql> CREATE USER ‘dev’@’localhost’ IDENTIFIED BY ‘m3g4s3cr3t’;

mysql> GRANT SELECT ON *.* TO ‘dev’@’localhost’;

Now when a user connects to the machine as the dev user, over ssh, they drop straight into a mysql monitor, with restricted privileges.

Many thanks to James Sheridan for this tip.

Getting a recent Python on Red Hat or CentOS

2009-11-02T21:03:44Z

The IUS Community Project is a free and unsupported effort by the RPM engineers at Rackspace Hosting to provide recent editions of popular open source tools. Packages include PHP, MySQL and Python. Rackspace have made up-to-date packages available to their Enterprise customers for some time, but recently the idea of making packages available as “The Fedora of Rackspace RPMs” has emerged. Packages make the journey from the Testing IUS repository, to the Stable IUS repository, until eventually they make it into the Enterprise edition, which is only available to paying rackspace customers. IUS is not a Rackspace service - it’s a volunteer project. If you want to get involved, read the project FAQ here.

Naturally, using these packages carries with it the usual warnings - these packages are not supported by Red Hat, or the CentOS Project. They are bleeding edge, and for production use it is always best to use your vanilla distribution packages. However, if you have to upgrade, this is probably the best way to do it. But if it breaks, you get to keep the pieces.

Getting Set Up

First we need to obtain and install two RPMs - these provide entries in /etc/yum.repos.d for the Fedora EPEL repository, and the IUS repository.

$ wget http://dl.iuscommunity.org/pub/ius/stable/Redhat/5/x86_64/epel-release-1-1.ius.el5.noarch.rpm
$ wget http://dl.iuscommunity.org/pub/ius/stable/Redhat/5/x86_64/ius-release-1-2.ius.el5.noarch.rpm
$ yum localinstall epel-release-1-1.ius.el5.noarch.rpm ius-release-1-2.ius.el5.noarch.rpm --nogpgcheck

We then clean the yum repository metadata:

$ yum clean all

Now we can see new packages:

# yum list available python26*
Excluding Packages from CentOS-5 - Updates
Finished
Excluding Packages from CentOS-5 - Base
Finished
Available Packages
python26-debuginfo.x86_64                2.6.2-1.ius.el5        ius             
python26-devel.x86_64                    2.6.2-1.ius.el5        ius             
python26-elixir.noarch                   0.6.1-1.ius.el5        ius             
python26-httplib2.noarch                 0.5.0-1.ius.el5        ius             
python26-jsonschema.noarch               0.2a-1.ius.el5         ius             
python26-lxml.i386                       2.0.11-1.ius.el5       ius             
python26-lxml-debuginfo.i386             2.0.11-1.ius.el5       ius             
python26-memcached.noarch                1.43-5.ius.el5         ius             
python26-mysqldb.x86_64                  1.2.3c1-1.ius.el5      ius             
python26-mysqldb-debuginfo.x86_64        1.2.3c1-1.ius.el5      ius             
python26-nose.noarch                     0.11.1-1.ius.el5       ius             
python26-setuptools.noarch               0.6c9-1.1.ius.el5      ius             
python26-simplejson.i386                 2.0.9-1.ius.el5        ius             
python26-simplejson-debuginfo.i386       2.0.9-1.ius.el5        ius             
python26-sqlalchemy.noarch               0.5.5-1.ius.el5        ius             
python26-test.x86_64                     2.6.2-1.ius.el5        ius             
python26-tools.x86_64                    2.6.2-1.ius.el5        ius

Now let’s install it and have a look:

# yum install python26
# rpm -ql python26 | grep '/usr/bin'
/usr/bin/pydoc2.6
/usr/bin/python2.6
/usr/bin/python2.6-config

Let’s give it a try:

# python2.6
Python 2.6.2 (r262:71600, Sep 26 2009, 03:35:59) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> blog_entry = {}
>>> blog_entry['title'] = 'Portable Serialisation with JSON'
>>> blog_entry['categories'] = ('programming', 'json', 'python')
>>> blog_entry['published'] = 'Yes'
>>> blog_entry['author'] = 'Stephen Nelson-Smith'
>>> with open('blog.json', mode='w') as file:
...     json.dump(blog_entry, file)
... 

# cat blog.json 
{"published": "Yes", "author": "Stephen Nelson-Smith", "categories": ["programming", "json", "python"], "title": "Portable Serialisation with JSON"}

We’ve just tested two new features - a built-in JSON library, and Python’s new ‘with’ statement. Congratulations, you now have a modern Python.

Agile Sysadmin

London DevOps Meetup

A short introduction to Chrome OS

About the author

How to build 100 web servers in a day

Strategies for configuration management: imaging and automation

The golden image

Automation tools

Combined strategies

Further reading

About the author

Advanced Dependency Management with Yum Shell

How to print every nth line of a file

The Atalanta Systems Guide to Open Source Operating Systems

Red Hat

CentOS

Fedora

Debian

Ubuntu

Slackware

Suse

Mandriva

Gentoo

OpenSolaris

*BSD

FreeBSD

NetBSD

OpenBSD

Conclusion

DNS Zone File Fun with Python and Emacs

Building puppet and facter RPMs for CentOS or RHEL

Set up an rpmbuilding environment

Get the facter and puppet sources

Stage the spec files and any patches

Make sure we have the requirements

Build facter

Build puppet

Setting the MySQL client as a user's shell

Getting a recent Python on Red Hat or CentOS

Getting Set Up