Building a Devops Team (Part One)

Another entry from the archives. I had the honour to work alongside and mentor Brian Henerey, for Sony Computer Entertainmenr Europe, in 2011. It was a fascinating time for us, deeply fulfilling and rewarding. This post is the first of two that Brian wrote in 2012, talking about the strategies he employed as he built out an effective engineering team.

Background

I've had 3 roles at Sony since joining in August 2008. Nearly a year ago I took over the management of the original engineering team I joined. This was a failing team by any definition, but I was excited about the opportunity to reshape it. I knew the remaining team was deeply unhappy and likely to quit at any moment, so I had a few immediate goals:

  • Hire!
  • Keep people from quitting.
  • Hire!

Side story: I stumbled on one important objective I didn't list however. Keep customers happy. It doesn't matter how awesome you think your team can be if no one wants to work with you based on past experiences. I didn't appreciate how much a demotivated employee could jeopardise customer relationships by virtue of not caring. It has taken me months to restore trust with one customer. I've heard a story about a manager offering employees £500 to quit on a regular basis. I think that probably has some practical problems, but its a tempting idea to cull the unmotivated.

I come from a long background of small/medium size enterprises. It has been a challenge adapting to a large corporation, but I don't think there's much unique to Sony about the anti-Devops patterns I've encountered. I know several people in small companies who says they've been practicing Devops before there was such a word and I completely agree. The trouble of silos, bureaucracy, organizational boundaries, politics, etc, seem pretty common in larger businesses though. I can't speak to how to create a Devops culture across a large organisation from the top down, but I've been working really hard to create one from the inside.

The beginning

A year ago I'd never heard of the term Devops. If you're in the same boat, it is easy to find a great deal to read about what Devops is:

And what it is not:

However, I suspect some people will have trouble finding the read-worthy gems amongst all the chatter. Here's a good place to get started: getting started with devops. The gigantic list of Devops related bookmarks compiled by Patrick Debois shows why you may not want to try and read everything: devops bookmarks

If you're in the know already and Devops resonates with you, and you want to build a team around the concept, here's how I went about it.

Networking

The terms Devops didn't really take shape for me until I started to talk about it with others. Fortunately, London has a really active Devops community so I've had ample opportunity. The tireless Gareth Rushgrove organises many events, and The Guardian is a frequent host. I've been to sessions discussing Continuous Integration, Deployments, Google App Engine, Load Balancers, Chef, CloudFoundry, etc. I've found people to be incredibly open about technology, processes, culture, difficulties and successes they've had.

While Devops is of course about more than technology and tools, I personally have found Devops to be an excellent banner under which to have really interesting conversations. Having a forum which brings people from diverse backgrounds together has helped me shape my own internal understanding of what Devops should be about.

I felt a bit of an imposter going to the initial London Devops meetups because I was so keen on recruiting. However, the quality of the discussions has been so good I eagerly anticipate each upcoming meetup even though I'm no longer hiring. I've also discovered that half the attendees are also hiring. It's a Devopsee's market.

Result!: I met and subsequently hired Stephen Nelson-Smith from Atalanta Systems. (He's @LordCope on twitter, and the author of agilesysadmin.net)

Working definition of Devops

If you're going to hire people with Devops in mind, its good to have a working definition. I like the pillars of Devops (CAMS) put forth by John Willis: what devops means to me

  • Culture
  • Automation
  • Measurement
  • Sharing

SMAC might have been a better acronym, but I'll go with CAMS.

A Devops job spec

I don't think Devops is a role, though I've seen jobs posting for such a thing. I only mentioned that I was looking for someone 'Devops-savvy', and later changed it to 'Devops-minded' or something similar. The job posting expired and I'd have to dig it out, but R.I.Pinearr described in on Twitter as the 'perfect devops job posting'. I'm pretty keen on revising a job spec until the requirements are only things I actually require and can measure against. Saying that, how to write a job spec is way outside the scope of this post. To summarize, I was looking for:

  • problem solving skills
  • 'can do' attitude
  • good team fit (really hard to quantify)
  • a broad set of skills (LAMP, Java, C++, Ruby, Python, Oracle, Scaling/Capacity, High-Availability, etc, etc)

My team works on a ton of different technology stacks, and the landscape is constantly changing. Its a techie-dream job, but the interpersonal skills are the most important.

Recruiters

I strongly believe in giving recruiters a fair bit of my time. I've seen many people be rude to recruiters, ignore them, etc, and then wonder why they don't get good candidates through. I'm quite keen on engaging the recruiters, explaining the role I'm trying to fill thoroughly, and having the occasional coffee or beer with them. Feedback is of course vital to candidates, and I try to give it honestly and quickly, letting the recruiter worry about sugar coating things.

CV selection

This is tough. I regularly get CV blindness where everyone starts to look the same. And generally ill-suited. I try to remember there are human beings on the other end and force myself to have concrete reasons why I'm rejecting someone. Talking to a recruiter about this helps me be concrete.

First interview - remote technical test

This is where things get interesting! I don't know if this is unique to London, but I've had a LOT of candidates from other countries apply to join this team. If someone has a good CV and the recruiter vouches for their English language skills, I developed a great screening test which can be conducted remotely. This saves a trip to London + hotel, and I can end it promptly if things aren't going well. Here's how it works:

  • I email the candidate/recruiter a url to an ec2 instance that I spin up on the day about 20 minutes before the interview.
  • The instance is running a web browser which contains instructions for the test. These only state that the candidate will need a terminal such as Putty if they're on Windows.
  • At the arranged time I phone the candidate. I explain that there will be two tests. The first is a sys admin task which will be time bound to 20 minutes. The second is a programming task which they can use the remainder of the time to complete. The call will end after 1 hour.
  • I explain the rules: They are to perform all of their work on the ec2 instance. They have a test account/password, and sudo root access. They can use any resources they want to solve the problems. Google, man pages, libraries are not only fair game, but fully expected.
  • I explain what I want from them: They need to talk to me, tell me what they are thinking, and walk me through the problem solving process. I'm far more interested in that dialogue than whether they solve either problem I give them.
  • I also add that we're using Screen, and I can see everything they type.
  • I swap the index.html with the complete instructions in place, make note of the time, and let them begin.

The problems

  1. Its really quite simple: install Wordpress and configure it to work properly. The catch is that we install mysql first, break it, and then watch as candidates wonder what the heck is going on. For an experienced sysadmin this is child's play. I tended to interview people with stronger development background and less familiar installing applications. I could tell almost immediately how well someone knew there way around a Linux system. It was interesting to see what kinds of assumptions people made about the system itself (I never mentioned the OS that was running. Several just assumed Ubuntu.) Some people read instructions, some don't. I give people the mysqladmin password, but some people search on how to reset a lost password because they didn't read what I gave them. I had one guy spend 10 minutes trying to ssh to http://ec2....... I gave him a pass on nerves, but he continued to suck and I ended it soon there after. He blamed language barrier (Eastern European), and said if only I had been more clear to him. If I can't communicate with him, I think that's a pretty big problem and it doesn't really matter who's fault it is.

  2. We provide sanitized Production Tomcat logs for a real application we support and ask the candidate to write a log parsing script in a language of their choice. We want the output of the script to show methods calls, call counts, frequencies, average and 90% latencies. Our preference is Ruby, but they can do it however they'd like. I had one candidate choose to implement this in Bash and was writing some serious regex-fu that I had no idea how it worked. He got stuck however, and I couldn't help but ask as he claimed to be a Ruby developer why he didn't do it in Ruby, which was my stated preference. He started over in Ruby and did okay. Depending how much time was spent on problem 1, this part of the interview is really boring for me. I stay on the phone in case they have questions, I ask them to explain their approach before they begin coding, but then I just start checking email/etc. After 60 minutes total is up, I explain to the candidate that they can continue working on the coding task as long as they need and to send me an email when they've finished. I get off the phone however, stating that we'll give them feedback as soon as we've reviewed the code they submit and explain the next steps.

Results

I put several candidates through this process. In the beginning of creating this test, I'd have a couple members of my team on this call as well, but we found this too time consuming and a bit intimidating to certain candidates. Timeboxing problem 1 was a HUGE improvement, and once Stephen Nelson-Smith was on board I had someone better than me at evaluating the Ruby code. We all felt this test process was extremely revealing of candidates skillsets and I highly recommend it.

One of my favourite candidates conducted this interview on a laptop in the shared wifi area of a crowded and noisy London hostel. In the background were screaming people and overbearing Christmas music. He was able to tune out the distractions and nailed both problems with ease, and got major bonus points for doing so.

Round 2 - Face to face interview

Round 2 actually has a few parts:

  • Coffee/lunch/dinner informal chat up to 1 hour in length. I explain what I'm looking for; they can talk about themselves; we can find out if we have a good match.
  • Hypothetical whiteboard problem solving exercise: You receive a call saying customer goes to http://yoursite.com and gets a blank page. What do you do next? We can improvise a bit here on what the actual problem is, but we're hoping to learn two things: How does this person approach problem solving? What level of architectural complexity have they been exposed to?
  • 2 hours of pair programming with a member of my team. This is usually a real bit of work that needs doing. It could be writing a chef cookbook, or a cucumber test, etc. We want to learn what its like to work closely with this person. My team pair programs often. Do we want to pair with this person day in / day out?

Round 3 - my boss + any member of my team who hasn't met the candidate yet.

  • This is generally very open, though my boss has her own techniques for evaluating people.

Its very important to me that everyone on my team have a voice. I was quite keen on one candidate, but when one of my team member's voiced vague concerns about the person's team-fit, we all stopped and took it on board. We rejected the candidate in the end because once the first doubts were out in the open, other people's concerns started to be raised as well. I recognised that I was a bit too keen to hire someone to fill a pressing need and am glad how things worked out..

A GREAT candidate/hire

One of my favourite hires not only does he know C, Java, and Linux, but wrote a sample Ruby application because he knew we were looking to hire Ruby skills within the team. His app worked out the shortest path between tube stations, though only in terms of number of stops, not time travelled. This initiative told me a lot about him, and its been 100% the same since he joined the team. Eager to learn and try new things. Any problem/task put in front of him is 'easy'. My only trouble is he tends to consider problems solved when he's worked out in his head how he will solve it. This is a bit of a joke really. I accused him the other day of declaring checkmate on a task because he was so confident it would be completed in his next 7 seven steps.

Beyond hiring

Now what? Well, hiring the right people is HUGE. We celebrated each hire, as opposed to the typical 'leaving drinks' when people move on. How I manage the team will be a future blog post (I hope), but I'll add one quick comment. Hiring people according to the vision I had means that I am held accountable as well. Whenever I find myself explaining that the reason for a decision I'm making is 'politics', I know I have to change.

About the author

Brian Henerey is now Chief Information Security Officer (CISO) at OneMain Financial, in Chicago, and also heads up technical operations for all cloud (AWS) infrastructure.

Show Comments