Interviewing a sysadmin: questions and good answers

Questions I used to interview a Linux sysadmin in October 2013, and what I was listening for. A developer hiring for ops, so I wrote down what good looked like.

General

Preferred language for shell scripts? Bash, because it's on everything and you don't want to be installing packages on an old server. A bit of Perl or Ruby is a bonus.

How would you improve a monitoring script you wrote? A good answer: order the output by priority, summarise it in one email, write it for the business user. A better one: throw it away and use an off-the-shelf monitoring tool.

How do you manage system passwords and other secrets? "Always impossible to do properly as soon as more than one person is involved." Then: SSH keys wherever possible, one key per person per machine, so any one can be revoked. A password manager for the rest.

Running servers

Looking for:

• Package management and patching.
• Alerting on disk space, log rotation.
• Flagging logins and checking logs for anything odd.
• Monitoring and graphs: disk, IO wait. "Anything the computer will give you, get it graphed."
• Configuration management, so servers are built from config, not by hand.

The line I wrote down:

Legacy servers are like pets, you keep them going for as long as you can.

Security and networks

External firewalls, a segmented network (a DMZ), an audit log of logins with unusual activity flagged, and if someone does get in, delete the machine. Then TCP/IP, load balancing and failover.

Process

Tell me about a project where a process was defined and then followed. Looking for: there actually was some documentation, it was followed methodically with success criteria, and problems were fed back at the end.

How would you back up a web server? The trade-off between daily and weekly: speed to restore against how much data you store.

The scenario

A mission-critical web server goes down. How do you handle it?

What I hoped for:

1. "We'd already know, because monitoring picked it up."
2. "No sweat, it's load balanced now; I did that last week."
3. A written process: who gets alerted, by SMS or phone, the escalation route, what to try, who to call to spin up a new machine.

Someone used to a big ops team around them can struggle with this one, and in a small company there's no team behind them.

These are also the questions to ask whoever runs an app you've just taken over: see the inherited web app checklist.