I found myself summarizing this concept AGAIN on IRC, and figured it was time I actually wrote it down.

Over the years, I’ve found that there are primarily two kinds of Systems administrators[1] : the Scottys and the La Forges. Anyone who is familiar with Star Trek[2] will get the reference. But for those who don’t, let me explain, and then I’ll talk about the pros and cons of each.

Scotty[3] is the Systems Administrator who feels the need to make everything into a miracle. All jobs are extremely difficult or just plain impossible, but they can get that task done. Scotty will tell you it’ll take four hours and won’t be that impressive, then custom builds a solution in two hours with bells and whistles. Server hardware fails at 3am? Scotty emails you that it’ll be tomorrow before it can be working, and then has it back up before 8am.

Scotty, in essence, under promises and over delivers, usually with one-off or custom solutions.

La Forge[4], on the other hand, is the System Administrator who makes everything routine. Everything, from the simplest task to the most difficult, gets done in a timely manner with little or no grandstanding. If La Forge says it’ll take two hours, it takes two hours, and is delivered to spec. La Forge can think on his feet, is cool under pressure, and keeps things running smoothly, even in an emergency. Server down at 4am? The system is redundant, so no-one even knows it went down. Need secure access to a customer site? La Forge builds it so that it can be done the exact same way for the next 50 sites.

La Forge, in essence, delivers to spec and on time, and plans for contingencies.

There are pros and cons to each. And each will excel in different environments.

In a small company, a Scotty will do well. Resources are tight, miracles need to happen regularly, and for the first year or so, one-off solutions aren’t a bad thing. The down side to a Scotty is twofold. One, the miracles they’ve worked for the last year or so can only be supported by Scotty. Sooner or later there are five (or more) different solutions to achieve the same results, each one better than the last, and all five solutions are in production. Documentation is sporadic – either over-complete or missing entirely.

Scotty will either get phased out or leave a small organization when it grows past a certain point. In larger companies, Scotty is either part of an R&D group, or a minor player in a much larger IT department, where these sorts of one-off or customized solutions have a low impact on the company as a whole.

La Forge, on the other hand, excels in medium to large companies. With a focus on stability, repeatability, and scalability, a La Forge is what a lot of people think of when they hear the words “IT Manager.” he can do the job, he has talented staff do delegate to, and has the knowledge to keep big, complex systems running smoothly. La Forge has a plan for almost every contingency, and if they’re hit with something new, you can be sure the solution will be documented, and the exact same solution used next time it comes up. La Forge will make sure that documentation is complete, up-to-date, and that the organization can go on without them.

In Medium to Large companies, La Forge usually rises to either team lead of department manager. In smaller organizations, they often come in after a Scotty has left, or are hired to manage a Scotty (or group of Scottys). A La Forge won’t fit very will in a startup.

Scottys can become La Forges and vice-versa. It’s not an easy transition, and generally a Scotty will transform into a La Forge as time goes on – as the company grows, as they mature as a sysadmin, or as the needs of the organization (big or small) change. If not, a Scotty will move on to the next job (voluntarilly or when they are no longer a “good fit” organizationally).

A La Forge can also turn into a Scotty if the job demands it. But this transition is usually much more painful, and they will do everything they can to re-assert their La Forge tendancies as the job goes on.

As for me? I’ve been both. And in the end, I MUCH prefer the La Forge style to the Scotty style. I get more sleep, my systems run better, and my customers are happier. Right now, though, I have to be a Scotty/La Forge hybrid, trying to convert a set of systems built by a Scotty into something scalable, managable, and repeatable. Something the next sysadmin after me can manage, maintain, and grow.

(Thanks to Scott and Ian for reviewing this post before publication)

[1] And I do mean primary – one of my reviewers of this post identified at least two other types. I’ll talk about those in a later blog post.
[2] Flame wars aside over which series was better, if you haven’t seen or heard of Star Trek, where have you been for the last 40 years?
[3] Chief Engineer Montgomery Scott
[4] Chief Engineer Geordi La Forge

So when I took this job, back in May of last year, I knew there would be some challenges. I was taking over after they were without a Linux admin for close to 8 months, the prior systems philosophy had been that each server was a unique and special hand-crafted snowflake, that the client sites are all unique and unpredictable, and that it has to be that way because, well, no two sites or internal uses are alike.

I call bullshit on all of that.

I have spent the last 7 months making that a scalable, standardized, and maintainable environment :

• As noted in a prior post, I’ve implemented a real monitoring solution.
• I’ve augmented that with scripts to gather additional data and display it in a friendly manner.
• I’ve implemented OpenVPN for client access[1]
• I’ve implemented both puppet and kickstart for building and deploying servers.
• I’ve been standardizing configurations as I go.[2]
• I set up naming schemes[3] for the next 3 years

And then there’s all the other bits and pieces that go with day-to-day systems administration. But those? Those are the milestones.

On my list for 2011 :

• Finish retiring the monitoring scripts that send emails from the remote servers[4]
• Implement a proper request tracking system
• Complete a cleanup of the internal wiki or migrate all the pertinent data to a new, clean one
• Implement a NAS or SAN, depending on what gets approved
• Server redundancy for the virtual infrastructure[5]

You can expect to hear about some of that as I go about it this year. And if you have suggestions or comments, I’m always open to hear how others are solving similar problems.

But for now? I’m off to kill webmin with extreme prejudice on all the company servers.

[1] To replace, and I am not making this up, a set of SSH tunnels to a set of SSH reverse tunnels
[2] Hey, you know what’s fun? 20 servers all with apache httpd & tomcat installed, and no two have the same version, configuration file layout, or installation location.
[3] When I got here, the servers were being named after Star Wars characters and locations. I am not making this up. The only thing more cliche I could come up with was the Matrix, Dilbert, or Star Trek. And I still have to fight the urge to yell “It’s a TRAP!” every time someone complains about server ackbar being problematic.
[4] Hey, you know what else is fun? Finding out that there is a script running to email you every minute to tell you that the mail relay is unreachable, and that the server can’t send email. *headdesk*
[5] I like to sleep at night, so I’ve come to grips with[6] the fact that, as of right now, if I lose any ONE of the ESXi servers I inherited, we’re dead in the water until it and all it’s VMs can be rebuilt.
[6] Where “I’ve come to grips with” = “I drink heavily to forget”