So when I took this job, back in May of last year, I knew there would be some challenges. I was taking over after they were without a Linux admin for close to 8 months, the prior systems philosophy had been that each server was a unique and special hand-crafted snowflake, that the client sites are all unique and unpredictable, and that it has to be that way because, well, no two sites or internal uses are alike.
I call bullshit on all of that.
I have spent the last 7 months making that a scalable, standardized, and maintainable environment :
- As noted in a prior post, I’ve implemented a real monitoring solution.
- I’ve augmented that with scripts to gather additional data and display it in a friendly manner.
- I’ve implemented OpenVPN for client access
- I’ve implemented both puppet and kickstart for building and deploying servers.
- I’ve been standardizing configurations as I go.
- I set up naming schemes for the next 3 years
And then there’s all the other bits and pieces that go with day-to-day systems administration. But those? Those are the milestones.
On my list for 2011 :
- Finish retiring the monitoring scripts that send emails from the remote servers
- Implement a proper request tracking system
- Complete a cleanup of the internal wiki or migrate all the pertinent data to a new, clean one
- Implement a NAS or SAN, depending on what gets approved
- Server redundancy for the virtual infrastructure
You can expect to hear about some of that as I go about it this year. And if you have suggestions or comments, I’m always open to hear how others are solving similar problems.
But for now? I’m off to kill webmin with extreme prejudice on all the company servers.
 To replace, and I am not making this up, a set of SSH tunnels to a set of SSH reverse tunnels
 Hey, you know what’s fun? 20 servers all with apache httpd & tomcat installed, and no two have the same version, configuration file layout, or installation location.
 When I got here, the servers were being named after Star Wars characters and locations. I am not making this up. The only thing more cliche I could come up with was the Matrix, Dilbert, or Star Trek. And I still have to fight the urge to yell “It’s a TRAP!” every time someone complains about server ackbar being problematic.
 Hey, you know what else is fun? Finding out that there is a script running to email you every minute to tell you that the mail relay is unreachable, and that the server can’t send email. *headdesk*
 I like to sleep at night, so I’ve come to grips with the fact that, as of right now, if I lose any ONE of the ESXi servers I inherited, we’re dead in the water until it and all it’s VMs can be rebuilt.
 Where “I’ve come to grips with” = “I drink heavily to forget”