Friday, April 29, 2011

Those Hot (and Helpful) Servers

It's been a rough week the past two days (from a work rant).

After all the normal crap at work, we had to deal with the Great Air Conditioning Debacle of 2011 (as opposed to the Great Air Conditioning Debacle of 2010). We almost fell out the door and poured ourselves into our cars. Fortunately the cars automatically drive us home or we wouldn't know how we made it (and half of us don't drink).

I rushed home to enjoy the lack of air conditioning there and work on my 2nd job. An hour into it my main job's boss calls: by the way, he tells me... it's 80 degrees in the server room.

There could be entire comedic routines written about obvious phrases, but I'll just use the phrase for now: "That's not good."

We discuss options, I suggest he locate some building people (yeah right, at 7pm). He calls back 10 minutes later to tell me he got some fans set up. One of the a/c units isn't functioning (go figure, it's the building's unit, as opposed to our in-wall cooler).

He calls a third time to tell me the temp is now 85 and maybe we should consider shutting down some servers. Righto - I told him I'd VPN in and take care of it myself.

After this it really got fun.

I'm using my main linux box for job#2, which features a brand new Windows virtual machine. Unfortunately because it's brand new, I haven't had a chance to load the VPN software so I can't use it to connect.

No problem - I fire up the seldom-used Windows box next to the linux box. As it comes up I realize I don't have the VPN software on that one either (although I thought I did). Hmm.... must be on the laptop.

You know where the laptop is, of course... it's upstairs. I've had this laptop for almost a year now and it's always in the same spot. Except today.

It's not really a problem having two jobs. The problem comes in when I have to do both of them at the same time.

I bolt upstairs to get the laptop. It's right where I left it. And the headphones I left with it have knotted themselves tightly around the latop in a death-grip.

Back downstairs with the laptop safely(?) booting up, I try to keep job#2 humming along smoothly. This will not happen. The pen I am using keeps leaping off the papers it's sitting on and trying to hide itself. It may even be trying to commit suicide by diving off the papers onto the cold cruel floor, but it won't get away for long. If I catch it running away I can always find it. If my wife uses it, I will never see it again.

While playing Save the Pen, I'm also noticing that my typing has gotten completely out of hand - worse than normal. Some people are dyslexic. Some just can't type. I'm a dyslexic typist. Every time I type`the' it comes out `teh' or `hte'. If I had a dollar for every time I typed `the' correctly, I'd have a dollar the next time it happens.

As bad as being a dyslexic typist is, my laptop has taken to playing games with me. If you have a touchpad, you know exactly what I'm talking about. The cursor will randomly move to a different line for no apparent reason (like it has three times while I have tried to type this very line). Oh yeah, I use an ergonomic keyboard except on the laptop, making it just that much more fun.

By this time (three hours later) the laptop has booted and I'm running the Windows virtual machine (yes, it's a linux laptop) with the VPN software. Only the VPN software doesn't want to come up. It wants to politely wait its turn til after the antivirus updates. It still hasn't learned that even Windows can multitask enough to update the antivirus and start up the VPN at the same time.

Once I convinced the VPN to come up and connect, I was securely connected to work #1, or so I thought. I mentally went over the list of servers and their IP addresses, realizing that I don't have all the new ones memorized yet. No problem- I just went and shut down the servers whose IP's I remembered.

Or so I thought.

There were a number of servers that would not let me connect. They returned pings but would not allow me to remote into them. One didn't even return a ping. Most interesting, especially with 85 degree and rising temperatures. And the continued efforts of a pen to shuffle off its mortal coil and join the choir invisible.

I went to get the server IP config document and realized it would be a somewhat difficult job, made moreso by the fact that I had just shut down the server that had the docs stored on it.

With a few stubborn servers left, I remoted to a different server and fired up the remote client from there. BINGO. The servers that refused to return a ping or let me remote in started returning pings and allowing me to remote in.

And I continued to type like a drunk dyslexic with a hangover and a bad case of nerves. My neighbors think there's some sort of madman living in my house, who is prone to the most egregious explosions of temper randomly. And that's in the winter with all the doors and windows closed.

Speaking of madman, I just felt something weird and looked down to find my phone's belt clip attached to the laptop's power cord. I could not explain how it got there for all the money in the world and I stopped trying a long time ago. My friend, a woman much wiser in the ways of the mystical than me, said I have a weird house. Stuff randomly leaps up for no particular reason (much like this #*@&ing cursor). When I went to put it next to my phone I noticed there was already a belt clip on the phone. This means that for the first time, something has actually helpfully appeared out of the ether, as opposed to disappearing into the ether. I won't even bother to think about this, as it could cause headaches the likes of which I have never endured.

There are still four servers that won't come down. One won't come down because I can't remote into it. Why? Because we discovered one day that whenever you remote into this blighter, it reboots. So we pretended it never happened and shut off terminal services. Ah, Windows, we love you.

As anyone who has ever used a computer knows, you can't simply shut down a Windows machine, no sir. 2000 Server wants to install updates and shut down by default. If you're not paying careful attention this is what will happen. Server 2003 isn't content to default to applying updates; it wants a reason for the shutdown. As if `BECAUSE I SAID SO' weren't good enough. I need to be second-guessed by an operating system from a state with even more rain than London. An OS that needs to be rebooted, unlike any other OS, such as linux.

And then there's the other server. It told me I wasn't allowed to log in because all the terminal server slots/licenses were being used. Neat trick, especially at 7:30pm, when there's no possible way anybody else could be using it. This turned out to be what Microsoft undoubtedly refers to as a Feature of terminal services (as opposed to Yet Another Bug). When people log out, the connection reads disconnected, but never bothers to go away. I figure this is yet another facet of Windows, the Helpful Operating System. It hasn't dropped the connection slot because it's helpfully holding it in case the person who dropped the connection wants to reconnect.

If you're following along at home, I'm using a linux laptop with a Windows virtual machine, opon which is VPN software that connects me to work. I have to remote desktop into a work server because certain other servers won't let me in. In the case of other servers, I have to remote desktop into a different server so I can access terminal services for a third server, kicking off already disconnected sessions so I can start a real session to stop the machine in the first place (at which point it asks me if I'm sure and what's the reason I'm turning it off).

Got it?

At this point I send a group text message to my team, alerting them of the problem and asking the first person in to bring up the servers, assuming we have attained server room temperatures below 85 degrees. If not, they're to shut down the remaining servers and tell everyone in the company to go pound sand.

Before I started my mad shutting-down of servers, I made certain to send an email to the entire company, letting them know that remote workers would be out of luck and that when they came in, some servers may still be down but will be up as soon as they can safely be brought back up.

The above is an act of pure optimism on my part. I do it partly because my boss asks me to and partly for mmy own amusement. What my boss fails to believe is that all of my messages to the company come with the Auto-Ignore flag attached to them. Thus no one reads them or even knows that I sent any messages in the first place. This has been documented, yet my boss still thinks I'm kidding. Sometimes I have other members of the team send out important messages in the hopes of them being read.

The other battle is that I'm not entirely certain the people at work can read. A quick visual survey reveals to even the most casual observer that we don't hire for looks. A somewhat longer stay at work reveals that we don't hire for brains either. One day perhaps we'll take a poll on what people think we do hire for. Some say it's a pulse. Othes say that a pulse isn't specifically ncessary so long as the body occasionally occupies some space in the vicinity of the desk. Not every day, mind you, but occasionally. The biggest sin is apparently not picking up your pay stub. Since we're required to have electronic deposit, we have to go to Payroll to pick up our stubs. If one doesn't pick up one's stub after a week or two, they go on Payroll's Bad List. After a few similar public humiliations don't smoke you out, they figure you either quit and neglected to notify them or the body is starting to smell.

Tomorrow morning will start predictably. People will amass in the general vicinity of their desks. A small percentage of them will actuall read the email about the servers. The large majority will immediately start screaming that their computer is broken. How have they come to this conclusion? Because they can't get to their documents. Where are their documents? On the file server that is down.

"WHY IS THE FILE SERVER DOWN?" they will scream, as if on the rack.

Didn't you read the email?

WHAT email??

MIS sent out an email. The servers were overheating.

Oh, yeah, I just knew that $&#@ing MIS was behind this. MIS sucks.

[we won't go into the logical impossibility surrounding this, lest we develop a headache of unfathomable proportions]

Even though we will all get there early to lessen the impact of this, there will still be massive whining about people not being able to do their jobs. We will be wondering what all the fuss is about... after all, the internet is still up and they can madly go downloading their Beyonce videos. Since this is more or less what they do all day anyway, it will be very confusing for us.

1 comment:

  1. My HVAC guy claims that somebody should change the filters for the A/C once in awhile.

    This will be the "turning point" that will cause your company to adopt Linux servers. Oh, wait, sorry, that turning point has been predicted annually for the past, what, 8-9 years?

    This event will undoubtedly cause a redundant A/C unit (limited to preserving the data center) to be installed which will be periodically tested to ensure its fail-over capability.

    Your boss will never believe Microsoft ever again.