Current Events > Anyone ever celebrate a power outage at work?

(+) Yeah! (+)

When I was about 3 years into doing ISP tech support, I was fucking miserable.

• I hated the company I worked for
• I hated my boss
• I felt like I was stuck in a rut and spinning my wheels in life
• I hated taking tech support calls and dealing with morons and assholes all day
• I had just recently bombed an interview for a networking job. I thought I knew enough about networking at the time to move into that realm, but that was a soul crushing way to find out that I didn't.
• Had no plan to improve beyond where I was. I had been a level 2 tech support rep for about 2 years, and they had stopped promoting people to level 3 and just made the level 2s start doing level 3 work. They also made the level 2s start doing supervisor calls instead of having the team leads do them.

I remember being in the middle of a call and the whole call center went dark.

As soon as I saw the lights turn out, my call disconnected. I stood up, raised my fists in the air and shouted "YES! YES!"

My boss called me over to his desk so he could scold me, but I did not give two shits. I hated that place and stayed there way too long.

not a customer, but my own house; sure...

Yeah, just because I get time and a half at that point to do the same stuff

Not a power outage but an internet outage at work. I put in the call to the ISP and it was an area outage. Nothing for me to fix.

As a sysadmin, no.....fuck no. It's terrifying, quickly shutting down all the servers before the battery backup runs out, then hoping they all come back up once power is restored (protip- they never do...something always breaks).

pogo_rabid posted...

As a sysadmin, no.....fuck no. It's terrifying, quickly shutting down all the servers before the battery backup runs out, then hoping they all come back up once power is restored (protip- they never do...something always breaks).

Yup it sucks. Thankfully I've only experienced that once in my career.

Not really... But I definitely empathize with feeling stuck where you are...

i do IT also. one day the fire alarm went off and i had to evacuate. while i was waiting outside some lady started desperatly messaging me

i was like lol i can try to help but i dont have any of my stuff so dunno how i can help much lol

BlazinBlue88 posted...

Yup it sucks. Thankfully I've only experienced that once in my career.

My first IT job, I had about 12 seconds of backup, and the server room circuit was tied into the bathrooms, and one of the hand air dryers had a short in it, so every few months we would get a hard crash of everything when the dryer heat up the circult and tripped the breaker. We couldn't get budget or approval to get dedicated power.

So happy I don't work there anymore.

pogo_rabid posted...

My first IT job, I had about 12 seconds of backup, and the server room circuit was tied into the bathrooms, and one of the hand air dryers had a short in it, so every few months we would get a hard crash of everything, and we couldn't get budget or approval to get dedicated power. So happy I don't work there anymore.

Holy shit. That's incredible.

BlazinBlue88 posted...

Holy shit. That's incredible.

it was a construction company that wasn't a union shop, take that for what you will.

Yes... but they made us stay the whole shift with only the emergency lights on and no work to do so it became less fun pretty quick.

yeah we were let out early a couple of times

When I was washing dishes at a nursing home, a well-timed power outage meant the meal would be served on paper plates instead (because the machine was inoperable). So, yeah.

pogo_rabid posted...

My first IT job, I had about 12 seconds of backup, and the server room circuit was tied into the bathrooms, and one of the hand air dryers had a short in it, so every few months we would get a hard crash of everything when the dryer heat up the circult and tripped the breaker. We couldn't get budget or approval to get dedicated power.

So happy I don't work there anymore.

It's insane how some companies only consider up front cost, but refuse to consider the cost of outages. This is one of the problems I had with my last job.

At first, they said they were only ever going to open one brick and mortar store. We did the full network setup... full size server rack, redundant switches, redundant firewalls, redundant servers, etc.

Then somehow they changed their minds and wanted to open 75 stores.

Then someone in retail operations decided that they didn't want to spend $72k per store for the network and systems infrastructure any more. They only wanted to spend $5k on the network infrastructure. One switch cost about $6k. So, they also didn't want to pay for redundant firewalls any more.

I sent lengthy emails explaining the nightmare scenarios that could happen. They just shrugged them off as fear mongering. I was stressed AF. I was already working 60 to 80 hour weeks all the time and I didn't have time to chase down every little outage immediately to keep these proposed stores up and running with shitty $5,000 network designs.

I ended up putting in my 2 weeks notice a few months later for a number of reasons, this included. My 2nd to last day, the security camera vendor unplugged the only firewall they wanted to pay for at one of the stores where they didn't want to pay for redundancy. Store down. No sales happening. The firewall model they used takes ~30 minutes just to boot up.

So then at 10am, I had to drive through Austin traffic jams on Mopac trying to get from Southwest Austin to Northwest Austin. It took an hour. About 45 minutes into the drive, the firewall had come back up. I was pissed, and it just further solidified my reason for leaving.

Yes but the celebration was short lived. I used to work at a grocery store over night. Stocking shelves. The power went out cause someone drove into a transformer I think. I thought we were going home but then our boss started handing out flashlights

I build out storage systems and SANs for companies and I hit snags like that all the time.

oh we'll run prod dev test and replication all on this same network...

just document the risk and wait for the call when shit hits the fan

A couple weeks ago power went out and I got to go home early :D

BuckVanHammer posted...

I build out storage systems and SANs for companies and I hit snags like that all the time.

oh we'll run prod dev test and replication all on this same network...

just document the risk and wait for the call when shit hits the fan

Nice man. My main job duty is to manage my company's storage infra. What are you typically working with? EMC, Nutanix, etc?

BlazinBlue88 posted...

Nice man. My main job duty is to manage my company's storage infra. What are you typically working with? EMC, Nutanix, etc?

I've been in storage for 12 years so I've touched about everything at one point or another. I kinda leaned NetApp for a good bit, but mixed it up quite a bit over the last few years.