Current Events > The downside of being a network engineer is the odd hours.

(+) Yeah! (+)

My job's primary internet provider had another outage this morning. They keep claiming that they don't see anything wrong. I opened a ticket with them a couple days ago about this same problem and they said nothing was wrong.

So, I turned on a debug to watch for inbound BGP route updates from them, and at 2:50am today:

Sep 7 02:50:11 CDT: BGP(0): ___.___.___.___ rcvd UPDATE w/ attr: nexthop ___.___.___.___, origin i, metric 122, merged path 11427, AS_PATH
Sep 7 02:50:11 CDT: BGP(0): ___.___.___.___ rcvd 0.0.0.0/0
Sep 7 02:50:11 CDT: BGP(0): Revise route installing 1 of 1 routes for 0.0.0.0/0 -> ___.___.___.___(global) to main IP table
Sep 7 02:55:41 CDT: BGP(0): ___.___.___.___ rcvd UPDATE w/ attr: nexthop ___.___.___.___, origin i, metric 112, merged path 11427, AS_PATH
Sep 7 02:55:41 CDT: BGP(0): ___.___.___.___ rcvd 0.0.0.0/0
Sep 7 02:55:41 CDT: BGP(0): Revise route installing 1 of 1 routes for 0.0.0.0/0 -> ___.___.___.___(global) to main IP table
Sep 7 02:56:53 CDT: BGP(0): ___.___.___.___ rcvd UPDATE w/ attr: nexthop ___.___.___.___, origin i, metric 122, merged path 11427, AS_PATH
Sep 7 02:56:53 CDT: BGP(0): ___.___.___.___ rcvd 0.0.0.0/0
Sep 7 02:56:53 CDT: BGP(0): Revise route installing 1 of 1 routes for 0.0.0.0/0 -> ___.___.___.___(global) to main IP table
Sep 7 03:00:23 CDT: BGP(0): ___.___.___.___ rcvd UPDATE w/ attr: nexthop ___.___.___.___, origin i, metric 112, merged path 11427, AS_PATH
Sep 7 03:00:23 CDT: BGP(0): ___.___.___.___ rcvd 0.0.0.0/0
Sep 7 03:00:23 CDT: BGP(0): Revise route installing 1 of 1 routes for 0.0.0.0/0 -> ___.___.___.___(global) to main IP table

With BGP, you typically advertise one or more routes so that the next device knows to send traffic for a certain IP to that router. Spectrum is advertising 0.0.0.0/0, which is called a default route. All traffic will be sent to Spectrum's IP unless the router has a more specific route that points a different direction. For example, if I had a route for 192.168.1.1/32 pointed toward our firewall, traffic destined for 192.168.1.1 would be sent to our firewall.

So now I have definitive proof that Spectrum had an outage. With BGP, no route updates are sent unless the router loses the route itself. So if their router doesn't know about 0.0.0.0/0 any more, then it will immediately stop advertising that route to us.

Now to get THEM to understand that. I've been up since 2:50AM CDT because of this.
---

?
---

Shit like this really winds me up. Had a client hounding me for a full month because they were getting company wide freezing with their software. Several times a day for 10 seconds at a time. We checked everything on our end, all fine. I asked them to really dig into their server for the obvious. Assured me nothing was amiss. They really put the pressure on, got the directorial staff involved. Stuff got heated. Trust was damaged. Would get calls on a saturday morning about it.

Turns out they had been running a report job every 2 hours against the entire database with chaotic, badly written SQL and nobody thought to mention this.

The client contact I was speaking to is their general IT manager, mind you. Didn't think it worth mentioning that they had thrown in this dodgy SQL at any point.

I mean that's barely related to being a network engineer but it's annoying working out of hours for stuff that is really somebody else's problem.
---

I get sick of explaining what packet loss is to the dipshits Spectrum hires for tech support

Rexdragon125 posted...

I get sick of explaining what packet loss is to the dipshits Spectrum hires for tech support

Dude, a few years ago a taught someone at Spectrum's business tech support how to do a ping test from command prompt.

The problem is that a lot of ISPs (or the companies ISPs contract tech support to) hire people for tech support that don't know much about networking or computers in general. That's essentially how I got started in tech support back in 2007, but they also don't train you to understand stuff at a very deep level. They just teach people to follow scripts and written procedures.

The people in those positions with actual technical acumen eventually start to figure out things and become higher level techs, but most just kinda flounder/coast and don't actually learn anything other than following scripts.
---

CableZL posted...

Dude, a few years ago a taught someone at Spectrum's business tech support how to do a ping test from command prompt.

Wtf?
---

ISP technical support is the most hair ripping thing in existence. Unless it's an area wide outage it's ALWAYS on your end according to them until you prove otherwise.
---

BlazinBlue88 posted...

CableZL posted...

Dude, a few years ago a taught someone at Spectrum's business tech support how to do a ping test from command prompt.

Wtf?

Yeah, apparently they rely on some web site to do pings remotely.
---

This really has to do with being unable to own up to your mistakes and that can happen to basically anyone in any industry.
---

sauceje posted...

This really has to do with being unable to own up to your mistakes and that can happen to basically anyone in any industry.

Yup. I remember being about 10 months into ISP tech support and being on the phone with a dude who was an MCSE. He couldn't send email. After about 45 minutes of arguing with him, he finally let me troubleshoot from his machine remotely.

He turned the firewall in the modem up to high (default is low), which caused the modem to block port TCP25.

He eventually owned up to changing the firewall to that, but... Sheesh.

I consider myself to be pretty good at networking, but that still doesn't mean I'm always right about networking issues.
---

Spectrum has finally identified the problem that I've been calling them about for days. Sheesh. The first step is over and I am going to sleep.
---

This issue was affecting the entire city of Austin at the very least. Holy crap.
---

Feels like this shit is getting more common.

"There was an event upstream in our network, but that shouldn't have affected you."

You gotta be fucking kidding me
---

When they identified the problem, I thought the train was rolling toward resolution. NOPE. They don't even understand how this would affect people...

Holy moly
---

Nice
---

TC tell me why my system admins don't want me running Windows spotlight
---

Do they even have engineers employed? lol

P4wn4g3 posted...

TC tell me why my system admins don't want me running Windows spotlight

Windows Spotlight is a feature included by default in Windows 10 that downloads pictures and advertisements automatically from Bing and displays them when the lock screen is being shown on a computer running Windows 10. Users are occasionally given an opportunity to mark whether they wish to see more or fewer images of a similar type, and sometimes the images are overlaid with links to advertisements. In 2017, Microsoft began adding location information for many of the photographs.

I can see why a system admin wouldn't want 3rd party ads to be displayed on the lock screen. Probably a company security policy if I had to guess.
---

Rexdragon125 posted...

Do they even have engineers employed? lol

That's what I'm questioning at the moment.

1) The building manager at my job said someone else in town with Spectrum fiber internet had an outage at the same time we did, but they weren't sure why.
2) My boss has Spectrum cable internet (coax) at his home and he's got automated failover setup. He had an outage at the same time we did
3) The logs I've sent them show unequivocally that it DID affect us, so I'm confused as to how they can claim it shouldn't have affected us.
---

CableZL posted...

P4wn4g3 posted...

TC tell me why my system admins don't want me running Windows spotlight

Windows Spotlight is a feature included by default in Windows 10 that downloads pictures and advertisements automatically from Bing and displays them when the lock screen is being shown on a computer running Windows 10. Users are occasionally given an opportunity to mark whether they wish to see more or fewer images of a similar type, and sometimes the images are overlaid with links to advertisements. In 2017, Microsoft began adding location information for many of the photographs.

I can see why a system admin wouldn't want 3rd party ads to be displayed on the lock screen. Probably a company security policy if I had to guess.

The main thing it does is download pictures. The ads seem to be separate, or at least there are other registry entries to stop them. Also the ads aren't shown on the lock screen. It's kind of terrible how much bloatware Windows 10 comes packed with though. I don't honestly understand why we use it.
---

The engineers have escalated it to an even higher level team now. How were they entirely unaware that these outages were affecting customers?

The fiber support team directly mentioned they had a lot of people calling in this morning at fucking 3AM when it happened. Of course it's affecting people, holy shit.
---

Yeah, our network engineer has stayed at work up to like 2am doing stuff on multiple occasions. Kind of makes me glad that I prefer the server route.
---

I'm now told their "CBO team" is going to be looking into it, but no telling if they'll actually provide an update tonight. I'm gonna drop it for now and try to enjoy the rest of my day.

I'm trying to laugh at this whole situation to try to alleviate the frustration... I'll probably be able to do that better in the morning after I've had a full night's sleep.
---

Solar_Crimson posted...

Yeah, our network engineer has stayed at work up to like 2am doing stuff on multiple occasions. Kind of makes me glad that I prefer the server route.

When do you think server maintenance happens buddy?
---

I hate that I'm now in this crappy mood. I'm normally very even keel, but today's issues have thrown me for a loop.
---

was i right about vol2tex's thing was it his nic card.
---

Pancake posted...

was i right about vol2tex's thing was it his nic card.

I've completely forgotten what issue that was at this point. Care to refresh me?
---

sounded like semi-constant slow internet speeds. his phone was getting better speeds than his PC.
---

Pancake posted...

sounded like semi-constant slow internet speeds. his phone was getting better speeds than his PC.

I don't remember this one at all. Was this with wifi on his PC?
---

yeah!

once you guys showed up i rolled out.
---

I dont envy you. I am a Security Admin and I go home at 5 every day.
---

Pancake posted...

yeah!

once you guys showed up i rolled out.

Yeah, I'm not sure why I don't remember this one, but... My initial (2nd?) hunch is that the network card has a lower capability for wifi connectivity than the phone.

Like the phone may be able to do 802.11ac, but the computer can only do 802.11n or maybe even 802.11b/g.

Either that or he has a dual-band router that broadcasts 2.4ghz and 5ghz networks... And the phone was connected to the 5ghz band while the PC was connected to the 2.4ghz band.
---

well i'm glad i know what those are or else i'd feel like a big moron right about now!
---

Pancake posted...

well i'm glad i know what those are or else i'd feel like a big moron right about now!

Nah, there's a ton to networking and even a ton to different networking topics... I last year I got almost half way through the CCNA wireless book and learned a bunch of info about how wireless works down to the signal modulation techniques, but... I've forgotten most of it now.
---

it'll come back to you when you need it. it's like one for all.

also this topic is kinda funny because i have spectrum here and it's been dropping too.
---

Pancake posted...

it'll come back to you when you need it. it's like one for all.

also this topic is kinda funny because i have spectrum here and it's been dropping too.

Yeah, I'm suspecting that this issue is actually affecting quite a wide range of Spectrum customers. It also affected our backup MPLS circuit and once they escalated to Spectrum, they were told that "they had a hit on their backbone network."
---

if it drops again over here i could give you a time if you want. that'll show 'em.
---

Oh damn it, now there's packet loss... gonna be a long night
---

Can't even ping the 2nd hop into their network without getting packet loss here and there. This is crap.
---

Charging customers and not providing service is like an ISP's wet dream

I'm finally talking to someone who can actually reroute traffic... They did so and testing looks good so far... trying to do a bunch of tests
---

more packet loss
---

They're dispatching a tech on Monday night to test everything. Company will remain on the secondary internet connection until then.
---

The packet loss isn't happening anymore. I'm hoping they fixed something in this regard overnight.
---