Current Events > Wow... I figured out a weird network issue and proved it was on AT&T's end.

(+) Yeah! (+)

We've had connectivity problems over AT&T at a couple sites for like 3 months now. We opened tickets with AT&T, but they kept closing the tickets saying nothing was wrong. We replaced our firewalls at one of the sites. Same problem. We've finally got the necessary folks at AT&T on a conference call to troubleshoot everything. We've been on a bridge with them for like 9.5 hours now.

Problem: Massive amounts of packet loss when pinging from our firewall.
Weirdness:

• When AT&T connects their test device to the circuit, they don't see any problems.
• When AT&T connects their test device in the other direction facing our equipment, no problems there.

Root cause: Our firewalls are using a specific virtual MAC address for their aggregated ports that ends in a 9 (last bit in the MAC address is a 1). Somewhere in the carrier networks, they're load balancing based on source MAC address, and it's load balancing our traffic over a bad path somewhere. The MAC addresses of their test equipment all happened to end in numbers or letters that would have a last bit of 0, so they were load balanced differently.

So now AT&T has to fix their shit and likely will need to give us refunds for trouble at two sites.

Also one of the guys on the call sounds just like *ANT-MAN 3 SPOILERS* Immortus from the mid-credits scene *ANT-MAN 3 SPOILERS*