Current Events > I found something specific for that weird DHCP issue I've been troubleshooting

(+) Yeah! (+)

So 3 of our branch locations have been affected by a DHCP problem after we upgraded the software on our head-end firewalls. It happened while I was on call a few weeks ago, so I'm the one that has to drive the problem to its solution.

So we have two data centers... We'll call them DC1 and DC2

We've got a high availability (HA) pair of firewalls in each data center for each of our branch locations to connect to. Each firewall HA pair has two data center switches that it load balances traffic over to send traffic to anything and everything that lives in the data center. DNS servers, DHCP servers, our edge internet firewall, etc.

The firewalls in DC1 are load balancing to both data center switches as expected.
The firewalls in DC2 are load balancing to both data center switches EXCEPT FOR DHCP RELAY TRAFFIC

• The firewalls are only sending the traffic to the 2nd data center switch. As far as our data center network team can tell, the packet doesn't go anywhere from there.
• When I forced the traffic to go to the first data center switch, the DHCP relay traffic worked as expected.
• Traffic to our DNS servers is load balanced as expected

So it's my suspicion now that 1) the firewall is somehow sending a bad packet to the ecore2 switch and 2) we've confirmed that the firewall isn't load balancing DHCP relay traffic properly.

Hopefully the vendor can figure out what's wrong.

[LFAQs-redacted-quote]


Yeah, the only change was the firmware upgrade. According to the routing table, the subnets that the DHCP servers live in should be load balanced as well. DNS traffic is load balanced. HTTPS traffic is load balanced. We have about 8 or so different subnets for DHCP servers and it's happening for all of them.

Hell yeah, our firewall traffic logs show a definite change in load balancing behavior for DHCP relay traffic before/after the software upgrade.

This has got to be a software bug... HTTPS, HTTP, NTP, and SNMP traffic are all being load balanced. DHCP traffic isn't.

The vendor has some splaining to do

Ded Hot Chili Peppers

We get around this at my job by not using DHCP at our DCs.

Every machine gets a manually assigned number.

CableZL posted...

This has got to be a software bug... HTTPS, HTTP, NTP, and SNMP traffic are all being load balanced. DHCP traffic isn't.

The vendor has some splaining to do

Could your Vendor be "Cisco"?

We've known that they are pretty lazy sometimes when it doesn't suit them.

Did you do any research on the firmware before upgrading? Known issues, changelog, etc.?

Can you not roll it back to the previous version to confirm its the new FW?

Kamen_Rider_Blade posted...

Could your Vendor be "Cisco"?

We've known that they are pretty lazy sometimes when it doesn't suit them.

I have literally never had an issue with our Cisco ASA and its apparently been in deployment for almost 15 years. Cisco is the good shit.

Sophos on the other hand shit cant maintain a stable tunnel to save its life.

CableZL posted...

This has got to be a software bug... HTTPS, HTTP, NTP, and SNMP traffic are all being load balanced. DHCP traffic isn't.

The vendor has some splaining to do

My second thought after reading the op, first being the ol' a config change was applied sometime before upgrade but not saved.

CableZL posted...

This has got to be a software bug

First thing I thought after reading OP. I've had experience with Cisco releasing bad firmware on occasion.

voldothegr8 posted...

irst being the ol' a config change was applied sometime before upgrade but not saved.

This is a good possibility as well.

Tyranthraxus posted...

We get around this at my job by not using DHCP at our DCs.

Every machine gets a manually assigned number.

Well, everything in the data center uses static IPs, but most of our branch locations don't have local DHCP servers, so devices at branch #56, for example, would get DHCP from the DHCP servers that live in the data center.

Kamen_Rider_Blade posted...

Could your Vendor be "Cisco"?

We've known that they are pretty lazy sometimes when it doesn't suit them.

Fortinet

ThisIsAKnoife posted...

Did you do any research on the firmware before upgrading? Known issues, changelog, etc.?

Can you not roll it back to the previous version to confirm its the new FW?

We did some, but it doesn't sound like this is a known issue for them yet. Their TAC group is researching it now. Because our environment is so large, we manage them through Fortinet's cloud management platform... There are some rules around what software version the cloud management portal needs to be at and what software version the firewalls it's managing have to be at... It'd be a lot more work to roll the cloud management platform back AND the head-end firewalls AND the branch firewalls.

voldothegr8 posted...

first being the ol' a config change was applied sometime before upgrade but not saved.

With Fortinet firewalls, all config changes are saved immediately

Oh, it's Fortinet. Def a software fuck up.

CableZL posted...

Well, everything in the data center uses static IPs, but most of our branch locations don't have local DHCP servers, so devices at branch #56, for example, would get DHCP from the DHCP servers that live in the data center.

Oh that's fucking confusing IDK how you would DHCP across the internet. I'm guessing you have a switch with a leased line or VPN?

Tyranthraxus posted...

Oh that's fucking confusing IDK how you would DHCP across the internet. I'm guessing you have a switch with a leased line or VPN?

DHCP Relay either through:

• IPsec tunnel
• Layer 2 point to point circuits (EVPL)
• Layer 2 or layer 3 MPLS circuits

Our engineers found undocumented behavior in our sql databases which should never happen.

these companies be slippin

Gritty posted...

Our engineers found undocumented behavior in our sql databases which should never happen.

these companies be slippin

I've found in the course of experience that the documentation doesn't matter.

We tried to build clusters of databases and then put the clusters in availability groups all of which is documented and when it kept not working we reached out to Microsoft whose response was "literally nobody has ever done this before"

voldothegr8 posted...

Oh, it's Fortinet. Def a software fuck up.

Yep I dont touch our Fortigate when everything is working properly.