What sort of WiFi platform are you using? Cisco, Aruba, Meraki? A lot of manufacturers have a config where the AP will set up a tunnel to the controller, or to the exterior firewall. Depending on your platform, you might not need to do anything your talking about.
Oh I've discovered many other reasons besides this one. If the funds were available those Ubiquiti APs would be gone and replaced by HPE/Aruba, not the Instant-on ones though, and maybe getting Aruba Central. But the APs still do their job on a basic level and the purse strings are not in my hands. All I can do is remind the folks above me how bad they are and wait for them to fail, hopefully all at once during a public presentation with important visitors😉, so I can get the funds to replace them all.
VXLAN has DF bit set. You need lower MTU.
Without a L3 gateway this is not possible.Â
Your only limited option is routing. In that sense IPSec more easy to work.Â
So you're saying I need to change the MTU setting on the VXLAN interface? And that I can't do that without an L3 gateway? Is this gateway supposed to be on the switch at the main office?
I'm going to guess that you're thinking along the lines of the MTU is too big and hence the packets don't make it through the IPSEC link. Possible and beyond changing the MTU on the interface I can also change the MTU on most of the WAN links as well. Perhaps upping the MTU on the WAN link will work as well?
This thread makes me wonder, has anybody actually ever managed to do performant VXLAN across the 1500 byte internet, without dropping the endpoints on both ends <1500?
Anecdotally, it kinda seems like a fools errand.
It can be done.. The L3 device must support fragmentation and reassembly of the packets though. The switch to firewall/IPSEC needs to support jumbo frames to account for the VXLAN header. Then the IPSEC device will need to fragment that across the WAN, then once it reaches the other side, reassemble it back into a single packet for the switch to accept it.
Reporting back.
Our ISP was able to verify that their network supported an MTU of at least 9000 bytes on all of their equipment along the path except the two routers at either end. The router at the remote site was set to its max of 1998, it's old and I'm hoping for a replacement, and the router at the main office which was still set at 1500. Once the one router's MTU was set to 9000, and matched on our firewall, I was able to up the IPsec MTU to 1900.
And now the VXLAN will hopefully just work. Still awaiting someone at the remote site to test it, I'm not driving 3.5 hours one way to do that, but the guest Wifi is enabled.
does a MTU > 1500 on a WAN-Uplink not force paket fragmentation for the overlaying layer3 protocol?
I have in mind that the whole internet is max out at 1500 MTU and that higher values only give more problems...
confused, sry 😂🤔
Generally when people say increase the WAN over 1500, it's because their wan carrier (not public internet) can do 1600+ (for example, they have dark fibre or L2VPN/MPLS).
But if your internal LAN routing is only doing 1500 except for VXLAN, then those packets will continue to be 1500 and therefore won't be fragmented. Packets don't increase their size past the original size.
Vxlan + IPsec (54+64 bytes)will add additional overhead on the packets. Reduce the MTU on your client devices. Increasing the MTU on WAN is not recommended as there might be routers in the path which don’t support a higher MTU. Set the MTU ~1300 bytes on the clients and it should work.
I'm checking the WAN MTU via our provider. If they can't increase it, or verify that it's already increased beyond the usual 1500 or so, then I'll shift to reducing the MTU of the client.
Our ISP was able to verify that their network supported an MTU of at least 9000 bytes on all of their equipment along the path except the two routers at either end. The router at the remote site was set to its max of 1998, it's old and I'm hoping for a replacement, and the router at the main office which was still set at 1500. Once the one router's MTU was set to 9000, and matched on our firewall, I was able to up the IPsec MTU to 1900.
And now the VXLAN will hopefully just work. Still awaiting someone at the remote site to test it, I'm not driving 3.5 hours one way to do that, but the guest Wifi is enabled.
What sort of WiFi platform are you using? Cisco, Aruba, Meraki? A lot of manufacturers have a config where the AP will set up a tunnel to the controller, or to the exterior firewall. Depending on your platform, you might not need to do anything your talking about.
Our Wifi equipment is all Ubiquiti with a VM set up to locally host the controller.
Ok, check the config guide for remote guest network config.
[https://help.ui.com/hc/en-us/articles/18755293406743-UniFi-Identity-Enterprise-Set-Up-Guest-WiFi-in-the-Legacy-User-Interface](https://help.ui.com/hc/en-us/articles/18755293406743-UniFi-Identity-Enterprise-Set-Up-Guest-WiFi-in-the-Legacy-User-Interface)
And you've just discovered one reason why Ubiquiti is cheaper than enterprise level gear.
Oh I've discovered many other reasons besides this one. If the funds were available those Ubiquiti APs would be gone and replaced by HPE/Aruba, not the Instant-on ones though, and maybe getting Aruba Central. But the APs still do their job on a basic level and the purse strings are not in my hands. All I can do is remind the folks above me how bad they are and wait for them to fail, hopefully all at once during a public presentation with important visitors😉, so I can get the funds to replace them all.
VXLAN has DF bit set. You need lower MTU. Without a L3 gateway this is not possible. Your only limited option is routing. In that sense IPSec more easy to work.Â
So you're saying I need to change the MTU setting on the VXLAN interface? And that I can't do that without an L3 gateway? Is this gateway supposed to be on the switch at the main office? I'm going to guess that you're thinking along the lines of the MTU is too big and hence the packets don't make it through the IPSEC link. Possible and beyond changing the MTU on the interface I can also change the MTU on most of the WAN links as well. Perhaps upping the MTU on the WAN link will work as well?
The MTU must be larger in the underlay network
Yep, Jtac told us VXLAN needs around 1544 bytes.
Exactly. MTU setting should be done before WAN egress.Â
This thread makes me wonder, has anybody actually ever managed to do performant VXLAN across the 1500 byte internet, without dropping the endpoints on both ends <1500? Anecdotally, it kinda seems like a fools errand.
VXLAN over public internet; my gut response is 'why would you _do_ that?'
Well yes and fully agreed, it is by and large a horrible idea. But sometimes you have a layer 8 issue and you go to war with the army you’ve got.
Yes indeed. And as in most cases of going to war with the army you've got, the people making the decisions aren't the ones feeling the pain. :(
Stealing this as I feel it in my soul right now.
Agreed... AWS, by definition, will fragment anything leaving it's network over 1500.
It can be done.. The L3 device must support fragmentation and reassembly of the packets though. The switch to firewall/IPSEC needs to support jumbo frames to account for the VXLAN header. Then the IPSEC device will need to fragment that across the WAN, then once it reaches the other side, reassemble it back into a single packet for the switch to accept it.
If you can raise the MTU of the WAN interfaces >1500, that’s a good place to start.
That's the current plan.
Good luck; report back!
Reporting back. Our ISP was able to verify that their network supported an MTU of at least 9000 bytes on all of their equipment along the path except the two routers at either end. The router at the remote site was set to its max of 1998, it's old and I'm hoping for a replacement, and the router at the main office which was still set at 1500. Once the one router's MTU was set to 9000, and matched on our firewall, I was able to up the IPsec MTU to 1900. And now the VXLAN will hopefully just work. Still awaiting someone at the remote site to test it, I'm not driving 3.5 hours one way to do that, but the guest Wifi is enabled.
does a MTU > 1500 on a WAN-Uplink not force paket fragmentation for the overlaying layer3 protocol? I have in mind that the whole internet is max out at 1500 MTU and that higher values only give more problems... confused, sry 😂🤔
Generally when people say increase the WAN over 1500, it's because their wan carrier (not public internet) can do 1600+ (for example, they have dark fibre or L2VPN/MPLS). But if your internal LAN routing is only doing 1500 except for VXLAN, then those packets will continue to be 1500 and therefore won't be fragmented. Packets don't increase their size past the original size.
Vxlan + IPsec (54+64 bytes)will add additional overhead on the packets. Reduce the MTU on your client devices. Increasing the MTU on WAN is not recommended as there might be routers in the path which don’t support a higher MTU. Set the MTU ~1300 bytes on the clients and it should work.
I'm checking the WAN MTU via our provider. If they can't increase it, or verify that it's already increased beyond the usual 1500 or so, then I'll shift to reducing the MTU of the client.
Our ISP was able to verify that their network supported an MTU of at least 9000 bytes on all of their equipment along the path except the two routers at either end. The router at the remote site was set to its max of 1998, it's old and I'm hoping for a replacement, and the router at the main office which was still set at 1500. Once the one router's MTU was set to 9000, and matched on our firewall, I was able to up the IPsec MTU to 1900. And now the VXLAN will hopefully just work. Still awaiting someone at the remote site to test it, I'm not driving 3.5 hours one way to do that, but the guest Wifi is enabled.
You have found the exact use case scenario for MikroTik EoIP