Let’s say I have a Linux VM. Default route is the gateway to the top of rack switch for public internet and a public IP is bound on one virtual nic.
2nd interface is on a private network so the VM can be reached anywhere on the VPN. This is a management network where the gateway is on the other side of the data center.
A lot of stuff sits on the 10.0.0.0/8 that needs to reach this vm so a static route for the second interface points that /8 to that gateway on say 10.100.100.1
Now inside the same cabinet are devices sitting on 10.20.20.0/24.
If I didn’t do anything, would hitting something on say 10.20.20.2 route traffic through gateway outside of the cab and back? I would think so as it sees the routing table and has no way of knowing.
If I want to optimize traffic so nothing is routed and traffic stays local to the cab, could I just add a third nic and give it an IP of say 10.20.20.3 and hitting .2 would arp / hit it directly through the switch in the cab?
It blindly follows the routing table from most specific and/or lower metric to the least specific/higher metric.
I don’t fully follow your example (a diagram would really help here), but I think it would plainly not work. Packets needs to be able to flow both ways to work correctly (unless you’re using a stateful firewall), so the source needs to be able to route all the way to the destination and the destination also needs to be able to send the response all the way back to the source. Both sides must agree on the routing.
ARP might save you there but there’s still conditions where you might end up hitting the gateway for a local connection. It’s kinda weird to have two overlapping subnets on the same local network, I would just avoid that.
If the 10.0.0.0/8 network route is the most specific one, it will be found and used first.
Merely looking at your routing table should answer your question. You should not need static routes at all in your scenario.
If both networks 10.100.100.0/24? And 10.20.20.0/24 share the same level 2 Ethernet segment/vlan/broadcast domain, you don’t even need the third nic, you can setup a secondary IPv4 address on the private nic on the 10.20.20.0/24 network.
I would not call that best practice, but if the number of host on the network is reasonable and you are aware of the security problems created, there’s nothing really wrong with this setup.
Having two nics on the same Ethernet network is actually trickier since you have to do ARP filtering.
Honestly, I can’t make head or tail of this description. It doesn’t matter where the network devices are physically located; what’s the layer 2 topology?
Like others here, I have trouble following your post, but one thing is instantly sticking out to me: you mentioned 10.0.0.0/8 and 10.20.20.0/24 networks being present.
Unless your 10.20.20.x are actually in a 10.0.0.0/8 network (so configured with addresses like 10.20.20.x/8), this is definitely gonna cause a lot of issues. Like most things in networking, it can probably be made to somewhat work for some specific setup with some voodoo magic, but you really need to understand what you’re doing on a very deep level to do that.
The first issue you will probably encounter if you want to communicate between the two networks is that ARP cannot be forwarded, so your /8 network will never discover the /24 network (unless you just bridge them together - that will let the /8 network discover the the /24 devices, but not the other way around).
If you want to make your network routed, you will need to split it into non-overlapping networks. Otherwise give everything a 10.x.y.z/8 IP address and make sure that the whole network is connected via switches (hardware) / bridges (software).
