SUSE Linux Enterprise Server 16.0

Configuring Multihome Networking on VM Hosts

Publication Date: 24 Oct 2025

WHAT?

This article explains how to configure multihome networking on a SUSE Linux Enterprise Server VM host using the strong host model.

WHY?

Switching from the default weak host model to the strong host model improves security and network reliability in multihome environments.

EFFORT

The setup takes about 20 minutes. Allow up to an hour to fully understand the VM host and multihome networking concepts.

GOAL

Gain a basic understanding of how to configure VM host networking and multihome settings.

REQUIREMENTS

Access to a machine that serves as the VM host

Basic understanding of networking and IP addresses

Revision History: Configuring Multihome Networking on VM Hosts

1 Network topology #

The network topology in this setup includes three virtual machines: vm7, vm8, vm9 and one uplink router gw1.

The vm7 VM acts as a router between an uplink router gw1 and the external network (192.168.7.0/24) of vm8 and vm9. The configuration of the vm7 uplink is not relevant for this setup.

The vm8 and vm9 have an uplink in the external network (192.168.7.0/24) with vm7 as a router and are connected to two private networks (192.168.8.0/24 and 192.168.9.0/24).

    gw1 (1)
              |
           ---+----       net1: 192.168.1.0/24 on eth1/enp1s0
              |           [uplink network: not relevant here]
             vm7 (107)
              |
      --+-----+-----+--   net7: 192.168.7.0/24 on eth7/enp7s0
        |           |
 (108) vm8         vm9 (109)
       | |         | |
       | +---------+ |    net8: 192.168.8.0/24 on eth8/enp8s0
       +-------------+    net9: 192.168.9.0/24 on eth9/enp9s0

2 Checking the VM host network configuration #

The following procedure outlines the steps for setting up a VM host.

The VM host has four bridges involved in the setup.

When the VM host is on a productive network with IP connectivity independent of the test networks, there is no IP configured on any of these bridges, including net1. However, you can use the same network service as the host, such as DHCP on net1.

The net1 bridge contains an uplink port eth1 providing access to an external test/LAN network for the vm7 router VM.

Procedure 1: Setting up VM Host #

Check the configuration of the ifcfg-net1 bridge:

$ cat /etc/sysconfig/network/ifcfg-net1
  
  STARTMODE='auto'
  BOOTPROTO='none'
  LINK_REQUIRED=no
  BRIDGE='yes'
  BRIDGE_STP='off'
  BRIDGE_FORWARDDELAY='0'
  BRIDGE_PORTS='eth1'

The net7, net8 and net9 are host-only bridges that have only dynamic VM ports, specific and relevant for this setup. To view the net7 network settings, run:

$ cat /etc/sysconfig/network/ifcfg-net7
  
  STARTMODE='auto'
  BOOTPROTO='none'
  LINK_REQUIRED=no
  LLADDR=66:00:00:00:00:07
  BRIDGE='yes'
  BRIDGE_STP='off'
  BRIDGE_FORWARDDELAY='0'
  BRIDGE_PORTS=''

View the ifcfg-net8 network settings:

$ cat /etc/sysconfig/network/ifcfg-net8
  
  STARTMODE='auto'
  BOOTPROTO='none'
  LINK_REQUIRED=no
  LLADDR=66:00:00:00:00:08
  BRIDGE='yes'
  BRIDGE_STP='off'
  BRIDGE_FORWARDDELAY='0'
  BRIDGE_PORTS=''

View the ifcfg-net9 network settings:

$ cat /etc/sysconfig/network/ifcfg-net9
  
  STARTMODE='auto'
  BOOTPROTO='none'
  LINK_REQUIRED=no
  LLADDR=66:00:00:00:00:09
  BRIDGE='yes'
  BRIDGE_STP='off'
  BRIDGE_FORWARDDELAY='0'
  BRIDGE_PORTS=''

3 Checking the VM network configuration #

The following procedure outlines how the network of a VM host must be configured.

Configuration of the VM network shown in this document uses a so-called tweaked assignment of the IP, MAC and interface names. Every packet uses identical VM identification, just like packet captures. The following list shows the IP addresses for VMs on each network:

net1
vm7: no relevant uplink
net7
- vm7: 192.168.7.107 52:54:00:00:07:07 (eth7/enp7s0)
- vm8: 192.168.7.108 52:54:00:00:07:08 (eth7/enp7s0)
- vm9: 192.168.7.109 52:54:00:00:07:09 (eth7/enp7s0)
net8
- vm8: 192.168.8.108 52:54:00:00:08:08 (eth8/enp8s0)
- vm9: 192.168.8.109 52:54:00:00:08:09 (eth8/enp8s0)
net9
- vm8: 192.168.9.108 52:54:00:00:09:08 (eth9/enp9s0)
- vm9: 192.168.9.109 52:54:00:00:09:09 (eth9/enp9s0)

Procedure 2: Configure Virtual Machines #

Set up vm7.

This setup uses SLES 15 SP7, with the VM acting as a router: it connects to the external network through the eth1 uplink (for example, assigned to an external firewalld zone for masquerading) and to the common test network net7, which serves as the uplink for the other VMs. vm7 has no connection to the private, net8 and net9 networks.

View the IP address:

ip a s
[...]
2: eth1: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:01:07 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 192.168.1.107/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
3: eth7: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:07:07 brd ff:ff:ff:ff:ff:ff
    altname enp7s0
    inet 192.168.7.107/24 brd 192.168.7.255 scope global eth7
       valid_lft forever preferred_lft forever

View the routing table:

default via 192.168.1.1 dev eth1
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.107
192.168.7.0/24 dev eth7 proto kernel scope link src 192.168.7.107

In /etc/sysctl.d/90-network.conf, set the following value to 1 to ensure this machine acts as a router:
```
net.ipv4.conf.all.forwarding = 1
```

Set up vm8.

This configuration runs on SLES 15 SP7, using eth7 as the network uplink and including interfaces for two private networks: net8 and net9.

View the IP address:

> 
  ip a s
3: eth7: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:07:08 brd ff:ff:ff:ff:ff:ff
    altname enp7s0
    inet 192.168.7.108/24 brd 192.168.7.255 scope global eth7
       valid_lft forever preferred_lft forever
4: eth8: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:08:08 brd ff:ff:ff:ff:ff:ff
    altname enp8s0
    inet 192.168.8.108/24 brd 192.168.8.255 scope global eth8
       valid_lft forever preferred_lft forever
5: eth9: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:09:08 brd ff:ff:ff:ff:ff:ff
    altname enp9s0
    inet 192.168.9.108/24 brd 192.168.9.255 scope global eth9
       valid_lft forever preferred_lft forever

View the network routes

> ip r s
      default via 192.168.7.107 dev eth7
192.168.7.0/24 dev eth7 proto kernel scope link src 192.168.7.108
192.168.8.0/24 dev eth8 proto kernel scope link src 192.168.8.108
192.168.9.0/24 dev eth9 proto kernel scope link src 192.168.9.108

Set the following value in /etc/sysctl.d/90-network.conf to 0 to ensure that the machine acts as a host.
```
net.ipv4.conf.all.forwarding = 0
```

Set up vm9.

This configuration runs on SLES 16.0, using eth7 as the network uplink and connecting to two private networks: net8 and net9.

View the IP address:

> ip a s
3: enp7s0: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:07:09 brd ff:ff:ff:ff:ff:ff
    altname enx525400a8076d
    inet 192.168.7.109/24 brd 192.168.7.255 scope global noprefixroute enp7s0
       valid_lft forever preferred_lft forever
4: enp8s0: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:08:09 brd ff:ff:ff:ff:ff:ff
    altname enx525400a8086d
    inet 192.168.8.109/24 brd 192.168.8.255 scope global noprefixroute enp8s0
       valid_lft forever preferred_lft forever
5: enp9s0: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:00:09:09 brd ff:ff:ff:ff:ff:ff
    altname enx525400a8096d
    inet 192.168.9.109/24 brd 192.168.9.255 scope global noprefixroute enp9s0
       valid_lft forever preferred_lft forever

View the network routes:

> 
ip r s

default via 192.168.7.107 dev enp7s0 proto static metric 100 
192.168.7.0/24 dev enp7s0 proto kernel scope link src 192.168.7.109 metric 100 
192.168.8.0/24 dev enp8s0 proto kernel scope link src 192.168.8.109 metric 101 
192.168.9.0/24 dev enp9s0 proto kernel scope link src 192.168.9.109 metric 102

Set the following value in /etc/sysctl.d/90-network.conf to 0 to ensure that the machine acts as a host.
```
net.ipv4.conf.all.forwarding = 0
```

4 IP `sysctl` settings for the multihome host #

System-level networking parameters in Linux are stored in /proc/sys/net/ and can be managed using the sysctl command or by modifying configuration files.

4.1 The `sysctl` command variables #

The sysctl variables offered by the Linux kernel are documented here.

The following net.ipv4.conf.all.arp_* variables affect the arp behavior on a multihome host.

arp_announce = 0

Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on the interface:
1. 0 (default) Use any local address, configured on any interface.
2. 1 Try to avoid local addresses that are not in the target's subnet for this interface. This mode is useful when target hosts reachable via this interface require the source IP address in ARP requests to be part of their logical network configured on the receiving interface. When generating the request, all subnets that include the target IP are checked. If such a subnet exists, the source address is preserved. Otherwise, the source address is selected according to the rules for level 2.
3. 2 Always use the best local address for this target. In this mode, the source address in the IP packet is ignored, and the preferred local address is selected from the outgoing interface that includes the target IP address. If no suitable address is found, the first local address on the outgoing interface or other interfaces is used, hoping to receive a reply.
The max value from conf/{all,interface}/arp_announce is used.
Increasing the restriction level gives a better chance of receiving an answer from the resolved target, while decreasing the level announces more valid sender information.
aarp_ignore = 0
Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:
1. 0 (default) Reply to requests for any local target IP address, configured on any interface.
2. 1 Reply only if the target IP address is configured on the incoming interface.
3. 2 Reply only if the target IP address is configured on the incoming interface and both it and the sender's IP address are in the same subnet on this interface.
4. 3Do not reply to requests for local addresses configured with scope host; only resolutions for global and link addresses are replied.
5. 4–7 Reserved.
6. 8Do not reply to requests for any local address.
The max value from conf/{all,interface}/arp_ignore is used when ARP requests are received on the {interface}.
arp_filter = 0
Controls ARP response behavior for interfaces on the same subnet:
1. 0 (default) The kernel can respond to ARP requests with addresses from other interfaces. This increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. In more complex setups like load-balancing, this behavior can cause problems.
2. 1 Each interface responds only if the kernel would route the packet through it. Requires source-based routing and ensures ARP replies are interface-specific.
arp_filter for the interface will be enabled if at least one of conf/{all,interface}/arp_filter is set to TRUE; otherwise it is disabled.

Additionally, the net.ipv4.conf.all.rp_filter settings permit filtering of L3 / IPv4 packets using reverse routing table lookups:

rp_filter = 0
Modes for source validation:
1. 0 No source validation.
2. 1 Strict mode (RFC3704). Each incoming packet is tested against the FIB. If the interface is not the best reverse path, the packet check fails, and the packet is discarded.
3. 2 Loose mode (RFC3704). Each incoming packet's source address is tested against the FIB, and if it is not reachable via any interface, the packet check fails.
Current recommended practice in RFC3704 is to enable strict mode to prevent IP spoofing from DDoS attacks. If using asymmetric routing or other complicated routing, loose mode is recommended.
The max value from conf/{all,interface}/rp_filter is used when doing source validation on the interface.
The default value is 0. Note that some distributions enable it in startup scripts.

On SUSE systems, net.ipv4.conf.all.rp_filter = 2 (RFC3704 Loose Reverse Path) is enabled by default instead of the kernel default to disable it (rp_filter = 0). Unlike the strict rp_filter = 1 mode, rp_filter = 2 permits asymmetric routing, using the same subnet on multiple interfaces (considers arp_announce=0, arp_ignore=0 and arp_filter=0 defaults).

The net.ipv4.conf.all.log_martians enables logging of packets filtered/dropped by rp_filter:

log_martians = 0

This logs packets with impossible addresses to the kernel log. The log_martians setting for an interface will be enabled if at least one of conf/{all,interface}/log_martians is set to TRUE. Otherwise, it will be disabled.

5 Default behavior #

Linux uses a weak host model as default, configurable via IP-Sysctl settings offered by the kernel.

In this model, an IP address belongs to the machine and is not restricted to being used only on the interface it is assigned to. The machine can use/associate any interface MAC address with any IP address, regardless of the interface.

The following procedures describe the default behavior using vm7, vm8, and vm9.

Procedure 3: Default behavior #

vm7

This VM acts as a router in net7 and even though it is not connected to net8 and net9, it is still involved in the ARP resolution process.

The following ping commands illustrate the default behavior:

vm7:~ ping -c 1 192.168.7.108: reply (regular)

Regular pinging of an IP on a directly connected network (network route) triggers a routing table lookup that selects the eth7 interface and the preferred source from the route (src 192.168.7.107):

192.168.7.0/24 dev eth7 proto kernel scope link src 192.168.7.107

Output:

  PING 192.168.7.108 (192.168.7.108) 56(84) bytes of data.
  64 bytes from 192.168.7.108: icmp_seq=1 ttl=64 time=0.446 ms

  --- 192.168.7.108 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.446/0.446/0.446/0.000 ms
  
- capture (net7):
  
No.     Time           Source                Destination           Protocol Length Info
        1 0.000000000    52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.7.108? Tell 192.168.7.107
        2 0.000178399    52:54:00:00:07:08     52:54:00:00:07:07     ARP      42     192.168.7.108 is at 52:54:00:00:07:08
        3 0.000261527    192.168.7.107         192.168.7.108         ICMP     98     Echo (ping) request  id=0x0001, seq=1/256, ttl=64 (reply in 4)
        4 0.000334225    192.168.7.108         192.168.7.107         ICMP     98     Echo (ping) reply    id=0x0001, seq=1/256, ttl=64 (request in 3)
        5 5.180898743    52:54:00:00:07:08     52:54:00:00:07:07     ARP      42     Who has 192.168.7.107? Tell 192.168.7.108
        6 5.181116066    52:54:00:00:07:07     52:54:00:00:07:08     ARP      42     192.168.7.107 is at 52:54:00:00:07:07

vm7:~ ping -c 1 192.168.7.109: reply (regular)

Same as vm7:~ ping -c 1 192.168.7.108 (#vm7--ping--c-1-1921687108-reply).

vm7:~ ping -c 1 192.168.8.108: error (regular)

Regular pinging of an IP outside of a directly connected network causes the packet to be sent via the gateway (default) route. This uses the eth1 interface with the source IP (192.168.1.107) from the preferred source in the route used to reach the gateway:

default via 192.168.1.1 dev eth1 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.107

The result depends on the configuration of the uplink routers. If there is a kind router sending an unreachable ICMP message, for example, because of an unreachable route (on Linux, to avoid routing the packets to the Internet):

ip r s type unreachable
unreachable 192.168.0.0/16

or dropping the packets (because of a blackhole route entry or a firewall).

Output:

 PING 192.168.8.108 (192.168.8.108) 56(84) bytes of data.

  --- 192.168.8.108 ping statistics ---
  1 packets transmitted, 0 received, 100% packet loss, time 0ms
  
  or:
  
  PING 192.168.8.108 (192.168.8.108) 56(84) bytes of data.
  From 192.168.1.1 icmp_seq=1 Destination Host Unreachable

  --- 192.168.8.108 ping statistics ---
  1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
  
- capture (net1):
  - packet capture on net1 incl. unreachable message (if any):
  
  No.     Time           Source                Destination           Protocol Length Info

      131 1995.230774    192.168.1.107         192.168.8.108         ICMP     104    Echo (ping) request  id=0x002a, seq=1/256, ttl=64 (no response found!)
      132 1995.231275    192.168.1.1           192.168.1.107         ICMP     132    Destination unreachable (Host unreachable)

vm7:~ ping -c 1 192.168.8.109: error (regular)

Same as vm7:~ # ping -c 1 192.168.8.108 (#vm7--ping--c-1-1921688108-error).

vm7:~ ping -c 1 192.168.9.108: error (regular)

Same as vm7:~ # ping -c 1 192.168.8.108 (#vm7--ping--c-1-1921688108-error).

vm7:~ ping -c 1 192.168.9.109: error (regular)

Same as vm7:~ # ping -c 1 192.168.8.108 (#vm7--ping--c-1-1921688108-error).

vm7:~ ping -I eth7 -c 1 192.168.8.108: reply (enforced)

Pinging an IP outside of a directly connected network, while enforcing the use of the specified eth7 interface, is done instead of looking up the destination in the routing table to select an interface and source IP address. In other words, it ignores the routing table.

This causes the machine to resolve the MAC address of the machine having the IP address via ARP on the eth7 interface and to use the primary IP address on this interface as the source IP. The destination machine has a route back to the source IP network and can deliver the reply.

Output:

  PING 192.168.8.108 (192.168.8.108) from 192.168.7.107 eth7: 56(84) bytes of data.
  64 bytes from 192.168.8.108: icmp_seq=1 ttl=64 time=0.431 ms

  --- 192.168.8.108 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.431/0.431/0.431/0.000 ms
  
- capture:
  
  No.     Time           Source                Destination           Protocol Length Info
       55 1090.280252902 52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.7.107
       56 1090.280444897 52:54:00:00:07:08     52:54:00:00:07:07     ARP      42     192.168.8.108 is at 52:54:00:00:07:08
       57 1090.280492377 192.168.7.107         192.168.8.108         ICMP     98     Echo (ping) request  id=0x000c, seq=1/256, ttl=64 (reply in 58)
       58 1090.280553072 192.168.8.108         192.168.7.107         ICMP     98     Echo (ping) reply    id=0x000c, seq=1/256, ttl=64 (request in 57)
       59 1095.512647002 52:54:00:00:07:08     52:54:00:00:07:07     ARP      42     Who has 192.168.7.107? Tell 192.168.7.108
       60 1095.512893801 52:54:00:00:07:07     52:54:00:00:07:08     ARP      42     192.168.7.107 is at 52:54:00:00:07:07
  
- arp-cache:
  
  192.168.8.108 dev eth7 lladdr 52:54:00:00:07:08 REACHABLE
  192.168.7.108 dev eth7 lladdr 52:54:00:00:07:08 REACHABLE
  
- strace:
  
  socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP) = 3
  setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "eth7\0", 5) = 0
  sendto(3, "\10\0\2\371\377\377\0\1J\274sh\0\0\0\0j\16\16\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin
  _port=htons(0), sin_addr=inet_addr("192.168.8.108")}, 16) = 64
  recvmsg(3, {msg_name={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.8.108")}, msg_namelen=128 => 16, msg_i
  ov=[{iov_base="\0\0\n\346\0\23\0\1J\274sh\0\0\0\0j\16\16\0\0\0\0\0\20\21\22\23\24\25\26\27"..., iov_len=192}], msg_iovlen=1, m
  sg_control=[{cmsg_len=32, cmsg_level=SOL_SOCKET, cmsg_type=SO_TIMESTAMP_OLD, cmsg_data={tv_sec=1752415306, tv_usec=921573}}, {
  cmsg_len=20, cmsg_level=SOL_IP, cmsg_type=IP_TTL, cmsg_data=[64]}], msg_controllen=56, msg_flags=0}, 0) = 64
  write(1, "64 bytes from 192.168.8.108: icm"..., 61) = 61

vm7:~ ping -I eth7 -c 1 192.168.8.109: reply (enforced)

Same as vm7:~ # ping -I eth7 -c 1 192.168.8.108 (#vm7--ping--i-eth7--c-1-1921688108-reply).

vm7:~ ping -I eth7 -c 1 192.168.9.108: reply (enforced)

Same as vm7:~ # ping -I eth7 -c 1 192.168.8.108 (#vm7--ping--i-eth7--c-1-1921688108-reply).

vm7:~ # ping -I eth7 -c 1 192.168.9.109: reply (enforced)

Same as vm7:~ # ping -I eth7 -c 1 192.168.8.108 (#vm7--ping--i-eth7--c-1-1921688108-reply).

vm8
In our setup, vm8 and vm9 behave the same way. See vm9 (#vm9) for more details.

vm9

vm9:~ ping -c 1 192.168.7.107: reply (regular)

Regular pinging of an IP on a directly connected network (network route) triggers a routing table lookup that selects the enp7s0 interface and the preferred source from the route (src 192.168.7.109):

192.168.7.0/24 dev enp7s0 proto kernel scope link src 192.168.7.109 metric 100

Output:

  PING 192.168.7.107 (192.168.7.107) 56(84) bytes of data.
  64 bytes from 192.168.7.107: icmp_seq=1 ttl=64 time=0.567 ms

  --- 192.168.7.107 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.567/0.567/0.567/0.000 ms
  
- capture:

  No.     Time           Source                Destination           Protocol Length Info
        1 0.000000000    52:54:00:00:07:09     Broadcast             ARP      42     Who has 192.168.7.107? Tell 192.168.7.109
        2 0.000356939    52:54:00:00:07:07     52:54:00:00:07:09     ARP      42     192.168.7.107 is at 52:54:00:00:07:07
        3 0.000453833    192.168.7.109         192.168.7.107         ICMP     98     Echo (ping) request  id=0x0006, seq=1/256, ttl=64 (reply in 4)
        4 0.000512905    192.168.7.107         192.168.7.109         ICMP     98     Echo (ping) reply    id=0x0006, seq=1/256, ttl=64 (request in 3)
        5 23.564217607   52:54:00:00:07:07     52:54:00:00:07:09     ARP      42     Who has 192.168.7.109? Tell 192.168.7.107
        6 23.564482951   52:54:00:00:07:09     52:54:00:00:07:07     ARP      42     192.168.7.109 is at 52:54:00:00:07:09
 
- arp-cache:

  192.168.7.107 dev enp7s0 lladdr 52:54:00:00:07:07 REACHABLE

vm9:~ ping -c 1 192.168.7.108: reply (regular)

Same as vm9:~ # ping -c 1 192.168.7.107(#vm9--ping--c-1-1921687107-reply).

vm9:~ ping -c 1 192.168.8.108: reply (regular)

Regular pinging of an IP on a directly connected network (network route) triggers a routing table lookup that selects the enp8s0 interface and the preferred source from the route (src 192.168.8.109):

192.168.8.0/24 dev enp8s0 proto kernel scope link src 192.168.8.109 metric 101

Output:

PING 192.168.8.108 (192.168.8.108) 56(84) bytes of data.
  64 bytes from 192.168.8.108: icmp_seq=1 ttl=64 time=0.460 ms

  --- 192.168.8.108 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.460/0.460/0.460/0.000 ms
  
- capture:
  
  No.     Time           Source                Destination           Protocol Length Info
        1 0.000000000    52:54:00:00:08:09     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.8.109
        2 0.000155876    52:54:00:00:08:08     52:54:00:00:08:09     ARP      42     192.168.8.108 is at 52:54:00:00:08:08
        3 0.000268681    192.168.8.109         192.168.8.108         ICMP     98     Echo (ping) request  id=0x0008, seq=1/256, ttl=64 (reply in 4)
        4 0.000336540    192.168.8.108         192.168.8.109         ICMP     98     Echo (ping) reply    id=0x0008, seq=1/256, ttl=64 (request in 3)
        5 5.109132370    52:54:00:00:08:08     52:54:00:00:08:09     ARP      42     Who has 192.168.8.109? Tell 192.168.8.108
        6 5.109320257    52:54:00:00:08:09     52:54:00:00:08:08     ARP      42     192.168.8.109 is at 52:54:00:00:08:09
  
- arp-cache:
  
  192.168.8.108 dev enp8s0 lladdr 52:54:00:00:08:08 REACHABLE

vm9:~ ping -c 1 192.168.9.108: reply (regular)

Regular pinging of an IP on a directly connected network (network route) and using
a lookup of the destination IP via routing table causing to select the enp9s0
interface and the preferred source from the route (src 192.168.9.109):

Output:

 PING 192.168.9.108 (192.168.9.108) 56(84) bytes of data.
  64 bytes from 192.168.9.108: icmp_seq=1 ttl=64 time=0.488 ms

  --- 192.168.9.108 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 0.488/0.488/0.488/0.000 ms
  
- capture:
  
  No.     Time           Source                Destination           Protocol Length Info
        1 0.000000000    52:54:00:00:09:09     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.9.109
        2 0.000212864    52:54:00:00:09:08     52:54:00:00:09:09     ARP      42     192.168.9.108 is at 52:54:00:00:09:08
        3 0.000302044    192.168.9.109         192.168.9.108         ICMP     98     Echo (ping) request  id=0x0009, seq=1/256, ttl=64 (reply in 4)
        4 0.000364673    192.168.9.108         192.168.9.109         ICMP     98     Echo (ping) reply    id=0x0009, seq=1/256, ttl=64 (request in 3)
        5 5.157113256    52:54:00:00:09:08     52:54:00:00:09:09     ARP      42     Who has 192.168.9.109? Tell 192.168.9.108
        6 5.157392878    52:54:00:00:09:09     52:54:00:00:09:08     ARP      42     192.168.9.109 is at 52:54:00:00:09:09
  
- arp-cache:
  
  192.168.9.108 dev enp9s0 lladdr 52:54:00:00:09:08 REACHABLE

vm9:~ # ping -I enp9s0 -c 1 192.168.8.108: reply (enforced)

Pinging an IP address while binding the ping to the specified enp9s0 interface is done instead of looking up the destination in the routing table. In other words, it ignores the routing table.

This causes the machine to resolve the MAC address of the machine having the IP address via ARP on the enp9s0 interface and to use the primary IP address (192.168.9.109) as the source.

The destination machine answers with the MAC address of enp8s0 on enp9s0, causing an additional MAC and IP ARP association in caches, and sends the ICMP echo reply using the same interface as in the request.

Output:

 PING 192.168.8.108 (192.168.8.108) from 192.168.9.109 enp9s0: 56(84) bytes of data.
    64 bytes from 192.168.8.108: icmp_seq=1 ttl=64 time=0.895 ms

    --- 192.168.8.108 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.895/0.895/0.895/0.000 ms
    
  - capture (net9):
    
    No.     Time           Source                Destination           Protocol Length Info
         13 2139.146557619 52:54:00:00:09:09     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.9.109
         14 2139.146934545 52:54:00:00:09:08     52:54:00:00:09:09     ARP      42     192.168.8.108 is at 52:54:00:00:09:08
         15 2139.146993788 192.168.9.109         192.168.8.108         ICMP     98     Echo (ping) request  id=0x0015, seq=1/256, ttl=64 (reply in 16)
         16 2139.147062178 192.168.8.108         192.168.9.109         ICMP     98     Echo (ping) reply    id=0x0015, seq=1/256, ttl=64 (request in 15)
         17 2144.347165844 52:54:00:00:09:08     52:54:00:00:09:09     ARP      42     Who has 192.168.9.109? Tell 192.168.9.108
         18 2144.347432020 52:54:00:00:09:09     52:54:00:00:09:08     ARP      42     192.168.9.109 is at 52:54:00:00:09:09
    
  - arp-cache:
    (when used ping with/without interface binding multiple times)
    
    192.168.8.108 dev enp9s0 lladdr 52:54:00:00:09:08 REACHABLE 
    192.168.8.108 dev enp8s0 lladdr 52:54:00:00:08:08 REACHABLE

vm9:~ ping -I enp8s0 -c 1 192.168.9.108: reply (enforced)

Same as vm9:~ # ping -I enp9s0 -c 1 192.168.8.108 (#vm9--ping--i-enp9s0--c-1-1921688108-reply).

just binding the enp8s0 interface to ping IP on enp9s0.

vm9:~ ping -I enp7s0 -c 1 192.168.9.108: error (regular)

Pinging an IP address bound/enforced to the specified enp7s0 interface is done instead of looking up the destination in the routing table. In other words, it ignores the routing table.

Similar to vm7:~ # ping -c 1 192.168.8.108 (#vm7--ping--c-1-1921688108-error), failing regardless of the interface binding because the router on the enp7s0 interface does not have routing to the 192.168.9.0/24 network and routes the packet to the uplink.

Output (incl. unreachable error):

PING 192.168.9.108 (192.168.9.108) from 192.168.7.109 enp7s0: 56(84) bytes of data.
  From 192.168.1.1 icmp_seq=1 Destination Host Unreachable

  --- 192.168.9.108 ping statistics ---
  1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
  
- capture (net7):
  
  No.     Time           Source                Destination           Protocol Length Info
       35 468.566259392  192.168.7.109         192.168.9.108         ICMP     98     Echo (ping) request  id=0x000c, seq=1/256, ttl=64 (no response found!)
       36 471.671403634  192.168.1.1           192.168.7.109         ICMP     126    Destination unreachable (Host unreachable)
  
- arp-cache: no entry (on enp7s0)

vm9:~ # ping -I enp7s0 -c 1 192.168.8.108: error (regular)

Same as vm9:~ # ping -I enp7s0 -c 1 192.168.9.108 (#vm9--ping--i-enp7s0--c-1-1921689108-error).

6 Multihome setup #

This section shows how to configure a multihome system using kernel ARP and reverse path filtering settings to control network traffic.

6.1 Setting up multihoming #

The kernel's multihome ARP filtering is controlled by the following sysctl settings, which are typically placed in /etc/sysctl.d/90-network.conf config:

Procedure 4: Multihome setup #

Mode for announcing local IP in requests:
```
net.ipv4.conf.all.arp_announce = 2
```
Mode for sending replies to ARP requests:
```
net.ipv4.conf.all.arp_ignore = 1
```
or higher filtering levels (arp_ignore = 2 if desired) on all involved machines vm7, vm8, vm9.
The rp_filter = 2 filtering of IP packets according to the route table matches is already set by SUSE by default to rp_filter = 2. When desired, in can be “increased” to stricter filtering using rp_filter = 1, but please note that this may cause packet drops, for example, in an asymmetric routing setup.

The following ping commands illustrate the multihome behavior and difference to the default behavior:

vm7

vm7:~ ping -c 1 192.168.7.108`: reply (regular)

Same as default behavior (#vm7--ping--c-1-1921687108-reply).

vm7:~ ping -c 1 192.168.7.109`: reply (regular)

Same as default behavior (#vm7--ping--c-1-1921687108-reply).

vm7:~ # ping -c 1 192.168.8.108`: error (regular)

Same as default behavior (#vm7--ping--c-1-1921688108-error).

vm7:~ # ping -c 1 192.168.8.109`: error (regular)

Same as default behavior (#vm7--ping--c-1-1921688108-error).

vm7:~ # ping -c 1 192.168.9.108`: error (regular)

Same as default behavior (#vm7--ping--c-1-1921688108-error).

vm7:~ # ping -c 1 192.168.9.109`: error (regular)

Same as default behavior (#vm7--ping--c-1-1921688108-error).

vm7:~ # ping -I eth7 -c 1 192.168.8.108`: error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.8.108 (192.168.8.108) from 192.168.7.107 eth7: 56(84) bytes of data.
From 192.168.7.107 icmp_seq=1 Destination Host Unreachable

--- 192.168.8.108 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
     56 461.814511760  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.7.107
     57 462.819873108  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.7.107
     58 463.844043402  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.7.107

vm7:~ # ping -I eth7 -c 1 192.168.8.109`: error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.8.109 (192.168.8.109) from 192.168.7.107 eth7: 56(84) bytes of data.
From 192.168.7.107 icmp_seq=1 Destination Host Unreachable

--- 192.168.8.109 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
     74 623.897407366  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.109? Tell 192.168.7.107
     75 624.903906694  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.109? Tell 192.168.7.107
     76 625.928060996  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.8.109? Tell 192.168.7.107

vm7:~ ping -I eth7 -c 1 192.168.9.108: error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.9.108 (192.168.9.108) from 192.168.7.107 eth7: 56(84) bytes of data.
From 192.168.7.107 icmp_seq=1 Destination Host Unreachable

--- 192.168.9.108 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
     81 726.553781184  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.7.107
     82 727.562690381  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.7.107
     83 728.586537241  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.7.107

vm7:~ ping -I eth7 -c 1 192.168.9.109: error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.9.109 (192.168.9.109) from 192.168.7.107 eth7: 56(84) bytes of data.
From 192.168.7.107 icmp_seq=1 Destination Host Unreachable

--- 192.168.9.109 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
     92 817.265392026  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.109? Tell 192.168.7.107
     93 818.284912731  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.109? Tell 192.168.7.107
     94 819.309070330  52:54:00:00:07:07     Broadcast             ARP      42     Who has 192.168.9.109? Tell 192.168.7.107

vm8
In our setup, vm8 and vm9 behave the same way. See vm9 (#vm9-1) for more details.

vm9

vm9:~ # ping -c 1 192.168.7.107: reply (regular)

Same as default behavior (#vm9--ping--c-1-1921687107-reply-regular).

vm9:~ # ping -c 1 192.168.7.108: reply (regular)

Same as default behavior (#vm9--ping--c-1-1921687107-reply-regular).

vm9:~ # ping -c 1 192.168.8.108 : reply (regular)

Same as default behavior (#vm9--ping--c-1-1921688108-reply-regular).

vm9:~ # ping -c 1 192.168.9.108 : reply (regular)

Same as default behavior (#vm9--ping--c-1-1921689108-reply-regular).

vm9:~ ping -I enp9s0 -c 1 192.168.8.108 : error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.8.108 (192.168.8.108) from 192.168.9.109 enp9s0: 56(84) bytes of data.
From 192.168.9.109 icmp_seq=1 Destination Host Unreachable

--- 192.168.8.108 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000000    52:54:00:00:09:09     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.9.109
      2 1.053440783    52:54:00:00:09:09     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.9.109
      3 2.077406483    52:54:00:00:09:09     Broadcast             ARP      42     Who has 192.168.8.108? Tell 192.168.9.109

vm9:~ ping -I enp8s0 -c 1 192.168.9.108 : error (filtered)

The multihome ARP settings cause the system to ignore ARP requests due to a subnet mismatch, and as a result, the enforcement via the -I interface binding no longer works.

Output:

PING 192.168.9.108 (192.168.9.108) from 192.168.8.109 enp8s0: 56(84) bytes of data.
From 192.168.8.109 icmp_seq=1 Destination Host Unreachable

--- 192.168.9.108 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

- capture:

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000000    52:54:00:00:08:09     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.8.109
      2 1.030989481    52:54:00:00:08:09     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.8.109
      3 2.054972421    52:54:00:00:08:09     Broadcast             ARP      42     Who has 192.168.9.108? Tell 192.168.8.109

vm9:~ ping -I enp7s0 -c 1 192.168.9.108: error (regular)

Same as default behavior (#vm9--ping--c-1-1921689108-reply-regular).

vm9:~ ping -I enp7s0 -c 1 192.168.8.108: error (regular)

Same as default behavior (#vm9--ping--c-1-1921689108-reply-regular).

6.2 Policy routing #

By default, all unicast routing entries are placed in the main table (and local/broadcast routes local table) and the lookups use the destination only.

The direct routes (without a gateway) are created automatically by the kernel while adding the address to the interface, e.g. (on vm7). ip addr add 192.168.7.107/24 dev eth7 causes to create the following routes.

ip route show table main dev eth7
  192.168.7.0/24 dev eth7 proto kernel scope link src 192.168.7.107

ip route show table local dev eth7
    
    local 192.168.7.107 proto kernel scope host src 192.168.7.107
    broadcast 192.168.7.255 proto kernel scope link src 192.168.7.107

Using a /32 IP address prefix length ip addr add 192.168.7.107/32 causes the kernel to avoid adding routes automatically in favor of own routes.

In complex setups involving asynchronous routing or the use of multiple interfaces in the same subnet, arp_filter=1 may not work properly due to the filtering, especially in the strict rp_filter=1 mode.

Policy routing permits adding a rule that matches the source IP and then using a different routing table (for example, one for each interface) that has different direct and gateway routes than the standard main table, such as its own unique default gateway.

For more information, see also LARTC HOWTO, the man 5 ifrule, man 8 ip-rule, man 5 ifroute, man 8 ip-route man pages and the /etc/iproute2/rt_tables file, which is used to define names for custom tables.

7 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.

Configuring Multihome Networking on VM Hosts

1 Network topology #

2 Checking the VM host network configuration #

3 Checking the VM network configuration #

4 IP sysctl settings for the multihome host #

4.1 The sysctl command variables #

5 Default behavior #

6 Multihome setup #

6.1 Setting up multihoming #

6.2 Policy routing #

7 Legal Notice #

4 IP `sysctl` settings for the multihome host #

4.1 The `sysctl` command variables #