I need help in routing on CentOS 6.9 in below scenario. I tried a lot to figure it out before posting it here using multiple articles which are already there related to this requirement but could not fix it.

Scenario: There are three NICs on CentOS having kernel version 2.6.32-754.11.1.el6.x86_64. NIC1 is for WAN access. NIC2 and NIC3 are on same private subnet with below configuration.

eth0 - Network: 14.x.x.16/29
eth1 - Network: 10.185.122.148/30
eth2 - Network: 10.185.122.220/30
eth0 Interface configuration

DEVICE=eth0
HWADDR=C8:CB:B8:C6:43:C8
TYPE=Ethernet
UUID=af01e835-57a5-41ca-aee2-2bf38b2bccac
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
USERCTL=no
PEERDNS=yes
IPV6INIT=no
IPADDR=14.x.x.22
NETMASK=255.255.255.248
DNS2=8.8.8.8
GATEWAY=14.x.x.17
DNS1=4.2.2.2
DEFROUTE=yes
ARPCHECK=no
eth1 Interface configuration:

DEVICE=eth1
HWADDR=C8:CB:B8:C6:43:C9
TYPE=Ethernet
UUID=5c8b050a-6dd9-4747-9d62-24c20e044869
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
USERCTL=no
PEERDNS=yes
DEFROUTE=no
IPV6INIT=no
IPADDR=10.185.122.150
NETMASK=255.255.255.252
DNS2=8.8.8.8
GATEWAY=10.185.122.149
DNS1=4.2.2.2
ARPCHECK=no
eth2 interface configuration

DEVICE=eth2
HWADDR=00:1b:21:39:18:c9
TYPE=Ethernet
UUID=a98b74c3-561d-4fc0-9507-70360898bae2
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
USERCTL=no
PEERDNS=yes
IPV6INIT=no
ARPCHECK=no
DEFROUTE=no
IPADDR=10.185.122.222
NETMASK=255.255.255.252
DNS2=8.8.8.8
GATEWAY=10.185.122.221
DNS1=4.2.2.2
I want to route 10.0.0.0/8 network from both eth1 and eth2 in a way that if packet is sent out from eth1 and eth2, it should go with respective interface's ip as source ip. Similarly, if packet is destined for eth1 or eth2, it should reach to their respective interface. For this I have created two separate routing table and did routing as mentioned below.

/etc/iproute2/rt_tables

100 rt1
200 rt2

ip route add 10.185.122.148/30 dev eth1 src 10.185.122.150 table rt1
ip route add default via 10.185.122.149 dev eth1 table rt1
ip rule add from 10.185.122.150/32 table rt1
ip rule add to 10.185.122.150/32 table rt1

ip route add 10.185.122.220/30 dev eth2 src 10.185.122.222 table rt2
ip route add default via 10.185.122.221 dev eth2 table rt2
ip rule add from 10.185.122.222/32 table rt2
ip rule add to 10.185.122.222/32 table rt2

#ip route show
10.185.122.148/30 dev eth1 proto kernel scope link src 10.185.122.150
10.185.122.220/30 dev eth2 proto kernel scope link src 10.185.122.222
14.x.x.16/29 dev eth0 proto kernel scope link src 14.x.x.22
169.254.0.0/16 dev eth2 scope link metric 1003
169.254.0.0/16 dev eth0 scope link metric 1004
169.254.0.0/16 dev eth1 scope link metric 1005
default via 14.x.x.17 dev eth0

#ip route show table rt1
10.185.122.148/30 dev eth1 scope link src 10.185.122.150
default via 10.185.122.149 dev eth1

#ip route show table rt2
10.185.122.220/30 dev eth2 scope link src 10.185.122.222
default via 10.185.122.221 dev eth2

#ip rule show
0: from all lookup local
32762: from all to 10.185.122.222 lookup rt2
32763: from 10.185.122.222 lookup rt2
32764: from all to 10.185.122.150 lookup rt1
32765: from 10.185.122.150 lookup rt1
32766: from all lookup main
32767: from all lookup default
configuration from /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296

After above configuration, If I ping remote server ip 10.232.130.170 which is connected via VPN connectivity from any of the interface using source IP. It is working.

# ping -I 10.185.122.150 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.150 : 56(84) bytes of data.
64 bytes from 10.232.130.170: icmp_seq=1 ttl=253 time=1.72 ms
64 bytes from 10.232.130.170: icmp_seq=2 ttl=253 time=1.50 ms
64 bytes from 10.232.130.170: icmp_seq=3 ttl=253 time=1.52 ms
64 bytes from 10.232.130.170: icmp_seq=4 ttl=253 time=1.47 ms
^C
--- 10.232.130.170 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3448ms
rtt min/avg/max/mdev = 1.478/1.559/1.728/0.098 ms


# ping -I 10.185.122.222 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.222 : 56(84) bytes of data.
64 bytes from 10.232.130.170: icmp_seq=1 ttl=253 time=3.61 ms
64 bytes from 10.232.130.170: icmp_seq=2 ttl=253 time=1.52 ms
64 bytes from 10.232.130.170: icmp_seq=3 ttl=253 time=1.83 ms
64 bytes from 10.232.130.170: icmp_seq=4 ttl=253 time=1.58 ms
64 bytes from 10.232.130.170: icmp_seq=5 ttl=253 time=1.91 ms
64 bytes from 10.232.130.170: icmp_seq=6 ttl=253 time=1.50 ms
^C
--- 10.232.130.170 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5547ms
rtt min/avg/max/mdev = 1.501/1.997/3.613/0.739 ms
But when I try to ping using interface name, it is not working.

# ping -I eth1 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.150 eth1: 56(84) bytes of data.
From 10.185.122.150 icmp_seq=2 Destination Host Unreachable
From 10.185.122.150 icmp_seq=3 Destination Host Unreachable
From 10.185.122.150 icmp_seq=4 Destination Host Unreachable
From 10.185.122.150 icmp_seq=6 Destination Host Unreachable
From 10.185.122.150 icmp_seq=7 Destination Host Unreachable
From 10.185.122.150 icmp_seq=8 Destination Host Unreachable
^C
--- 10.232.130.170 ping statistics ---
9 packets transmitted, 0 received, +6 errors, 100% packet loss, time 8090ms

# ping -I eth2 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.222 eth2: 56(84) bytes of data.
From 10.185.122.222 icmp_seq=2 Destination Host Unreachable
From 10.185.122.222 icmp_seq=3 Destination Host Unreachable
From 10.185.122.222 icmp_seq=4 Destination Host Unreachable
From 10.185.122.222 icmp_seq=6 Destination Host Unreachable
From 10.185.122.222 icmp_seq=7 Destination Host Unreachable
From 10.185.122.222 icmp_seq=8 Destination Host Unreachable
^C
--- 10.232.130.170 ping statistics ---
9 packets transmitted, 0 received, +6 errors, 100% packet loss, time 8818ms
Please advise what wrong I am doing and how to correct it. This is the first time I am posting a question on this forum. Please excuse for any mistake I have done in posting.

Thanks,
Kamlesh

This is all 'Unless you know better,' because I'm aware some very complex setups exist out there.

I'd make 2 comments:

To ping a host by name, you have to have a name --> ip translation somewhere. You can put ip --> host translations in /etc/hosts, or your dns server, as long as that doesn't update anything else. As you have 2 nics connected to the same internal network (Why?) It will probably only ping through one, because the system uses the 1st available route.

Have you a route to the network 10.185.122.0? Otherwise, you;ll hardly reach it because that's a private network. Again, I would maje the comment that it will use the first nic you specify.

Thanks for your reply.

As I mentioned I am able to ping private network 10.185.122.x from both interfaces eth0 and eth1 if I use interface ip in source. For example:

# ping -I 10.185.122.150 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.150 : 56(84) bytes of data.
64 bytes from 10.232.130.170: icmp_seq=1 ttl=253 time=1.72 ms
64 bytes from 10.232.130.170: icmp_seq=2 ttl=253 time=1.50 ms
64 bytes from 10.232.130.170: icmp_seq=3 ttl=253 time=1.52 ms
64 bytes from 10.232.130.170: icmp_seq=4 ttl=253 time=1.47 ms

It is not working if I use interface name in source.

# ping -I eth1 10.232.130.170
PING 10.232.130.170 (10.232.130.170) from 10.185.122.150 eth1: 56(84) bytes of data.
From 10.185.122.150 icmp_seq=2 Destination Host Unreachable
From 10.185.122.150 icmp_seq=3 Destination Host Unreachable
From 10.185.122.150 icmp_seq=4 Destination Host Unreachable
From 10.185.122.150 icmp_seq=6 Destination Host Unreachable
From 10.185.122.150 icmp_seq=7 Destination Host Unreachable
From 10.185.122.150 icmp_seq=8 Destination Host Unreachable
^C

I am doing so because I have got two separate SIP PRIs from my service provider on separate private IP networks and he wants me to route traffic back and forth from the respective interface.
eth1 - Network: 10.185.122.148/30
eth2 - Network: 10.185.122.220/30

Thanks
Kamlesh

Try adding rules based on the interfaces. For example:

I tried to add rule with below commands but got an error.

[root@localhost]# ip route add iff eth1 table rt1
Error: an inet prefix is expected rather than "iff".

[root@localhost]# ip route add 10.0.0.0/8 eth1 table rt1
Error: either "to" is duplicate, or "eth1" is a garbage.

Sorry, that should be 'ip rule', not 'ip route':

This also gives below error.

# ip rule add iff eth1 table rt1
Error: argument "iff" is wrong: Failed to parse rule type

Thanks
Kamlesh