I am trying to create new docker network bridge that route outgoing traffic via my 2nd network interface, I managed to make containers traffic go through the desired interface, however i lost the ability to communicate with the container from the localhost, and likewise from inside the container. Here is the configuration so far.
My Network interfaces:
Code:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.3 netmask 255.255.255.0 broadcast 10.0.0.255
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.1.3 netmask 255.255.255.0 broadcast 10.0.1.255
docker_backup: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.18.0.1 netmask 255.255.0.0 broadcast 172.18.255.255
Docker network bridge configuration
Code:
[
{
"Name": "docker_backup",
"Id": "5252a641d2087e5d4ed6aa142327a3193c2142d8fe5cac6b6a163d0154819d6d",
"Created": "2021-11-19T16:36:37.037180993+03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.enable_ip_masquerade": "false",
"com.docker.network.bridge.name": "docker_backup"
},
"Labels": {}
}
]
cat /etc/iproute2/rt_tables
Code:
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
1 docker_backup
iptables
Code:
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
SNAT all -- 172.18.0.0/16 0.0.0.0/0 to:10.0.1.3
MASQUERADE all -- 172.19.0.0/16 0.0.0.0/0
MASQUERADE all -- 172.23.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.18.0.3 172.18.0.3 tcp dpt:80
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 127.0.0.1 tcp dpt:8081 to:172.18.0.3:80
Routes and ip tables
Code:
ip rule add from 172.18.0.0/16 tab docker_backup
ip route add 172.18.0.0/16 dev eth1 tab docker_backup
ip route add default via 10.0.1.1 dev eth1 tab docker_backup
iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker_backup -j SNAT --to-source 10.0.1.3
netstat -an | grep 'LISTEN '
Code:
# netstat -an | grep 'LISTEN '
tcp 0 0 127.0.0.1:3493 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8081 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5938 0.0.0.0:* LISTEN
tcp 0 0 10.0.1.3:8083 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5941 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN
tcp 0 0 10.0.1.3:6881 0.0.0.0:* LISTEN
route
Code:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default homerouter.cpe 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
10.0.1.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 eth1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker_backup
172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-3271a2013d45
172.23.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-1855923fbb3f
docker-compose example
Code:
version: "3.8"
services:
tester:
image: linuxserver/nginx
environment:
- TZ=Asia/Kuwait
volumes:
- ./config:/config
ports:
- 127.0.0.1:8081:80
networks:
docker_backup:
ipv4_address: 172.18.0.3
networks:
docker_backup:
external: true
from the bastion host curl attempts:
Code:
(07:27:55) root@server.local ~/tests
# curl -v localhost:8081
* Trying 127.0.0.1:8081...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.68.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
(07:28:10) root@server.local ~/tests
# curl -v 172.18.0.3:80
* Trying 172.18.0.3:80...
* TCP_NODELAY set
* connect to 172.18.0.3 port 80 failed: No route to host
* Failed to connect to 172.18.0.3 port 80: No route to host
* Closing connection 0
curl: (7) Failed to connect to 172.18.0.3 port 80: No route to host
from inside the container im able to connect to the outside world using eth1 IP address
Code:
root@50ed3e196cde:/# curl -LSs ifconfig.io
37.xx.xxx.xx
however any attempts to ping any other docker or main host ip i.e. (local addresses) fails.
Code:
root@50ed3e196cde:/# curl -v --connect-timeout 5 172.18.0.1
* Trying 172.18.0.1:80...
* After 5000ms connect time, move on!
* connect to 172.18.0.1 port 80 failed: Operation timed out
* Connection timeout after 5000 ms
* Closing connection 0
curl: (28) Connection timeout after 5000 ms
So, the question is how would i modify the routes or iptables rules to make the traffic flows from local address to the container and from the container to the local addresses while maintaining the outgoing ip?
