The situation: We are building a cluster that implements an iptables firewall on every node. The firewall blocks all but a small number of specified ports in the well-known ports range (1-1023) on the INPUT chain of the real IP address.

To handle failover for a vendor's daemon, we allocate a virtual IP address, and correspond through that. That daemon opens a random output port in the range from 512 to 1023.

When the firewall is down, or when we assign the daemon to use the ethernet port's "real" IP address, the vendor's daemon works fine.

However, when the firewall is up and the daemon is using the virtual IP address, the connection is prevented. It appears that traffic outbound on the virtual IP address is winding up on the INPUT chain for the real IP address?

In the following except from tcpdump,
the daemon is running on .46
the daemon is using virtual IP .20
the client is running on .48
"tcpdump -ln -i eth1" (in promiscuous mode) is running on .48
----- snip ---------
11:16:58.703317 172.20.0.48.1023 > 172.20.0.20.6879: udp 16 (DF)
11:16:58.703489 172.20.0.46.6879 > 172.20.0.48.1023: udp 28 (DF)
11:16:58.703523 172.20.0.48 > 172.20.0.46: icmp: host 172.20.0.48 unreachable - admin prohibited [tos 0xc0]
11:17:03.703086 arp who-has 172.20.0.20 tell 172.20.0.48
11:17:03.703152 arp reply 172.20.0.20 is-at 0:30:6e:4a:82:b8
11:17:05.812507 172.20.0.48.35484 > 172.20.0.47.5666: S 1603799250:1603799250(0) win 5840 <mss 1460,sackOK,timestamp 8204997 0,nop,wscale 0> (DF)
11:17:05.812605 172.20.0.47.5666 > 172.20.0.48.35484: S 1615423288:1615423288(0) ack 1603799251 win 5792 <mss 1460,sackOK,timestamp 8191574 8204997,nop,wscale 0> (DF)
----- snip ---------

In any case, how can we allow unfettered outbound access on the virtual IP address while blocking unwanted inputs on the real IP address?

[This hadoriginally been erroneously posted in "Linux networking"]

Does the config work with the firewall down and the daemon using the vitual IP? One would think that it would be problematic that the reply is coming from an entirely different machine than the request was made to. That actually might be the problem with the firewall config. If your basing access on any kind of connection state relationships, then this likely would not match either the RELATED or ESTABLISHED states. Could you post your firewall config? (remove any routable public IPs from it beforehand)

Stupid error on my part that had nothing to do with the use of a virtual IP address. Thanks for thinking about this.

/andy

Can you enlighten us to what the problem and solution was? Someone might have a similar problem and find your solution helpful. Thanks.

It turns out that, in addition to the ports we knew about, the vendor's daemon was *also* opening random *outbound* ports in the range 1-1023. Our firewall was configured to disallow all outbound traffic, so the communication failed.

The solution was to persuade the vendor to provide an option to prevent the daemon from opening inappropriate outbound ports.