Production server with static IPs on eth0 and eth0:1

stardotstar · 07-01-2010, 12:29 AM

OK, so I have two servers on a vlan at my datacentre/colocation and previously both servers had public IPs on their eth0 interfaces.

The servers are HP ProLiant DL360s - one is a G4 and one is a G5

The newer G5 is now the LAMP server and the G4 has been retired and I want to repurpose it as an iSCSI target using openfiler freenas or similar.

My G5 has public/static IPs lashed to the eth0 physical interface and the eth1 is not configured to do anything yet.

The G4 will have both interfaces available - perhaps one for ssh access from one of my static public IPs and the other to be a private IP on the local vlan.

Here is what I am trying to get my head around...

The G5
eth0 - Public IP - full LAMP services on two or three virtual interfaces
eth1 - Private IP 192.168.0.1

The G4
eth0 - Public IP for ssh
eth1 - Private IP 192.168.0.2

Because my traffic between eth1 on these boxes is via private IPs on the local private vlan it doesn't add to my quota for bandwidth.

How do I go about configuring the routing and gateways and other aspects of this so that I can run a private IP space network between the eth1s and still serve the outside world from the eth0s...

I am afraid that if I assign the private IPs to the eth1 interfaces the routing may either not work or interfere with the access to the production internet facing interfaces (eth0s)

If anyone can see my immediate predicament and thinks they can help me understand how to proceed I would appreciate the opportunity to solve this so I can get my iSCSI on the G4...

TIA
\\'

kbp · 07-01-2010, 03:08 AM

You don't need to do any routing or set up any gateways, the eth1 interfaces are on the same network so will communicate happily with each other assuming they have layer 2 connectivity

cheers

stardotstar · 07-01-2010, 04:22 AM

Thanks, that is encouraging. I am reading on some of the associated/related/suggested threads at the moment too and have already broken things a couple of times - thank HP for iLO

nothing broken for long...

OK so I have tried something very simple:

/sbin/ifconfig eth1 192.168.0.102 up

and now have:
eth1 Link encap:Ethernet HWaddr 00:22:64:9B

7:AA
inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:64ff:fe9b:d7aa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:64 (64.0 b) TX bytes:914 (914.0 b)
Interrupt:82 Memory:fa000000-fa012800

but of course the routing is all set up to direct everything via the default route right?
So whilst I can ping 192.168.0.102 now there is no way that I can ping 192.168.0.101 (the G4)

Here is my route -n

root@jupiter [/home/stardotstar]# /sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
202.xxx.yyy.117 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
202.xxx.yyy.112 0.0.0.0 255.255.255.248 U 0 0 0 eth0
202.xxx.zzz.48 0.0.0.0 255.255.255.248 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 202.xxx.zzz.49 0.0.0.0 UG 0 0 0 eth0

So I can see that the private network is added to eth1 but how is it that I can add a route to other interfaces on the 192.168.0.0 network (ie the G4 on the same vlan with its eth0 configured to 192.168.0.101) without interfering with the production IPs?

kbp · 07-01-2010, 10:19 PM

I think you're confusing yourself:

This route:

Code:

192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

.... shows that the kernel will send traffic destined for the 192.168.0.0/24 network out the eth1 interface. You don't need a route for every machine you want to communicate with.

This route:

Code:

0.0.0.0 202.xxx.zzz.49 0.0.0.0 UG 0 0 0 eth0

... is the default route, which means that traffic for any unknown (no route in the routing table) network will be sent to 202.xxx.zzz.49 to be forwarded on.

To reach a network that is not directly connected to your server and that is not accessible via the default gateway, you need to add a static route.

A static route says 'to reach network X send the traffic to gateway Y' - where one of gateway Y's interfaces is on a the same IP network as one of your servers interfaces.

eg.
[192.168.0.1/24](gateway Y)[10.0.0.1/24]---[10.0.0.2/24](server)[10.0.1.2/24]---[10.0.1.1/24](default gateway)...

If your server wants to communicate with hosts on the 192.168.0.0/24 network, you will need to add a static route.

Hope this make things a little clearer

cheers

stardotstar · 07-01-2010, 11:17 PM

OK, thank you things are beginning to be clearer to me.
Here is where I am at

G5:
has its internet production interface as originally configured - all working
PLUS I have manually upped the eth1 with the private address space.

Code:

root@jupiter [~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:22:64:9B:D7:98  
          inet addr:xxx.yyy.zzz.50  Bcast:xxx.yyy.zz.55  Mask:255.255.255.248
          inet6 addr: fe80::222:64ff:fe9b:d798/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3471398 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4274612 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:695683338 (663.4 MiB)  TX bytes:167763704 (159.9 MiB)
          Interrupt:185 Memory:f8000000-f8012800 

eth0:1    Link encap:Ethernet  HWaddr 00:22:64:9B:D7:98  
          inet addr:xxx.yyy.nn.117  Bcast:xxx.yyy.nn.119  Mask:255.255.255.248
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:185 Memory:f8000000-f8012800 

eth1      Link encap:Ethernet  HWaddr 00:22:64:9B:D7:AA  
          inet addr:192.168.0.102  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::222:64ff:fe9b:d7aa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:63 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4184 (4.0 KiB)  TX bytes:1462 (1.4 KiB)
          Interrupt:82 Memory:fa000000-fa012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:39717664 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39717664 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3075190905 (2.8 GiB)  TX bytes:3075190905 (2.8 GiB)

You can see the two production interfaces eth0 and eth0:1
I can confirm that there is a network route for the private network bound to eth1 created as part of upping the interface.
I can also confirm that the media is linked up...

Code:

root@jupiter [~]# mii-tool 
eth0: negotiated 100baseTx-FD, link ok
eth1: negotiated 100baseTx-FD, link ok
root@jupiter [~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
xxx.yyy.nn.117  0.0.0.0         255.255.255.255 UH    0      0        0 eth0
xxx.yyy.nn.112  0.0.0.0         255.255.255.248 U     0      0        0 eth0
xxx.yyy.zz.48   0.0.0.0         255.255.255.248 U     0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         xxx.yyy.zz.49   0.0.0.0         UG    0      0        0 eth0
root@jupiter [~]#

Now on G4 I have a very similar setup (the physical adapters I am using are switched and the public one only needed for ssh though)

Code:

helios stardotstar # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:12:79:90:b0:16  
          inet addr:192.168.0.101  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::212:79ff:fe90:b016/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:47282 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3491610 (3.3 MiB)  TX bytes:932 (932.0 B)
          Interrupt:25 

eth1      Link encap:Ethernet  HWaddr 00:12:79:90:b0:15  
          inet addr:xxx.yyy.nn.116  Bcast:xxx.yyy.nn.119  Mask:255.255.255.248
          inet6 addr: fe80::212:79ff:fe90:b015/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:79683 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19119 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:28206482 (26.8 MiB)  TX bytes:5809673 (5.5 MiB)
          Interrupt:26 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:46660 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46660 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3572767 (3.4 MiB)  TX bytes:3572767 (3.4 MiB)

and the link and routing looks fine:

Code:

helios stardotstar # mii-tool
eth0: negotiated 100baseTx-FD, link ok
eth1: negotiated 100baseTx-FD, link ok
helios stardotstar # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
xxx.yyy.nn.112  0.0.0.0         255.255.255.248 U     0      0        0 eth1
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         xxx.yyy.nn.113  0.0.0.0         UG    1      0        0 eth1

Following the understanding that a specific route shoudn't be necessary for packets for the 192.168.0.0 network to find their way onto the right physical adapter and thus the connected media and physical layer I expected that from each box I would be able to ping the other's private IP...

But as you can see here on each system the only ping that returns is the one to the local ip bound to the host's adapter...

G5

Code:

root@jupiter [~]# ping 192.168.0.102
PING 192.168.0.102 (192.168.0.102) 56(84) bytes of data.
64 bytes from 192.168.0.102: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 192.168.0.102: icmp_seq=2 ttl=64 time=0.018 ms
64 bytes from 192.168.0.102: icmp_seq=3 ttl=64 time=0.028 ms

--- 192.168.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.018/0.026/0.034/0.008 ms
root@jupiter [~]# ping 192.168.0.101
PING 192.168.0.101 (192.168.0.101) 56(84) bytes of data.

--- 192.168.0.101 ping statistics ---
28 packets transmitted, 0 received, 100% packet loss, time 27000ms

root@jupiter [~]#

and same holds for the G4

Code:

helios stardotstar # ping 192.168.0.101
PING 192.168.0.101 (192.168.0.101) 56(84) bytes of data.
64 bytes from 192.168.0.101: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 192.168.0.101: icmp_seq=2 ttl=64 time=0.028 ms
64 bytes from 192.168.0.101: icmp_seq=3 ttl=64 time=0.022 ms
64 bytes from 192.168.0.101: icmp_seq=4 ttl=64 time=0.024 ms
^C
--- 192.168.0.101 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.022/0.031/0.052/0.013 ms
helios stardotstar # ping 192.168.0.102
PING 192.168.0.102 (192.168.0.102) 56(84) bytes of data.
^C
--- 192.168.0.102 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12017ms

I guess I am now thinking that there may be something else wrong? Because as you say if there is not a need to define even a static route (ie I'm trying to reach two host addresses on the very same network 192.168.0.101 and 192.168.0.102 - not on another like you say 192.168.1.x)

Perhaps there is something else wrong.

What I can say beyond all this is that I can ping the public IPs from the boxes and ssh between them and so forth. But I am guessing that is all going outside the vlan due to the vlan routing table setup by the CoLo...

I have attached a pic to ensure that my desired config is beyond doubt.

\\'

[edit] I just got a sense of the problem from a static route point of view from what you said here:

Quote:

not accessible via the default gateway, you need to add a static route.

THe private space is not accessible by my default route and therefore even though the network is attached to the server in both cases the packets are likely being routed out via the default route (public network) and lost...

So I need to add the correct static route to the table so that all packets for 192.168.0.0 go to the appropriate ethX...

kbp · 07-02-2010, 03:02 AM

Quote:

So I need to add the correct static route to the table so that all packets for 192.168.0.0 go to the appropriate ethX...

.. no.

I'd suggest that you're probably lacking layer 2 connectivity. Either the eth0 and eth1 nics are patched into different switches or the switch ports are assigned to different vlans.. you'll need the network guys to tell you / fix things.

Ideally your config would look something like :

[server1:<internet>]-----vlanX------[server2:<internet>]
[server1:<private>]------vlanY-------[server2:<private>]

cheers

stardotstar · 07-02-2010, 05:46 AM

Thanks kpb!

I have lodged a ticket and see what I can find out from them.

Is there any tests I can do from either server to determine if this is the place where the problem lies.

Can i determine anything from the config of the other interfaces which are routing to the internet>

I am supposedly on one vlan - I have been given numbers for my interfaces on the switch and a number (107) which I understood to be my private vlan.

I run two public subnets with like 6 total hosts addresses - broadcast, network, gateway + 3 usable IIRC

I appreciate your insight and suggestions kpb thankyou

\\'

kbp · 07-02-2010, 09:51 AM

The fact that you can't ping one server from the other suggests that the interfaces are not on the same vlan... that's about the extent of the testing you can do

cheers

stardotstar · 07-03-2010, 12:27 AM

The plot thickens as it takes a turn in a new direction!

I thought that perhaps an over zealous firewall configuration may be interfering in all this and so I stopped the csf firewall on the G5 and stopped the iptables then did the same for the shorewall and iptables on the G4

Sure enough - PROGRESS.

So the Layer2 Connectivity is Fine (I have closed my ticket)

Now I am onto learning the finer points of my firewalls - thanks for all the help and encouragement guys!@