LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Virtualization and Cloud (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/)
-   -   [Openstack/Devstack] Cannot reach Internet via Floating IP (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/%5Bopenstack-devstack%5D-cannot-reach-internet-via-floating-ip-4175676401/)

czezz 06-03-2020 06:15 AM
[Openstack/Devstack] Cannot reach Internet via Floating IP
 
Hi all,
This is my devstack lab installed on VirtualBox.
Code:
stack@vlab:~$ cat /etc/devstack-version
DevStack Version: stein
Change: 669b0c5e4eba1a721dc02650e000fb0ac6bfd11b Merge "Remove crusty old python 3 package version logic" into stable/stein 2019-04-26 09:32:45 +0000
OS Version: Ubuntu 16.04 xenial
I have created instance (vm: heat_server), with floating IP/port hoping it will have access to internet.
Code:
stack@vlab:~$ openstack server list
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
| ID                                  | Name        | Status  | Networks                          | Image                    | Flavor  |
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
| 6f8fbc70-dfc5-4840-ba18-cf7869863af6 | heat_server | ACTIVE  | heat_network=10.1.1.4, 172.24.4.10 | cirros-0.4.0-x86_64-disk | m1.tiny |
| 0eb31f9d-4524-4382-9868-6bb1df1d84fd | testvm2    | SHUTOFF | fxnet=10.2.0.5                    | cirros-0.4.0-x86_64-disk | m1.tiny |
| 4766f91f-b1ec-44f6-96c4-6d1164ba754e | testvm1    | SHUTOFF | fxnet=10.2.0.11                    | cirros-0.4.0-x86_64-disk | m1.tiny |
+--------------------------------------+-------------+---------+------------------------------------+--------------------------+---------+
Unfortunately it does not reach Internet.
I did a small investigation:
1. I checked for network "public" - OK
2. I checked for subnet "public-subnet" - OK
3. I checked for router "public" and I see - NONE

Questions:
1. Could that be a reason of not reaching internet from my instance (missing public router)?
2. Is this true that connectivity to internet via Floating IP is in fact NATed by Neutron via internal network (in my case 10.1.1.x) ?
Code:
openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                  | Floating IP Address | Fixed IP Address | Port                                | Floating Network                    | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 54cd81a9-771c-475f-94db-d69fe92bc512 | 172.24.4.10        | 10.1.1.4        | e6681d9c-8456-4cf6-ae07-71abce21a823 | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1 | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+


Listings:

stack@vlab:~$ openstack network show public
+---------------------------+----------------------------------------------------------------------------+
| Field                    | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                        |
| availability_zone_hints  |                                                                            |
| availability_zones        | nova                                                                      |
| created_at                | 2019-05-07T13:47:00Z                                                      |
| description              |                                                                            |
| dns_domain                | None                                                                      |
| id                        | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1                                      |
| ipv4_address_scope        | None                                                                      |
| ipv6_address_scope        | None                                                                      |
| is_default                | True                                                                      |
| is_vlan_transparent      | None                                                                      |
| location                  | None                                                                      |
| mtu                      | 1500                                                                      |
| name                      | public                                                                    |
| port_security_enabled    | True                                                                      |
| project_id                | 7e31368f4f9a4b31a91257394592fef2                                          |
| provider:network_type    | flat                                                                      |
| provider:physical_network | public                                                                    |
| provider:segmentation_id  | None                                                                      |
| qos_policy_id            | None                                                                      |
| revision_number          | 3                                                                          |
| router:external          | External                                                                  |
| segments                  | None                                                                      |
| shared                    | False                                                                      |
| status                    | ACTIVE                                                                    |
| subnets                  | 51bd9836-acf8-4369-8f4c-8466307694a7, fec1ef4c-d70d-43d8-8c6e-8ea03325f52f |
| tags                      |                                                                            |
| updated_at                | 2019-05-07T13:47:52Z                                                      |
+---------------------------+----------------------------------------------------------------------------+


stack@vlab:~$ openstack subnet show public-subnet
+-------------------+--------------------------------------+
| Field            | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 172.24.4.2-172.24.4.254              |
| cidr              | 172.24.4.0/24                        |
| created_at        | 2019-05-07T13:47:14Z                |
| description      |                                      |
| dns_nameservers  |                                      |
| enable_dhcp      | False                                |
| gateway_ip        | 172.24.4.1                          |
| host_routes      |                                      |
| id                | fec1ef4c-d70d-43d8-8c6e-8ea03325f52f |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                |
| ipv6_ra_mode      | None                                |
| location          | None                                |
| name              | public-subnet                        |
| network_id        | e0ef1d2b-1d8e-42e0-9b3a-ecbe784f24e1 |
| project_id        | 7e31368f4f9a4b31a91257394592fef2    |
| revision_number  | 0                                    |
| segment_id        | None                                |
| service_types    |                                      |
| subnetpool_id    | None                                |
| tags              |                                      |
| updated_at        | 2019-05-07T13:47:14Z                |
+-------------------+--------------------------------------+

stack@vlab:~$ openstack router list
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+
| ID                                  | Name        | Status | State | Distributed | HA    | Project                          |
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+
| 1bb436b3-fe08-4b38-8542-30587a87bc40 | heat_router | ACTIVE | UP    | False      | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
| a71b95db-d12e-46de-85b5-14d4368186be | router1    | ACTIVE | UP    | False      | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
| beba3591-e0f2-4f03-a70a-b2022c9aca01 | fxrouter    | ACTIVE | UP    | False      | False | a5e6e3dbad9f4f4b8810fe0a1246e1f7 |
+--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+

berndbausch 06-04-2020 12:37 AM
Devstack creates a "fake" external network which is not connected to the external world at all, except if you deliberately connect it. For instructions, see https://docs.openstack.org/devstack/...essible-guests.

czezz 06-04-2020 05:17 PM
I made it finally to work.
But now my problem is that after instance reboot, I cannot access it via Floating IP :(
I can still get in via internal network (via qrouter).

berndbausch 06-04-2020 05:52 PM
If you want assistance, you need to describe your setup.

czezz 06-05-2020 02:58 AM
Hi Berndbausch,
Thank you for reply.

I did fresh installation of Devstack (from Master branch - DevStack Version: victoria)
In local.conf I have specified:
Code:
HOST_IP=192.168.56.109 - this is eth1/enp0s8 and refers to Host-Only interface in my VirtualBox
FLAT_INTERFACE=enp0s8
FLOATING_RANGE=192.168.56.224/27 - for this installation I assigned floating IPs on the same class as HOST_IP (its not clear yet to me whether it should be the same class or should be different. The point is that by adding Floating IP to the instance I could access it from Devstack itself - and that is/was OK for the moment).
I have deployed stack using this example https://docs.ukcloud.com/articles/op...-template.html , with 2 changes:
1. Removed volume (for some reason it gives errors and blocks deployment - to troubleshoot later)
2. Replace network "Internet" with "Public"


Creation went quite smooth:
Code:
stack@vlab:~$ openstack stack list
+--------------------------------------+-------------+-----------------+----------------------+--------------+
| ID                                  | Stack Name  | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+-------------+-----------------+----------------------+--------------+
| 8404831a-fd3b-46b6-8001-55484df072fd | basic-stack | CREATE_COMPLETE | 2020-06-04T20:42:31Z | None        |
+--------------------------------------+-------------+-----------------+----------------------+--------------+

stack@vlab:~$ heat resource-list basic-stack
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
| resource_name        | physical_resource_id                                                                | resource_type                      | resource_status | updated_time        |
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
| heat_network          | cd78103f-e606-40a7-8399-b1e99651757f                                                | OS::Neutron::Net                  | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_network_subnet  | d64acd35-b70b-4dd8-9a51-34a70aaf9f45                                                | OS::Neutron::Subnet                | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_router          | 7563b1f4-2635-412b-8053-dda164a5b279                                                | OS::Neutron::Router                | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_router_interface | 7563b1f4-2635-412b-8053-dda164a5b279:subnet_id=d64acd35-b70b-4dd8-9a51-34a70aaf9f45 | OS::Neutron::RouterInterface      | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server          | 95fce8b8-9d4d-4ccd-a17b-cb367537e504                                                | OS::Nova::Server                  | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_ip_assoc  | 1                                                                                  | OS::Neutron::FloatingIPAssociation | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_port      | cc3eef0e-7f01-4c5b-841e-ade976ae0dbc                                                | OS::Neutron::Port                  | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
| heat_server_public_ip | 882e79f8-f983-430c-b3e3-3758fa3158e7                                                | OS::Neutron::FloatingIP            | CREATE_COMPLETE | 2020-06-04T20:42:31Z |
+-----------------------+-------------------------------------------------------------------------------------+------------------------------------+-----------------+----------------------+
After adding 2 rules to the "default" security group:
Code:
openstack security group rule create default --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0
openstack security group rule create --protocol icmp default
I could access instance/vm heat_server via 192.168.56.227 from Devstack.
After reboot of the instance and later also all Devstack Vbox VM, I have no more access via 192.168.56.227.
I can still access it via: sudo ip netns exec qrouter-7563b1f4-2635-412b-8053-dda164a5b279 ssh cirros@10.1.1.9
Code:
stack@vlab:~$ openstack server list
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
| ID                                  | Name        | Status  | Networks                                                | Image                    | Flavor  |
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
| 95fce8b8-9d4d-4ccd-a17b-cb367537e504 | heat_server | ACTIVE  | heat_network=10.1.1.9, 192.168.56.227                  | cirros-0.5.1-x86_64-disk | m1.tiny |
| 1935c01d-7451-439d-9335-a0c2690e145a | vm2        | SHUTOFF | private=10.0.0.21, fd07:a88f:8dd3:0:f816:3eff:fee1:9f91 | cirros-0.5.1-x86_64-disk | m1.tiny |
+--------------------------------------+-------------+---------+---------------------------------------------------------+--------------------------+---------+
I dont know where to look now / where to start troubleshooting.

berndbausch 06-05-2020 04:10 AM
The IP address of external bridge br-ex is not persistent. When you reboot, you lose it. Same for the physical volumes used by Cinder and the filesystem used by Swift.

I use this script to recreate them (only tested on Ussuri and a few earlier versions, and it uses the standard IP address):
Code:
$ cat restore-after-reboot
sudo ip a add 172.24.4.1/24 dev br-ex
sudo ip link set dev br-ex up

sudo mount -t xfs -o loop,noatime,nodiratime,logbufs=8 /opt/stack/data/swift/drives/images/swift.img /opt/stack/data/swift/drives/sdb1

if ! losetup | grep -q volumes-default-backing-file
then
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-default-backing-file
else
    echo stack-volumes-default-backing-file exists already
fi

if ! losetup | grep -q volumes-lvmdriver-1-backing-file
then
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
else
    echo stack-volumes-lvmdriver-1-backing-file exists already
fi

sudo systemctl restart devstack@c-vol

czezz 06-05-2020 04:42 AM
YES!
Thank you very much for this information.

Floating IP:
I was not aware that br-ex IP is not persistent.
Adding IP and setting int up solves the problem.


Volume:
I didnt test it yet. Hopefully later today in the evening.
Will let know about the results here.

Update: 2020.06.15
Volume tested and it works with solution provided by Berndbausch

willlq 03-05-2024 03:23 AM
same here
 
this can help
```
sudo ip addr flush ens33
sudo ovs-vsctl add-port br-ex ens33
sudo ip addr add 192.168.72.33/24 dev br-ex

```


All times are GMT -5. The time now is 11:04 PM.