1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) Open Platform for NFV Project, Inc. and its contributors
12 This document contains details about using OPNFV Fuel ``Gambia`` release after
13 it was deployed. For details on how to deploy OpenStack, check
14 the installation instructions in the :ref:`fuel_userguide_references` section.
16 This is an unified documentation for both ``x86_64`` and ``aarch64``
17 architectures. All information is common for both architectures
18 except when explicitly stated.
23 Fuel uses several networks to deploy and administer the cloud:
25 +------------------+----------------------------------------------------------+
26 | Network name | Description |
28 +==================+==========================================================+
29 | **PXE/admin** | Used for booting the nodes via PXE and/or Salt |
31 +------------------+----------------------------------------------------------+
32 | **mcpcontrol** | Used to provision the infrastructure hosts (Salt & MaaS) |
33 +------------------+----------------------------------------------------------+
34 | **management** | Used for internal communication between |
35 | | OpenStack components |
36 +------------------+----------------------------------------------------------+
37 | **internal** | Used for VM data communication within the |
38 | | cloud deployment |
39 +------------------+----------------------------------------------------------+
40 | **public** | Used to provide Virtual IPs for public endpoints |
41 | | that are used to connect to OpenStack services APIs. |
42 | | Used by Virtual machines to access the Internet |
43 +------------------+----------------------------------------------------------+
45 These networks - except ``mcpcontrol`` - can be Linux bridges configured
46 before the deploy on the Jumpserver.
47 If they don't exists at deploy time, they will be created by the scripts as
48 ``libvirt`` managed networks.
50 Network ``mcpcontrol``
51 ~~~~~~~~~~~~~~~~~~~~~~
53 ``mcpcontrol`` is a virtual network, managed by libvirt. Its only purpose is to
54 provide a simple method of assigning an arbitrary ``INSTALLER_IP`` to the Salt
55 master node (``cfg01``), to maintain backwards compatibility with old OPNFV
56 Fuel behavior. Normally, end-users only need to change the ``INSTALLER_IP`` if
57 the default CIDR (``10.20.0.0/24``) overlaps with existing lab networks.
59 ``mcpcontrol`` has both NAT and DHCP enabled, so the Salt master (``cfg01``)
60 and the MaaS VM (``mas01``, when present) get assigned predefined IPs (``.2``,
61 ``.3``, while the jumpserver bridge port gets ``.1``).
63 +------------------+---------------------------+-----------------------------+
64 | Host | Offset in IP range | Default address |
65 +==================+===========================+=============================+
66 | ``jumpserver`` | 1st | ``10.20.0.1`` |
67 +------------------+---------------------------+-----------------------------+
68 | ``cfg01`` | 2nd | ``10.20.0.2`` |
69 +------------------+---------------------------+-----------------------------+
70 | ``mas01`` | 3rd | ``10.20.0.3`` |
71 +------------------+---------------------------+-----------------------------+
73 This network is limited to the ``jumpserver`` host and does not require any
81 ``PXE/admin`` does not usually use an IP range offset in ``IDF``.
85 During ``MaaS`` commissioning phase, IP addresses are handed out by
90 Default addresses in below table correspond to a ``PXE/admin`` CIDR of
91 ``192.168.11.0/24`` (the usual value used in OPNFV labs).
93 This is defined in ``IDF`` and can easily be changed to something else.
95 .. TODO: detail MaaS DHCP range start/end
97 +------------------+-----------------------+---------------------------------+
98 | Host | Offset in IP range | Default address |
99 +==================+=======================+=================================+
100 | ``jumpserver`` | 1st | ``192.168.11.1`` |
101 | | | (manual assignment) |
102 +------------------+-----------------------+---------------------------------+
103 | ``cfg01`` | 2nd | ``192.168.11.2`` |
104 +------------------+-----------------------+---------------------------------+
105 | ``mas01`` | 3rd | ``192.168.11.3`` |
106 +------------------+-----------------------+---------------------------------+
107 | ``prx01``, | 4th, | ``192.168.11.4``, |
108 | ``prx02`` | 5th | ``192.168.11.5`` |
109 +------------------+-----------------------+---------------------------------+
110 | ``gtw01``, | ... | ``...`` |
113 +------------------+-----------------------+---------------------------------+
117 +------------------+-----------------------+---------------------------------+
121 +------------------+-----------------------+---------------------------------+
125 +------------------+-----------------------+---------------------------------+
129 +------------------+-----------------------+---------------------------------+
133 +------------------+-----------------------+---------------------------------+
137 +------------------+-----------------------+---------------------------------+
147 +------------------+-----------------------+---------------------------------+
151 +------------------+-----------------------+---------------------------------+
153 Network ``management``
154 ~~~~~~~~~~~~~~~~~~~~~~
158 ``management`` often has an IP range offset defined in ``IDF``.
162 Default addresses in below table correspond to a ``management`` IP range of
163 ``172.16.10.10-172.16.10.254`` (one of the commonly used values in OPNFV
164 labs). This is defined in ``IDF`` and can easily be changed to something
165 else. Since the ``jumpserver`` address is manually assigned, this is
166 usually not subject to the IP range restriction in ``IDF``.
168 +------------------+-----------------------+---------------------------------+
169 | Host | Offset in IP range | Default address |
170 +==================+=======================+=================================+
171 | ``jumpserver`` | N/A | ``172.16.10.1`` |
172 | | | (manual assignment) |
173 +------------------+-----------------------+---------------------------------+
174 | ``cfg01`` | 1st | ``172.16.10.11`` |
175 +------------------+-----------------------+---------------------------------+
176 | ``mas01`` | 2nd | ``172.16.10.12`` |
177 +------------------+-----------------------+---------------------------------+
178 | ``prx`` | 3rd, | ``172.16.10.13``, |
180 | ``prx01``, | 4th, | ``172.16.10.14``, |
181 | ``prx02`` | 5th | ``172.16.10.15`` |
182 +------------------+-----------------------+---------------------------------+
183 | ``gtw01``, | ... | ``...`` |
186 +------------------+-----------------------+---------------------------------+
192 +------------------+-----------------------+---------------------------------+
198 +------------------+-----------------------+---------------------------------+
204 +------------------+-----------------------+---------------------------------+
210 +------------------+-----------------------+---------------------------------+
216 +------------------+-----------------------+---------------------------------+
222 +------------------+-----------------------+---------------------------------+
240 +------------------+-----------------------+---------------------------------+
244 +------------------+-----------------------+---------------------------------+
251 ``internal`` does not usually use an IP range offset in ``IDF``.
255 Default addresses in below table correspond to an ``internal`` CIDR of
256 ``10.1.0.0/24`` (the usual value used in OPNFV labs).
257 This is defined in ``IDF`` and can easily be changed to something else.
259 +------------------+------------------------+--------------------------------+
260 | Host | Offset in IP range | Default address |
261 +==================+========================+================================+
262 | ``jumpserver`` | N/A | ``10.1.0.1`` |
263 | | | (manual assignment, optional) |
264 +------------------+------------------------+--------------------------------+
265 | ``gtw01``, | 1st, | ``10.1.0.2``, |
266 | ``gtw02``, | 2nd, | ``10.1.0.3``, |
267 | ``gtw03`` | 3rd | ``10.1.0.4`` |
268 +------------------+------------------------+--------------------------------+
269 | ``cmp001``, | 4th, | ``10.1.0.5``, |
270 | ``cmp002``, | 5th, | ``10.1.0.6``, |
271 | ``...`` | ... | ``...`` |
272 +------------------+------------------------+--------------------------------+
279 ``public`` often has an IP range offset defined in ``IDF``.
283 Default addresses in below table correspond to a ``public`` IP range of
284 ``172.30.10.100-172.30.10.254`` (one of the used values in OPNFV
285 labs). This is defined in ``IDF`` and can easily be changed to something
286 else. Since the ``jumpserver`` address is manually assigned, this is
287 usually not subject to the IP range restriction in ``IDF``.
289 +------------------+------------------------+--------------------------------+
290 | Host | Offset in IP range | Default address |
291 +==================+========================+================================+
292 | ``jumpserver`` | N/A | ``172.30.10.72`` |
293 | | | (manual assignment, optional) |
294 +------------------+------------------------+--------------------------------+
295 | ``prx``, | 1st, | ``172.30.10.101``, |
297 | ``prx01``, | 2nd, | ``172.30.10.102``, |
298 | ``prx02`` | 3rd | ``172.30.10.103`` |
299 +------------------+------------------------+--------------------------------+
300 | ``gtw01``, | 4th, | ``172.30.10.104``, |
301 | ``gtw02``, | 5th, | ``172.30.10.105``, |
302 | ``gtw03`` | 6th | ``172.30.10.106`` |
303 +------------------+------------------------+--------------------------------+
304 | ``ctl01``, | ... | ``...`` |
307 +------------------+------------------------+--------------------------------+
309 +------------------+------------------------+--------------------------------+
313 +------------------+------------------------+--------------------------------+
315 Accessing the Salt Master Node (``cfg01``)
316 ==========================================
318 The Salt Master node (``cfg01``) runs a ``sshd`` server listening on
321 To login as ``ubuntu`` user, use the RSA private key ``/var/lib/opnfv/mcp.rsa``:
323 .. code-block:: console
325 jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no \
326 -i /var/lib/opnfv/mcp.rsa \
332 User ``ubuntu`` has sudo rights.
336 The Salt master IP (``10.20.0.2``) is not hard set, it is configurable via
337 ``INSTALLER_IP`` during deployment.
341 Starting with the ``Gambia`` release, ``cfg01`` is containerized, so this
342 also works (from ``jumpserver`` only):
344 .. code-block:: console
346 jenkins@jumpserver:~$ docker exec -it fuel bash
349 Accessing Cluster Nodes
350 =======================
352 Logging in to cluster nodes is possible from the Jumpserver, Salt Master etc.
354 .. code-block:: console
356 jenkins@jumpserver:~$ ssh -i /var/lib/opnfv/mcp.rsa ubuntu@192.168.11.52
360 ``/etc/hosts`` on ``cfg01`` has all the cluster hostnames, which can be
361 used instead of IP addresses.
363 ``/root/.ssh/config`` on ``cfg01`` configures the default user and key:
364 ``ubuntu``, respectively ``/root/fuel/mcp/scripts/mcp.rsa``.
366 .. code-block:: console
368 root@cfg01:~$ ssh ctl01
370 Debugging ``MaaS`` Comissioning/Deployment Issues
371 =================================================
373 One of the most common issues when setting up a new POD is ``MaaS`` failing to
374 commission/deploy the nodes, usually timing out after a couple of retries.
376 Such failures might indicate misconfiguration in ``PDF``/``IDF``, ``TOR``
377 switch configuration or even faulty hardware.
379 Here are a couple of pointers for isolating the problem.
381 Accessing the ``MaaS`` Dashboard
382 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
384 ``MaaS`` web-based dashboard is available at
385 ``http://<mas01 IP address>:5240/MAAS``, e.g.
386 ``http://172.16.10.12:5240/MAAS``.
388 The administrator credentials are ``opnfv``/``opnfv_secret``.
392 ``mas01`` VM does not automatically get assigned an IP address in the
393 public network segment. If ``MaaS`` dashboard should be accesiable from
394 the public network, such an address can be manually added to the last
395 VM NIC interface in ``mas01`` (which is already connected to the public
398 Ensure Commission/Deploy Timeouts Are Not Too Small
399 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
401 Some hardware takes longer to boot or to run the initial scripts during
402 commissioning/deployment phases. If that's the case, ``MaaS`` will time out
403 waiting for the process to finish. ``MaaS`` logs will reflect that, and the
404 issue is usually easy to observe on the nodes' serial console - if the node
405 seems to PXE-boot the OS live image, starts executing cloud-init/curtin
406 hooks without spilling critical errors, then it is powered down/shut off,
407 most likely the timeout was hit.
409 To access the serial console of a node, see your board manufacturer's
410 documentation. Some hardware no longer has a physical serial connector these
411 days, usually being replaced by a vendor-specific software-based interface.
413 If the board supports ``SOL`` (Serial Over LAN) over ``IPMI`` lanplus protocol,
414 a simpler solution to hook to the serial console is to use ``ipmitool``.
418 Early boot stage output might not be shown over ``SOL``, but only over
419 the video console provided by the (vendor-specific) interface.
421 .. code-block:: console
423 jenkins@jumpserver:~$ ipmitool -H <host BMC IP> -U <user> -P <pass> \
424 -I lanplus sol activate
426 To bypass this, simply set a larger timeout in the ``IDF``.
428 Check Jumpserver Network Configuration
429 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
431 .. code-block:: console
433 jenkins@jumpserver:~$ brctl show
434 jenkins@jumpserver:~$ ifconfig -a
436 +-----------------------+------------------------------------------------+
437 | Configuration item | Expected behavior |
438 +=======================+================================================+
439 | IP addresses assigned | IP addresses should be assigned to the bridge, |
440 | to bridge ports | and not to individual bridge ports |
441 +-----------------------+------------------------------------------------+
443 Check Network Connectivity Between Nodes on the Jumpserver
444 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
446 ``cfg01`` is a Docker container running on the ``jumpserver``, connected to
447 Docker networks (created by docker-compose automatically on container up),
448 which in turn are connected using veth pairs to their ``libvirt`` managed
451 For example, the ``mcpcontrol`` network(s) should look like below.
453 .. code-block:: console
455 jenkins@jumpserver:~$ brctl show mcpcontrol
456 bridge name bridge id STP enabled interfaces
457 mcpcontrol 8000.525400064f77 yes mcpcontrol-nic
461 jenkins@jumpserver:~$ docker network ls
462 NETWORK ID NAME DRIVER SCOPE
463 81a0fdb3bd78 docker-compose_docker-mcpcontrol macvlan local
466 jenkins@jumpserver:~$ docker network inspect docker-compose_mcpcontrol
469 "Name": "docker-compose_mcpcontrol",
472 "parent": "veth_mcp1"
477 Before investigating the rest of the cluster networking configuration, the
478 first thing to check is that ``cfg01`` has network connectivity to other
479 jumpserver hosted nodes, e.g. ``mas01`` and to the jumpserver itself
480 (provided that the jumpserver has an IP address in that particular network
483 .. code-block:: console
485 jenkins@jumpserver:~$ docker exec -it fuel bash
486 root@cfg01:~# ifconfig -a | grep inet
487 inet addr:10.20.0.2 Bcast:0.0.0.0 Mask:255.255.255.0
488 inet addr:172.16.10.2 Bcast:0.0.0.0 Mask:255.255.255.0
489 inet addr:192.168.11.2 Bcast:0.0.0.0 Mask:255.255.255.0
491 For each network of interest (``mcpcontrol``, ``mgmt``, ``PXE/admin``), check
492 that ``cfg01`` can ping the jumpserver IP in that network segment, as well as
493 the ``mas01`` IP in that network.
497 ``mcpcontrol`` is set up at VM bringup, so it should always be available,
498 while the other networks are configured by Salt as part of the
499 ``virtual_init`` STATE file.
501 .. code-block:: console
503 root@cfg01:~# ping -c1 10.20.0.1 # mcpcontrol jumpserver IP
504 root@cfg01:~# ping -c1 10.20.0.3 # mcpcontrol mas01 IP
508 ``mcpcontrol`` CIDR is configurable via ``INSTALLER_IP`` env var during
509 deployment. However, IP offsets inside that segment are hard set to ``.1``
510 for the jumpserver, ``.2`` for ``cfg01``, respectively to ``.3`` for
513 .. code-block:: console
515 root@cfg01:~# salt 'mas*' pillar.item --out yaml \
516 _param:infra_maas_node01_deploy_address \
517 _param:infra_maas_node01_address
518 mas01.mcp-ovs-noha.local:
519 _param:infra_maas_node01_address: 172.16.10.12
520 _param:infra_maas_node01_deploy_address: 192.168.11.3
522 root@cfg01:~# ping -c1 192.168.11.1 # PXE/admin jumpserver IP
523 root@cfg01:~# ping -c1 192.168.11.3 # PXE/admin mas01 IP
524 root@cfg01:~# ping -c1 172.16.10.1 # mgmt jumpserver IP
525 root@cfg01:~# ping -c1 172.16.10.12 # mgmt mas01 IP
529 Jumpserver IP addresses for ``PXE/admin``, ``mgmt`` and ``public`` bridges
530 are user-chosen and manually set, so above snippets should be adjusted
531 accordingly if the user chose a different IP, other than ``.1`` in each
534 Alternatively, a quick ``nmap`` scan would work just as well.
536 .. code-block:: console
538 root@cfg01:~# apt update && apt install -y nmap
539 root@cfg01:~# nmap -sn 10.20.0.0/24 # expected: cfg01, mas01, jumpserver
540 root@cfg01:~# nmap -sn 192.168.11.0/24 # expected: cfg01, mas01, jumpserver
541 root@cfg01:~# nmap -sn 172.16.10.0/24 # expected: cfg01, mas01, jumpserver
543 Check ``DHCP`` Reaches Cluster Nodes
544 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
546 One common symptom observed during failed commissioning is that ``DHCP`` does
547 not work as expected between cluster nodes (baremetal nodes in the cluster; or
548 virtual machines on the jumpserver in case of ``hybrid`` deployments) and
551 To confirm or rule out this possibility, monitor the serial console output of
552 one (or more) cluster nodes during ``MaaS`` commissioning. If the node is
553 properly configured to attempt PXE boot, yet it times out waiting for an IP
554 address from ``mas01`` ``DHCP``, it's worth checking that ``DHCP`` packets
555 reach the ``jumpserver``, respectively the ``mas01`` VM.
557 .. code-block:: console
559 jenkins@jumpserver:~$ sudo apt update && sudo apt install -y dhcpdump
560 jenkins@jumpserver:~$ sudo dhcpdump -i admin_br
564 If ``DHCP`` requests are present, but no replies are sent, ``iptables``
565 might be interfering on the jumpserver.
570 If networking looks fine, yet nodes still fail to commission and/or deploy,
571 ``MaaS`` logs might offer more details about the failure:
573 * ``/var/log/maas/maas.log``
574 * ``/var/log/maas/rackd.log``
575 * ``/var/log/maas/regiond.log``
579 If the problem is with the cluster node and not on the ``MaaS`` server,
580 node's kernel logs usually contain useful information.
581 These are saved via rsyslog on the ``mas01`` node in
582 ``/var/log/maas/rsyslog``.
584 Recovering Failed Deployments
585 =============================
587 The first deploy attempt might fail due to various reasons. If the problem
588 is not systemic (i.e. fixing it will not introduce incompatible configuration
589 changes, like setting a different ``INSTALLER_IP``), the environment is safe
590 to be reused and the deployment process can pick up from where it left off.
592 Leveraging these mechanisms requires a minimum understanding of how the
593 deploy process works, at least for manual ``STATE`` runs.
598 OPNFV Fuel's ``deploy.sh`` script offers a dedicated argument for this, ``-f``,
599 which will skip executing the first ``N`` ``STATE`` files, where ``N`` is the
600 number of ``-f`` occurrences in the argument list.
604 The list of ``STATE`` files to be executed for a specific environment
605 depends on the OPNFV scenario chosen, deployment type (``virtual``,
606 ``baremetal`` or ``hybrid``) and the presence/absence of a ``VCP``
607 (virtualized control plane).
609 e.g.: Let's consider a ``baremetal`` enviroment, with ``VCP`` and a simple
610 scenario ``os-nosdn-nofeature-ha``, where ``deploy.sh`` failed executing the
611 ``openstack_ha`` ``STATE`` file.
613 The simplest redeploy approach (which usually works for **any** combination of
614 deployment type/VCP/scenario) is to issue the same deploy command as the
615 original attempt used, then adding a single ``-f``:
617 .. code-block:: console
619 jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
620 -s <scenario> [...] \
621 -f # skips running the virtual_init STATE file
623 All ``STATE`` files are re-entrant, so the above is equivalent (but a little
624 slower) to skipping all ``STATE`` files before the ``openstack_ha`` one, like:
626 .. code-block:: console
628 jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
629 -s <scenario> [...] \
630 -ffff # skips virtual_init, maas, baremetal_init, virtual_control_plane
634 For fine tuning the infrastructure setup steps executed during deployment,
635 see also the ``-e`` and ``-P`` deploy arguments.
639 On rare occassions, the cluster cannot idempotently be redeployed (e.g.
640 broken MySQL/Galera cluster), in which case some cleanup is due before
641 (re)running the ``STATE`` files. See ``-E`` deploy arg, which allows
642 either forcing a ``MaaS`` node deletion, then redeployment of all
643 baremetal nodes, if used twice (``-EE``); or only erasing the ``VCP`` VMs
644 if used only once (``-E``).
649 Instead of leveraging the full ``deploy.sh``, one could execute the ``STATE``
650 files one by one (or partially) from the ``cfg01``.
652 However, this requires a better understanding of how the list of ``STATE``
653 files to be executed is constructed for a specific scenario, depending on the
654 deployment type and the cluster having baremetal nodes, implemented in:
656 * ``mcp/config/scenario/defaults.yaml.j2``
657 * ``mcp/config/scenario/<scenario-name>.yaml``
659 e.g.: For the example presented above (baremetal with ``VCP``,
660 ``os-nosdn-nofeature-ha``), the list of ``STATE`` files would be:
665 * ``virtual_control_plane``
669 To execute one (or more) of the remaining ``STATE`` files after a failure:
671 .. code-block:: console
673 jenkins@jumpserver:~$ docker exec -it fuel bash
674 root@cfg01:~$ cd ~/fuel/mcp/config/states
675 root@cfg01:~/fuel/mcp/config/states$ ./openstack_ha
676 root@cfg01:~/fuel/mcp/config/states$ CI_DEBUG=true ./networks
678 For even finer granularity, one can also run the commands in a ``STATE`` file
679 one by one manually, e.g. if the execution failed applying the ``rabbitmq``
682 .. code-block:: console
684 root@cfg01:~$ salt -I 'rabbitmq:server' state.sls rabbitmq
686 Exploring the Cloud with Salt
687 =============================
689 To gather information about the cloud, the salt commands can be used.
690 It is based around a master-minion idea where the salt-master pushes config to
691 the minions to execute actions.
693 For example tell salt to execute a ping to ``8.8.8.8`` on all the nodes.
695 .. code-block:: console
697 root@cfg01:~$ salt "*" network.ping 8.8.8.8
699 ^^^^^^^^^^^^ function to execute
700 ^^^^^^^ argument passed to the function
704 Complex filters can be done to the target like compound queries or node roles.
706 For more information about Salt see the :ref:`fuel_userguide_references`
709 Some examples are listed below. Note that these commands are issued from Salt
710 master as ``root`` user.
712 View the IPs of All the Components
713 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
715 .. code-block:: console
717 root@cfg01:~$ salt "*" network.ip_addrs
718 cfg01.mcp-odl-ha.local:
721 mas01.mcp-odl-ha.local:
725 .........................
727 View the Interfaces of All the Components and Put the Output in a ``yaml`` File
728 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
730 .. code-block:: console
732 root@cfg01:~$ salt "*" network.interfaces --out yaml --output-file interfaces.yaml
733 root@cfg01:~# cat interfaces.yaml
734 cfg01.mcp-odl-ha.local:
736 hwaddr: 52:54:00:72:77:12
739 broadcast: 10.20.0.255
741 netmask: 255.255.255.0
743 - address: fe80::5054:ff:fe72:7712
747 .........................
749 View Installed Packages on MaaS Node
750 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
752 .. code-block:: console
754 root@cfg01:~# salt "mas*" pkg.list_pkgs
755 mas01.mcp-odl-ha.local:
767 .........................
769 Execute Any Linux Command on All Nodes (e.g. ``ls /var/log``)
770 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
772 .. code-block:: console
774 root@cfg01:~# salt "*" cmd.run 'ls /var/log'
775 cfg01.mcp-odl-ha.local:
781 cloud-init-output.log
783 .........................
785 Execute Any Linux Command on Nodes Using Compound Queries Filter
786 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
788 .. code-block:: console
790 root@cfg01:~# salt -C '* and cfg01*' cmd.run 'ls /var/log'
791 cfg01.mcp-odl-ha.local:
797 cloud-init-output.log
799 .........................
801 Execute Any Linux Command on Nodes Using Role Filter
802 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
804 .. code-block:: console
806 root@cfg01:~# salt -I 'nova:compute' cmd.run 'ls /var/log'
807 cmp001.mcp-odl-ha.local:
815 cloud-init-output.log
817 .........................
822 Once the deployment is complete, Openstack CLI is accessible from controller
823 VMs (``ctl01`` ... ``ctl03``).
825 Openstack credentials are at ``/root/keystonercv3``.
827 .. code-block:: console
829 root@ctl01:~# source keystonercv3
830 root@ctl01:~# openstack image list
831 +--------------------------------------+-----------------------------------------------+--------+
832 | ID | Name | Status |
833 +======================================+===============================================+========+
834 | 152930bf-5fd5-49c2-b3a1-cae14973f35f | CirrosImage | active |
835 | 7b99a779-78e4-45f3-9905-64ae453e3dcb | Ubuntu16.04 | active |
836 +--------------------------------------+-----------------------------------------------+--------+
838 The OpenStack Dashboard, Horizon, is available at ``http://<proxy public VIP>``.
839 The administrator credentials are ``admin``/``opnfv_secret``.
841 .. figure:: img/horizon_login.png
845 A full list of IPs/services is available at ``<proxy public VIP>:8090`` for
846 ``baremetal`` deploys.
848 .. figure:: img/salt_services_ip.png
852 Guest Operating System Support
853 ==============================
855 There are a number of possibilities regarding the guest operating systems
856 which can be spawned on the nodes.
857 The current system spawns virtual machines for VCP VMs on the KVM nodes and VMs
858 requested by users in OpenStack compute nodes. Currently the system supports
859 the following ``UEFI``-images for the guests:
861 +------------------+-------------------+--------------------+
862 | OS name | ``x86_64`` status | ``aarch64`` status |
863 +==================+===================+====================+
864 | Ubuntu 17.10 | untested | Full support |
865 +------------------+-------------------+--------------------+
866 | Ubuntu 16.04 | Full support | Full support |
867 +------------------+-------------------+--------------------+
868 | Ubuntu 14.04 | untested | Full support |
869 +------------------+-------------------+--------------------+
870 | Fedora atomic 27 | untested | Full support |
871 +------------------+-------------------+--------------------+
872 | Fedora cloud 27 | untested | Full support |
873 +------------------+-------------------+--------------------+
874 | Debian | untested | Full support |
875 +------------------+-------------------+--------------------+
876 | Centos 7 | untested | Not supported |
877 +------------------+-------------------+--------------------+
878 | Cirros 0.3.5 | Full support | Full support |
879 +------------------+-------------------+--------------------+
880 | Cirros 0.4.0 | Full support | Full support |
881 +------------------+-------------------+--------------------+
883 The above table covers only ``UEFI`` images and implies ``OVMF``/``AAVMF``
884 firmware on the host. An ``x86_64`` deployment also supports ``non-UEFI``
885 images, however that choice is up to the underlying hardware and the
886 administrator to make.
888 The images for the above operating systems can be found in their respective
894 OpenStack Cinder is the project behind block storage in OpenStack and OPNFV
895 Fuel supports LVM out of the box.
897 By default ``x86_64`` supports 2 additional block storage devices, while
898 ``aarch64`` supports only one.
900 More devices can be supported if the OS-image created has additional
901 properties allowing block storage devices to be spawned as ``SCSI`` drives.
902 To do this, add the properties below to the server:
904 .. code-block:: console
906 root@ctl01:~$ openstack image set --property hw_disk_bus='scsi' \
907 --property hw_scsi_model='virtio-scsi' \
910 The choice regarding which bus to use for the storage drives is an important
911 one. ``virtio-blk`` is the default choice for OPNFV Fuel, which attaches the
912 drives in ``/dev/vdX``. However, since we want to be able to attach a
913 larger number of volumes to the virtual machines, we recommend the switch to
914 ``SCSI`` drives which are attached in ``/dev/sdX`` instead.
916 ``virtio-scsi`` is a little worse in terms of performance but the ability to
917 add a larger number of drives combined with added features like ZFS, Ceph et
918 al, leads us to suggest the use of ``virtio-scsi`` in OPNFV Fuel for both
921 More details regarding the differences and performance of ``virtio-blk`` vs
922 ``virtio-scsi`` are beyond the scope of this manual but can be easily found
923 in other sources online like `VirtIO SCSI`_ or `VirtIO performance`_.
925 Additional configuration for configuring images in OpenStack can be found in
926 the OpenStack Glance documentation.
931 For each OpenStack service three endpoints are created: ``admin``, ``internal``
934 .. code-block:: console
936 ubuntu@ctl01:~$ openstack endpoint list --service keystone
937 +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
938 | ID | Region | Service Name | Service Type | Enabled | Interface | URL |
939 +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
940 | 008fec57922b4e9e8bf02c770039ae77 | RegionOne | keystone | identity | True | internal | http://172.16.10.26:5000/v3 |
941 | 1a1f3c3340484bda9ef7e193f50599e6 | RegionOne | keystone | identity | True | admin | http://172.16.10.26:35357/v3 |
942 | b0a47d42d0b6491b995d7e6230395de8 | RegionOne | keystone | identity | True | public | https://10.0.15.2:5000/v3 |
943 +----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
945 MCP sets up all Openstack services to talk to each other over unencrypted
946 connections on the internal management network. All admin/internal endpoints
947 use plain http, while the public endpoints are https connections terminated
948 via nginx at the ``VCP`` proxy VMs.
950 To access the public endpoints an SSL certificate has to be provided. For
951 convenience, the installation script will copy the required certificate
952 to the ``cfg01`` node at ``/etc/ssl/certs/os_cacert``.
954 Copy the certificate from the ``cfg01`` node to the client that will access
955 the https endpoints and place it under ``/etc/ssl/certs/``.
956 The SSL connection will be established automatically after.
958 .. code-block:: console
960 jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no -i /var/lib/opnfv/mcp.rsa -l ubuntu 10.20.0.2 \
961 "cat /etc/ssl/certs/os_cacert" | sudo tee /etc/ssl/certs/os_cacert
963 Reclass Model Viewer Tutorial
964 =============================
966 In order to get a better understanding of the ``reclass`` model Fuel uses, the
967 `reclass-doc`_ tool can be used to visualise the ``reclass`` model.
969 To avoid installing packages on the ``jumpserver`` or another host, the
970 ``cfg01`` Docker container can be used. Since the ``fuel`` git repository
971 located on the ``jumpserver`` is already mounted inside ``cfg01`` container,
972 the results can be visualized using a web browser on the ``jumpserver`` at the
973 end of the procedure.
975 .. code-block:: console
977 jenkins@jumpserver:~$ docker exec -it fuel bash
978 root@cfg01:~$ apt-get update
979 root@cfg01:~$ apt-get install -y npm nodejs
980 root@cfg01:~$ npm install -g reclass-doc
981 root@cfg01:~$ ln -s /usr/bin/nodejs /usr/bin/node
982 root@cfg01:~$ reclass-doc --output ~/fuel/mcp/reclass/modeler \
985 The generated documentation should be available on the ``jumpserver`` inside
986 ``fuel`` git repo subpath ``mcp/reclass/modeler/index.html``.
988 .. figure:: img/reclass_doc.png
992 .. _fuel_userguide_references:
997 #. :ref:`OPNFV Fuel Installation Instruction <fuel-installation>`
998 #. `Saltstack Documentation`_
999 #. `Saltstack Formulas`_
1000 #. `VirtIO performance`_
1003 .. _`Saltstack Documentation`: https://docs.saltstack.com/en/latest/topics/
1004 .. _`Saltstack Formulas`: https://salt-formulas.readthedocs.io/en/latest/
1005 .. _`VirtIO performance`: https://mpolednik.github.io/2017/01/23/virtio-blk-vs-virtio-scsi/
1006 .. _`VirtIO SCSI`: https://www.ovirt.org/develop/release-management/features/storage/virtio-scsi/
1007 .. _`reclass-doc`: https://github.com/jirihybek/reclass-doc