nfvbenchvm: fix broken build after dib update

Attempting to build nfvbench VM images with diskimage_builder (dib)
3.16.0 leads to the following error:

diskimage_builder.element_dependencies.MissingElementException: Element
'block-device' not found

Something changed at some point of diskimage_builder history, and the
updated doc states: "When using the vm element, an element that provides
block-device should be included".

To fix this and prevent future similar issues, this patch:
- adds the "block-device-mbr" element to the dependency list
- sets diskimage_builder version

By the way:
- replace the deprecated "centos7" element with the more generic
  "centos" element.
- add missing dependencies in the pre-requisites section of nfvbenchvm
  doc
- install pip with package manager (the current latest version of
  get-pip.py is not compatible with python 3.6, so let's use the
  occasion to stop getting pip directly from the Internet)
- use Python 3 in build-image.sh

Change-Id: I3198a1042eca04224b2a62db443c39a76903cf22
Signed-off-by: Gwenael Lambrouin <gwenael.lambrouin@orange.com>

diff --git a/nfvbenchvm/README.rst b/nfvbenchvm/README.rst
index dad9f46..29ec91f 100644 (file)
--- a/nfvbenchvm/README.rst
+++ b/nfvbenchvm/README.rst
@@ -13,10 +13,23 @@ Pre-requisites
 --------------
 - must run on Linux
 - the following packages must be installed prior to using this script:
+    - python3 (+ python3-venv on Ubuntu)
+    - python3-pip
     - git
-    - qemu-utils
+    - qemu-img (CentOs) or qemu-utils (Ubuntu)
     - kpartx
 
+.. note:: the image build process is based on `diskimage-builder
+          <https://docs.openstack.org/diskimage-builder/latest/index.html>`_
+          that will be installed in a Python virtual environment by nfvbenchvm
+          build script build-image.sh.
+
+.. note:: build-image.sh uses the `gsutil <https://pypi.org/project/gsutil/>`_
+          tool to interact with Google cloud storage (to check if the images
+          exist and to upload the images).  This is normally only needed in the
+          context of OPNFV build infrastructure, and build-image.sh can be used
+          without that tool in development environments.
+
 Build the image
 ---------------
 - cd dib

diff --git a/nfvbenchvm/dib/build-image.sh b/nfvbenchvm/dib/build-image.sh
index 6339f01..4559bb6 100755 (executable)
--- a/nfvbenchvm/dib/build-image.sh
+++ b/nfvbenchvm/dib/build-image.sh
@@ -3,7 +3,8 @@
 # A shell script to build the VPP VM image or NFVbench+TRex VM image using diskinage-builder
 #
 # The following packages must be installed prior to using this script:
-# sudo apt-get -y install python-virtualenv qemu-utils kpartx
+# Ubuntu: sudo apt-get -y install python3 python3-venv qemu-utils kpartx
+# CentOS: sudo yum install -y python3 qemu-img kpartx
 
 usage() {
     echo "Usage: $0 [-l] [-g] [-v]"
@@ -77,9 +78,9 @@ function build_image {
         if [ -d dib-venv ]; then
            . dib-venv/bin/activate
         else
-           virtualenv dib-venv
+           python3 -m venv dib-venv
            . dib-venv/bin/activate
-           pip install diskimage-builder
+           pip install diskimage-builder==3.16.0
         fi
 
         # Add nfvbenchvm_centos elements directory to the DIB elements path
@@ -110,8 +111,11 @@ function build_image {
            export TREX_VER=$(awk '/ENV TREX_VER/ {print $3}' ../../docker/Dockerfile | sed 's/"//g' | sed 's/\r//g')
         fi
 
+        # Specify CentOS version
+        export DIB_RELEASE=7
+
         echo "Building $1.qcow2..."
-        time disk-image-create -o $1 centos7 nfvbenchvm
+        time disk-image-create -o $1 centos nfvbenchvm
     fi
 
     ls -l $1.qcow2
@@ -146,4 +150,4 @@ else
        echo "Build generator image"
        build_image $generator_image_name
     fi
-fi
\ No newline at end of file
+fi

diff --git a/nfvbenchvm/dib/elements/nfvbenchvm/element-deps b/nfvbenchvm/dib/elements/nfvbenchvm/element-deps
index c6be0aa..7fd8e5b 100644 (file)
--- a/nfvbenchvm/dib/elements/nfvbenchvm/element-deps
+++ b/nfvbenchvm/dib/elements/nfvbenchvm/element-deps
@@ -1,4 +1,5 @@
 vm
+block-device-mbr
 cloud-init-datasources
 install-static
 package-installs

diff --git a/nfvbenchvm/dib/elements/nfvbenchvm/package-installs.yaml b/nfvbenchvm/dib/elements/nfvbenchvm/package-installs.yaml
index 60efec1..6f86027 100644 (file)
--- a/nfvbenchvm/dib/elements/nfvbenchvm/package-installs.yaml
+++ b/nfvbenchvm/dib/elements/nfvbenchvm/package-installs.yaml
@@ -17,4 +17,5 @@ openssh-server:
 dpdk-tools:
 git:
 python3-dev:
-libpython3.6-dev:
\ No newline at end of file
+libpython3.6-dev:
+python3-pip:

diff --git a/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/02-pip-package b/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/02-pip-package
index 9707841..79dea6c 100755 (executable)
--- a/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/02-pip-package
+++ b/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/02-pip-package
@@ -1,8 +1,5 @@
 #!/bin/bash
 
-cd /tmp
-wget https://bootstrap.pypa.io/get-pip.py
-python3 get-pip.py
-
-pip3 install setuptools wheel pbr
-pip3 install pyyaml
+python3 -m pip install --upgrade pip
+python3 -m pip install setuptools wheel pbr
+python3 -m pip install pyyaml

diff --git a/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/52-nfvbench-script b/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/52-nfvbench-script
index 9162952..5cc52e3 100755 (executable)
--- a/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/52-nfvbench-script
+++ b/nfvbenchvm/dib/elements/nfvbenchvm/post-install.d/52-nfvbench-script
@@ -19,7 +19,7 @@ rm -rf /opt/trex/$TREX_VER/automation/trex_control_plane/interactive/trex
 cd /opt
 git clone https://gerrit.opnfv.org/gerrit/nfvbench
 cd nfvbench/
-pip3 install . --use-deprecated=legacy-resolver
+python3 -m pip install . --use-deprecated=legacy-resolver
 cp xtesting/testcases.yaml /usr/local/lib/python3.6/site-packages/xtesting/ci/testcases.yaml
 python3 ./docker/cleanup_generators.py
 rm -rf /opt/nfvbench/.git