patches/opnfv-fuel/multiarch-mirrors/0000-f_repobuild-Use-packetary-to-build-partial-mirror.patch

   1 From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
   2 Date: Wed, 6 Jul 2016 17:03:49 +0200
   3 Subject: [PATCH] f_repobuild: Use packetary to build partial mirror
   4
   5 Packetary [1] will replace fuel-mirror in upstream Fuel.
   6 fuel-mirror is using packetary under the hood already, so the
   7 transition should not be too hard.
   8
   9 Adapt OPNFV Ubuntu partial mirror build:
  10 - obsolete opnfv-config.yaml (packetary uses CLI args for this);
  11 - new opnfv_config.yaml will hold old ubuntu.yaml data, split at
  12   build runtime into:
  13   * mos_<arch>_mirror.yaml - MOS mirror cfg, only for deps analysis;
  14   * ubuntu_<arch>_mirror.yaml - Ubuntu mirror cfg, for pkgs download;
  15   * ubuntu_<arch>_packages.yaml - Ubuntu packages to download;
  16
  17 While the consumers of our mirror still use `debootstrap`, which only
  18 supports the "main" repo component, make sure the "main" package set
  19 is complete, in the scope of:
  20   * ubuntu_<arch>_mirror_main.yaml - Ubuntu [main only] mirror;
  21   * ubuntu_<arch>_packages_main.yaml - Ubuntu packages for [main];
  22
  23 New workflow (executed for EACH architecture):
  24 1. Collect bootstrap package deps from <fuel_bootstrap_cli.yaml>;
  25 2. Collect all fixture release pkgs from fuel-web's <openstack.yaml>;
  26 3. Add OPNFV extra pkg deps from "opnfv_config.yaml"
  27    (based on old fuel-mirror)
  28
  29    Currently, this is based on old fuel-mirror's
  30    <fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml>, holding a
  31    hardcoded list of package deps; plus eventual extra OPNFV packages,
  32    like additional kernels, bootloaders etc.
  33    Since we now inspect MOS repos for deps with packetary, this old
  34    list should be easy to throw away as soon as we confirm each item
  35    is directly required by another, already existing package in the
  36    mirror. For now, keep the old list as safety net for missing deps.
  37
  38    NOTE: Dependencies up to and including this group should also
  39    be satisfiable from Ubuntu "main" repo component only.
  40
  41 4. Append plugin dependencies to the package list;
  42
  43 5. Run `packetary unresolved` for MOS repos, gathering MOS deps that
  44    should be fetched from Ubuntu. The new packages will be appended
  45    to the predefined list from old "ubuntu.yaml";
  46
  47 6. Run `packetary clone` to download all required pkgs for the
  48    partial Ubuntu mirror.
  49
  50 7. IF repo component merging is disabled, run `packetary clone` only
  51    on Ubuntu[main] repo components to download any (still) missing
  52    dependencies for `debootstrap`, which only looks at this component,
  53    and not the whole mirror.
  54
  55 8. IF repo component merging is enabled, run first `dpkg-scanpackages`
  56    to filter out old duplicate versions of packages, then run
  57    `packetary create` to merge all repo comps into a single "main".
  58
  59 v1 -> v2:
  60 - multiarch support (activated via UBUNTU_ARCH envvar);
  61 - fixed debootstrap missing deps in "main" component of Ubuntu mirror;
  62 - factored out some hardcodes into variables/functions;
  63 - add .gitignore file;
  64 - move generated config files to "opnfv_config" subdir;
  65 - add arm64 Ubuntu repo URL (ports.ubuntu.com is separated from the
  66   main Ubuntu mirror, archive.ubuntu.com, which only holds x86 pkgs);
  67
  68 v3 -> v4:
  69 - introduce env var "MIRROR_UBUNTU_MERGE" to control local mirror
  70   repo-component merging into a single "main";
  71 - enable mirror repo-component merging by default, edit
  72   fuel_bootstrap_cli.yaml accordingly;
  73
  74 v4 -> v6:
  75 - edit dea_base to use only main mirror;
  76 - fix duplicated logs;
  77 - remove `-q` flag for `create`;
  78
  79 v6 -> v7:
  80 - rebase for Newton fuel_bootstrap_cli.yaml (s/trusty/xenial/);
  81 - include OPNFV plugin deps in local Ubuntu mirror;
  82 - add `vgabios` dependency, required for arm64 guest firmware;
  83 - remove all fuel-mirror references;
  84 - merge old fuel-mirror pkglist to OPNFV pkglist;
  85 - update packetary commit to include upstream bugfixes [2, 3];
  86
  87 v7 -> v11:
  88 - re-enable f_repobuild after Newton rebase;
  89 - fix leftover "trusty" ref to "xenial";
  90 - force f_repobuild rebuild to bypass cache (by editing Makefile);
  91 - extend f_repobuild .cacheid to include scripts and OPNFV config;
  92
  93 v11 -> v12:
  94 - fix install/install.sh old refs to "trusty";
  95 - fix out of date select_ubuntu_repo.s refs to "trusty";
  96
  97 v12 -> v13:
  98 - fix leftover mos9.0 / fuel-mirror references;
  99 - fix most issues reported by `pylint opnfv_mirror_ubuntu.py`;
 100
 101 v13 -> v14:
 102 - Sync fuel_bootstrap_cli.yaml with settings.yaml.sample from 10.0:
 103   [4] Stub pass for "Get root password for build image from settings"
 104   [5] Sync pkg list with "Switch to Ubuntu Xenial"
 105   [6] "Add puppet and daemonize to default packages list"
 106   [7] "Delete kernel debug related parameter for performance"
 107   [8] "Add to bootstrap image dependencies for network checker"
 108 - Remove obsolete Trusty kernel images from pkg lists;
 109 - Don't break when plugins set is empty;
 110 - Fix missing "vim" package, which was resolved using secondary
 111   provider packages like GUI-enabled vim flavors;
 112   The proposed fix is quite sensitive, also cutting the final
 113   mirror size in half (!), and allowing us to remove a previous
 114   hacky workaround (re-running packetary only for main).
 115 - Remove "main"-only packetary run (hacky), since the new fix for "vim"
 116   also takes care of packetary solving dependencies from "universe".
 117 - Add OPNFV blacklisting to workaround "upstart" package being installed,
 118   together with "systemd" (having them both installed confuses puppet);
 119
 120 NOTE: Without filtering out old package verisons using
 121 `dpkg-scanpackages`, bootstrap build cannot solve all deps.
 122
 123 FIXME: Packetary solves missing dependecies by also accepting
 124 different packages that provide the same package name (e.g. "ifupdown"
 125 is satisfied by "netscript" package).
 126
 127 FIXME: Repo component merging is sort of slow, since packetary insists
 128 on copying the source files to the destination dir even if they are
 129 pointing to the same repo.
 130
 131 FIXME: Packetary `create` uses a different directory scheme for the
 132 created mirror than the upstream Ubuntu, e.g.:
 133 [p] mirrors/ubuntu/pool/main/l/lvm2_2.02.98-6ubuntu2_amd64.deb
 134 [U] mirrors/ubuntu/pool/main/l/lvm2/lvm2_2.02.98-6ubuntu2_amd64.deb
 135
 136 This disencourages creating the "merged" mirror in the same location
 137 as the source partial mirror.
 138
 139 [1] https://wiki.openstack.org/wiki/Packetary
 140 [2] https://review.openstack.org/#/c/392936/
 141 [3] https://review.openstack.org/#/c/392937/
 142 [4] https://review.openstack.org/#/c/277353/
 143 [5] https://review.openstack.org/#/c/345516/
 144 [6] https://review.openstack.org/#/c/344909/
 145 [7] https://review.openstack.org/#/c/400165/
 146 [8] https://review.openstack.org/#/c/415912/
 147
 148 JIRA: FUEL-218
 149 JIRA: FUEL-223
 150 JIRA: ARMBAND-169
 151 JIRA: ARMBAND-185
 152
 153 Change-Id: If2ee86f348b7683c83bfaf686baba4f1b8f555f0
 154 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 155 ---
 156  build/config.mk                                    |   4 +
 157  build/f_isoroot/Makefile                           |   8 +-
 158  build/f_isoroot/f_repobuild/.gitignore             |   4 +
 159  ...s_tree-exact-match-trumps-other-providers.patch |  75 ++++++
 160  build/f_isoroot/f_repobuild/Makefile               |  45 ++--
 161  build/f_isoroot/f_repobuild/config.mk              |  23 ++
 162  .../f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml  |  41 ++-
 163  build/f_isoroot/f_repobuild/opnfv_config.yaml      | 166 ++++++++++++
 164  build/f_isoroot/f_repobuild/opnfv_mirror_conf.py   |  57 -----
 165  build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py | 279 +++++++++++++++++++++
 166  build/f_isoroot/f_repobuild/select_ubuntu_repo.sh  |  41 +--
 167  build/install/apt-ftparchive-udeb.conf             |   4 +-
 168  build/install/install.sh                           |  14 +-
 169  build/install/uninstall.sh                         |  16 +-
 170  deploy/config/dea_base.yaml                        |  16 +-
 171  15 files changed, 636 insertions(+), 157 deletions(-)
 172  create mode 100644 build/f_isoroot/f_repobuild/.gitignore
 173  create mode 100644 build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch
 174  create mode 100644 build/f_isoroot/f_repobuild/config.mk
 175  create mode 100644 build/f_isoroot/f_repobuild/opnfv_config.yaml
 176  delete mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
 177  create mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
 178
 179 diff --git a/build/config.mk b/build/config.mk
 180 index 56317cb..5448c5e 100644
 181 --- a/build/config.mk
 182 +++ b/build/config.mk
 183 @@ -17,6 +17,10 @@ FUEL_MAIN_TAG = master
 184  MOS_VERSION   = 10.0
 185  OPENSTACK_VERSION = newton-10.0
 186
 187 +# List of space-separated Ubuntu architectures supported with current build
 188 +# Format: same as `dpkg-architecture -qDEB_HOST_ARCH`
 189 +# NOTE: Currently only amd64 is supported by Fuel@OPNFV. Armband adds arm64.
 190 +export UBUNTU_ARCH ?= amd64
 191
 192  ##############################################################################
 193  # Fuel components pinning / remote tracking; use submodules from f_repos
 194 diff --git a/build/f_isoroot/Makefile b/build/f_isoroot/Makefile
 195 index e189e16..b4ef1d0 100644
 196 --- a/build/f_isoroot/Makefile
 197 +++ b/build/f_isoroot/Makefile
 198 @@ -17,16 +17,12 @@ NON_10-0_REBASED_PLUGINS = f_odlpluginbuild f_onosfwpluginbuild f_ovsnfv-dpdk-pl
 199
 200  # If the BUILD_FUEL_PLUGINS environment variable is set, only build the plugins
 201  # indicated therein.
 202 -# Temporarilu disabling f_repobuild that currently have some issues on
 203 -# Fuel Newton.
 204  ifdef BUILD_FUEL_PLUGINS
 205  export BUILD_FUEL_PLUGINS
 206 -#SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(BUILD_FUEL_PLUGINS)
 207 -SUBDIRS = f_kscfg f_bootstrap f_isolinux $(BUILD_FUEL_PLUGINS)
 208 +SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(BUILD_FUEL_PLUGINS)
 209  $(warning Overriding plugin build selection to $(BUILD_FUEL_PLUGINS))
 210  else
 211 -#SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(PLUGINS)
 212 -SUBDIRS = f_kscfg f_bootstrap f_isolinux $(PLUGINS)
 213 +SUBDIRS = f_kscfg f_bootstrap f_isolinux f_repobuild $(PLUGINS)
 214  endif
 215  SUBCLEAN = $(addsuffix .clean,$(SUBDIRS))
 216
 217 diff --git a/build/f_isoroot/f_repobuild/.gitignore b/build/f_isoroot/f_repobuild/.gitignore
 218 new file mode 100644
 219 index 0000000..09baca8
 220 --- /dev/null
 221 +++ b/build/f_isoroot/f_repobuild/.gitignore
 222 @@ -0,0 +1,4 @@
 223 +packetary
 224 +fuel-web
 225 +nailgun
 226 +opnfv_config
 227 diff --git a/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch b/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch
 228 new file mode 100644
 229 index 0000000..1df292c
 230 --- /dev/null
 231 +++ b/build/f_isoroot/f_repobuild/0001-packages_tree-exact-match-trumps-other-providers.patch
 232 @@ -0,0 +1,75 @@
 233 +From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 234 +Date: Sun, 15 Jan 2017 00:40:15 +0100
 235 +Subject: [PATCH] packages_tree: exact match trumps other providers
 236 +
 237 +Ubuntu frequently has multiple providers for the same package name,
 238 +e.g. "vim" is provided by "vim", "vim-gnome" etc.
 239 +
 240 +Below is a sample dump of packetary's list of matches for a generic
 241 +"vim" package requirement, without a version enforced, based on
 242 +current Xenial repos:
 243 +
 244 +vim-gtk3-py2 (2:7.4.1689-3ubuntu1)
 245 +vim-gnome (2:7.4.1689-3ubuntu1)
 246 +vim-gtk (2:7.4.1689-3ubuntu1)
 247 +vim-nox (2:7.4.1689-3ubuntu1)
 248 +vim-gtk-py2 (2:7.4.1689-3ubuntu1)
 249 +vim-gnome-py2 (2:7.4.1689-3ubuntu1)
 250 +vim-nox-py2 (2:7.4.1689-3ubuntu1)
 251 +vim (2:7.4.1689-3ubuntu1)
 252 +vim-gtk3 (2:7.4.1689-3ubuntu1)
 253 +vim-athena (2:7.4.1689-3ubuntu1)
 254 +vim-athena-py2 (2:7.4.1689-3ubuntu1)
 255 +vim-gtk (2:7.4.1689-3ubuntu1.2)
 256 +vim-gnome-py2 (2:7.4.1689-3ubuntu1.2)
 257 +vim-gtk-py2 (2:7.4.1689-3ubuntu1.2)
 258 +vim-nox (2:7.4.1689-3ubuntu1.2)
 259 +vim-nox-py2 (2:7.4.1689-3ubuntu1.2)
 260 +vim-gtk3 (2:7.4.1689-3ubuntu1.2)
 261 +vim (2:7.4.1689-3ubuntu1.2)
 262 +vim-athena (2:7.4.1689-3ubuntu1.2)
 263 +vim-athena-py2 (2:7.4.1689-3ubuntu1.2)
 264 +vim-gnome (2:7.4.1689-3ubuntu1.2)
 265 +vim-gtk3-py2 (2:7.4.1689-3ubuntu1.2)
 266 +
 267 +Currently, packetary solves the "vim" relation by using the last item
 268 +in the sorted list, in this case "vim-gtk3-py2"; instead of using the
 269 +exact package name match "vim".
 270 +
 271 +This leads to our final mirror clone missing the "vim" package, and
 272 +inherintely failing to build bootstrap/target images using only the
 273 +partial Ubuntu mirror in OPNFV ISO.
 274 +
 275 +The proposed fix is to first check the list for an exact name match,
 276 +which would trump any secondary providers, even if they have a
 277 +higher package version.
 278 +
 279 +Change-Id: I7279aa6526ff9133829be2e316932c9b052c7814
 280 +Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
 281 +---
 282 + packetary/objects/packages_tree.py | 5 +++++
 283 + 1 file changed, 5 insertions(+)
 284 +
 285 +diff --git a/packetary/objects/packages_tree.py b/packetary/objects/packages_tree.py
 286 +index f02df59..6081fa9 100644
 287 +--- a/packetary/objects/packages_tree.py
 288 ++++ b/packetary/objects/packages_tree.py
 289 +@@ -18,6 +18,7 @@
 290 +
 291 + from collections import defaultdict
 292 +
 293 ++import re
 294 + import six
 295 +
 296 + from packetary.objects.index import Index
 297 +@@ -57,6 +58,10 @@ class PackagesTree(object):
 298 +         """
 299 +         candidates = self.find_all(name, version_range)
 300 +         if len(candidates) > 0:
 301 ++            # exact pkg name match trumps other providers' versions
 302 ++            for candidate in reversed(candidates):
 303 ++                if re.match('{0}\s.*'.format(re.escape(name)), str(candidate)):
 304 ++                    return candidate
 305 +             # we return candidates in sorted order, so let's take the highest
 306 +             return candidates[-1]
 307 +         return None
 308 diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile
 309 index 8beb882..76aadf9 100644
 310 --- a/build/f_isoroot/f_repobuild/Makefile
 311 +++ b/build/f_isoroot/f_repobuild/Makefile
 312 @@ -1,7 +1,8 @@
 313  ##############################################################################
 314 -# Copyright (c) 2015 Ericsson AB and others.
 315 +# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others.
 316  # stefan.k.berg@ericsson.com
 317  # jonas.bjurel@ericsson.com
 318 +# Alexandru.Avadanii@enea.com
 319  # All rights reserved. This program and the accompanying materials
 320  # are made available under the terms of the Apache License, Version 2.0
 321  # which accompanies this distribution, and is available at
 322 @@ -10,41 +11,39 @@
 323
 324  SHELL := /bin/bash
 325  TOP := $(shell pwd)
 326 -TMP_ROOT_DIR := $(shell echo "$(MIRROR_UBUNTU_ROOT)" | cut -d "/" -f2)
 327
 328  include ../../config.mk
 329 +include config.mk
 330
 331  export MOS_VERSION
 332 -export OPENSTACK_VERSION
 333 +export MIRROR_UBUNTU_OPNFV_PATH:=$(TOP)/nailgun/mirrors/ubuntu
 334
 335  .PHONY: all
 336  all: nailgun
 337
 338  nailgun:
 339 -       sudo apt-get install -y git libxml2-dev libxslt-dev python-dev  python-pip libz-dev libyaml-dev createrepo python-yaml
 340 -       rm -Rf nailgun
 341 -       sudo mkdir -p /var/www/nailgun
 342 -       ln -sf ${F_SUBMOD_DIR}/fuel-mirror fuel-mirror
 343 -       sudo pip install -U -r ./fuel-mirror/requirements.txt
 344 -       sudo pip install ./fuel-mirror
 345 -       sudo pip install ./fuel-mirror/contrib/fuel_mirror
 346 -       ./opnfv_mirror_conf.py
 347 -       sudo fuel-mirror --debug --config ./opnfv-config.yaml create --group ubuntu --pattern=ubuntu
 348 -       sudo chmod -R 755 /var/www/nailgun
 349 -       cp -Rp /var/www/nailgun .
 350 -       # In the end we want to have ubuntu repository in mirrors/ubuntu directory
 351 -       -if [ "$(MIRROR_UBUNTU_ROOT)" != "/ubuntu/" ]; then \
 352 -         mkdir -p nailgun/mirrors/ubuntu;\
 353 -         mv nailgun/mirrors$(MIRROR_UBUNTU_ROOT)* nailgun/mirrors/ubuntu;\
 354 -         [ "$(MIRROR_UBUNTU_ROOT)" != "/" ] && rm -rf nailgun/mirrors/$(TMP_ROOT_DIR);\
 355 +       sudo apt-get install -y createrepo git libxml2-dev libxslt1-dev \
 356 +               python-dev zlib1g-dev
 357 +       rm -Rf nailgun packetary opnfv_config && mkdir opnfv_config
 358 +       # We will analyze fuel-web's fixture files for package lists
 359 +       ln -sf ${F_SUBMOD_DIR}/fuel-web fuel-web
 360 +       git clone --quiet $(PACKETARY_REPO)
 361 +       if [ -n $(PACKETARY_COMMIT) ]; then \
 362 +               git -C packetary checkout $(PACKETARY_COMMIT); \
 363         fi
 364 +       # FIXME(alav): Temporary, until we decide this is upstreamable
 365 +       git -C packetary am $(TOP)/*.patch
 366 +       sudo pip install -U -r ./packetary/requirements.txt
 367 +       sudo pip install -U ./packetary
 368 +       # Handle config and mirror build in one place
 369 +       ./opnfv_mirror_ubuntu.py
 370         # Store artifact in cache straight away if caching is enabled
 371         # (no .cacheid will be present unless this is a cached build)
 372         test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0
 373
 374  .PHONY: clean
 375  clean:
 376 -       @rm -rf ../release/opnfv/nailgun nailgun fuel-mirror opnfv-config.yaml ubuntu.yaml
 377 +       @rm -rf ../release/opnfv/nailgun nailgun packetary fuel-web opnfv_config
 378
 379  .PHONY: release
 380  release:nailgun
 381 @@ -59,10 +58,12 @@ release:nailgun
 382
 383  # Create a unique hash to be used for getting and putting cache, based on:
 384  #   - Year and week (causing the cache to be rebuilt weekly)
 385 -#   - The contents of this Makefile
 386 +#   - The contents of this Makefile + all sh,mk,py,yaml files in CWD
 387 +#   - repo packages fingerprint
 388 +#   - repo arch list
 389  .cacheid:
 390         date +"Repocache %G%V" > .cachedata
 391 -       sha1sum Makefile >> .cachedata
 392 +       sha1sum Makefile *.{sh,mk,py,yaml} >> .cachedata
 393         $(CACHETOOL) packages >> .cachedata
 394         echo -n $(UBUNTU_ARCH) | sha1sum | awk {'print $$1'} >> .cachedata
 395         cat .cachedata | $(CACHETOOL) getid > .cacheid
 396 diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk
 397 new file mode 100644
 398 index 0000000..9852093
 399 --- /dev/null
 400 +++ b/build/f_isoroot/f_repobuild/config.mk
 401 @@ -0,0 +1,23 @@
 402 +##############################################################################
 403 +# Copyright (c) 2016 Ericsson AB, Enea AB and others.
 404 +# stefan.k.berg@ericsson.com
 405 +# jonas.bjurel@ericsson.com
 406 +# Alexandru.Avadanii@enea.com
 407 +# All rights reserved. This program and the accompanying materials
 408 +# are made available under the terms of the Apache License, Version 2.0
 409 +# which accompanies this distribution, and is available at
 410 +# http://www.apache.org/licenses/LICENSE-2.0
 411 +##############################################################################
 412 +
 413 +# Use a recent master commit, since tags/branches are not yet mature
 414 +export PACKETARY_REPO?=https://github.com/openstack/packetary
 415 +export PACKETARY_COMMIT?=3021c001561b4baef352bf0b881d064ac687cc20
 416 +
 417 +# arm64 Ubuntu mirror is separated from archive.ubuntu.com
 418 +export MIRROR_UBUNTU_URL_arm64=http://ports.ubuntu.com/ubuntu-ports/
 419 +export MIRROR_UBUNTU_ROOT_arm64=ubuntu-ports
 420 +
 421 +# Merge all local mirror repo components/section into single "main"
 422 +# NOTE: When changing this, make sure to also update all consumer config, like:
 423 +# - fuel_bootstrap_cli.yaml
 424 +export MIRROR_UBUNTU_MERGE=true
 425 diff --git a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
 426 index fcf4257..81ca6eb 100644
 427 --- a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
 428 +++ b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
 429 @@ -8,19 +8,18 @@
 430  ##############################################################################
 431  ---
 432    root_ssh_authorized_file: /root/.ssh/id_rsa.pub
 433 -  extend_kopts: "biosdevname=0 net.ifnames=1 debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8"
 434 -  ubuntu_release: trusty
 435 +  extend_kopts: "biosdevname=0 net.ifnames=1"
 436 +  ubuntu_release: xenial
 437    extra_dirs:
 438 -    - /usr/share/fuel_bootstrap_cli/files/trusty
 439 +    - /usr/share/fuel_bootstrap_cli/files/xenial
 440    output_dir: /tmp/
 441 -  kernel_flavor: linux-image-generic-lts-trusty
 442 +  kernel_flavor: linux-image-generic-lts-xenial
 443    packages:
 444 +    - daemonize
 445      - fuel-agent
 446      - hwloc
 447 -    - hpsa-dkms
 448      - i40e-dkms
 449      - linux-firmware
 450 -    - linux-firmware-nonfree
 451      - linux-headers-generic
 452      - live-boot
 453      - live-boot-initramfs-tools
 454 @@ -33,13 +32,23 @@
 455      - nailgun-mcagents
 456      - network-checker
 457      - ntp
 458 +    - ntpdate
 459      - openssh-client
 460      - openssh-server
 461 +    - puppet
 462      - squashfs-tools
 463      - ubuntu-minimal
 464      - vim
 465      - wget
 466      - xz-utils
 467 +    # NOTE(el): Packages required for new generation
 468 +    # network checker to be run without an access
 469 +    # to repositories.
 470 +    - sysfsutils
 471 +    - bridge-utils
 472 +    - ifenslave
 473 +    - irqbalance
 474 +    - iputils-arping
 475    bootstrap_images_dir: /var/www/nailgun/bootstraps
 476    active_bootstrap_symlink: /var/www/nailgun/bootstraps/active_bootstrap
 477    flavor: ubuntu
 478 @@ -47,29 +56,19 @@
 479    https_proxy: ""
 480    repos:
 481      - name: ubuntu
 482 -      section: "main universe multiverse"
 483 +      section: "main"
 484        uri: "http://127.0.0.1:8080/mirrors/ubuntu"
 485        priority:
 486 -      suite: trusty
 487 -      type: deb
 488 -    - name: ubuntu-updates
 489 -      section: "main universe multiverse"
 490 -      uri: "http://127.0.0.1:8080/mirrors/ubuntu"
 491 -      priority:
 492 -      suite: trusty-updates
 493 -      type: deb
 494 -    - name: ubuntu-security
 495 -      section: "main universe multiverse"
 496 -      uri: "http://127.0.0.1:8080/mirrors/ubuntu"
 497 -      priority:
 498 -      suite: trusty-security
 499 +      suite: xenial
 500        type: deb
 501      - name: mos
 502        section: "main restricted"
 503        uri: "http://127.0.0.1:8080/ubuntu/x86_64"
 504        priority: 1050
 505 -      suite: mos9.0
 506 +      suite: mos10.0
 507        type: deb
 508    skip_default_img_build: false
 509    direct_repo_addresses:
 510      - "127.0.0.1"
 511 +  # User can provide default hashed root password for bootstrap image
 512 +  # hashed_root_password: "$6$IInX3Cqo$5xytL1VZbZTusOewFnG6couuF0Ia61yS3rbC6P5YbZP2TYclwHqMq9e3Tg8rvQxhxSlBXP1DZhdUamxdOBXK0."
 513 diff --git a/build/f_isoroot/f_repobuild/opnfv_config.yaml b/build/f_isoroot/f_repobuild/opnfv_config.yaml
 514 new file mode 100644
 515 index 0000000..b8d3441
 516 --- /dev/null
 517 +++ b/build/f_isoroot/f_repobuild/opnfv_config.yaml
 518 @@ -0,0 +1,166 @@
 519 +##############################################################################
 520 +# Copyright (c) 2016 Enea AB and others.
 521 +# Alexandru.Avadanii@enea.com
 522 +# All rights reserved. This program and the accompanying materials
 523 +# are made available under the terms of the Apache License, Version 2.0
 524 +# which accompanies this distribution, and is available at
 525 +# http://www.apache.org/licenses/LICENSE-2.0
 526 +##############################################################################
 527 +
 528 +# REPO definitions
 529 +
 530 +# Based on old fuel-mirror data [1], adapted for OPNFV and packetary.
 531 +# NOTE: 'uri' field will be added by opnfv_mirror_ubuntu.py from ENV.
 532 +# [1] https://github.com/openstack/fuel-mirror/blob/stable/mitaka/
 533 +#     contrib/fuel_mirror/data/ubuntu.yaml
 534 +
 535 +# Main is a required parameter which defines what repository will be used
 536 +# for images creation and that mirror should contain all packages for minimal
 537 +# system creation.
 538 +groups:
 539 +  ubuntu:
 540 +    - name: "ubuntu"
 541 +      main: true
 542 +      suite: "xenial"
 543 +      section:
 544 +        - "main"
 545 +        - "multiverse"
 546 +        - "restricted"
 547 +        - "universe"
 548 +      type: "deb"
 549 +      priority: null
 550 +
 551 +    - name: "ubuntu-updates"
 552 +      suite: "xenial-updates"
 553 +      section:
 554 +        - "main"
 555 +        - "multiverse"
 556 +        - "restricted"
 557 +        - "universe"
 558 +      type: "deb"
 559 +      priority: null
 560 +
 561 +    - name: "ubuntu-security"
 562 +      suite: "xenial-security"
 563 +      section:
 564 +        - "main"
 565 +        - "multiverse"
 566 +        - "restricted"
 567 +        - "universe"
 568 +      type: "deb"
 569 +      priority: null
 570 +
 571 +  mos:
 572 +    - name: "mos"
 573 +      suite: "mos$mos_version"
 574 +      section:
 575 +        - "main"
 576 +        - "restricted"
 577 +      type: "deb"
 578 +      priority: 1000
 579 +
 580 +    - name: "mos-updates"
 581 +      suite: "mos$mos_version-updates"
 582 +      section:
 583 +        - "main"
 584 +        - "restricted"
 585 +      type: "deb"
 586 +      priority: 1000
 587 +
 588 +    - name: "mos-security"
 589 +      suite: "mos$mos_version-security"
 590 +      section:
 591 +        - "main"
 592 +        - "restricted"
 593 +      type: "deb"
 594 +      priority: 1000
 595 +
 596 +    - name: "mos-holdback"
 597 +      suite: "mos$mos_version-holdback"
 598 +      section:
 599 +        - "main"
 600 +        - "restricted"
 601 +      type: "deb"
 602 +      priority: 1000
 603 +
 604 +# PACKAGES
 605 +
 606 +# This section lists packages that should be present in the local Ubuntu mirror,
 607 +# but are not direct dependencies of any other packages from  MOS or Ubuntu.
 608 +# e.g.: additional kernels, bootloaders etc.
 609 +
 610 +packages:
 611 +  - name: "vgabios"
 612 +# Packages are required to build bootstrap images for a system.
 613 +# The mirror should contiain such packages in addition to local mirror.
 614 +  - name: "acpi-support"
 615 +  - name: "anacron"
 616 +  - name: "aptitude"
 617 +  - name: "atop"
 618 +  - name: "acct"
 619 +  - name: "bash-completion"
 620 +  - name: "bc"
 621 +  - name: "build-essential"
 622 +  - name: "cloud-init"
 623 +  - name: "conntrackd"
 624 +  - name: "cpu-checker"
 625 +  - name: "cpufrequtils"
 626 +  - name: "debconf-utils"
 627 +  - name: "devscripts"
 628 +  - name: "fping"
 629 +  - name: "git"
 630 +  - name: "grub-pc"
 631 +  - name: "htop"
 632 +  - name: "hwloc"
 633 +  - name: "ifenslave"
 634 +  - name: "iperf"
 635 +  - name: "iptables-persistent"
 636 +  - name: "irqbalance"
 637 +  - name: "language-pack-en"
 638 +  - name: "libapache2-mod-fastcgi"
 639 +  - name: "libnss3-tools"
 640 +  - name: "linux-headers-generic-lts-xenial"
 641 +  - name: "linux-image-generic-lts-xenial"
 642 +  - name: "live-boot"
 643 +  - name: "livecd-rootfs"
 644 +  - name: "mc"
 645 +  - name: "memcached"
 646 +  - name: "monit"
 647 +  - name: "msmtp-mta"
 648 +  - name: "multipath-tools"
 649 +  - name: "multipath-tools-boot"
 650 +  - name: "nginx"
 651 +  - name: "ntp"
 652 +  - name: "openssh-server"
 653 +  - name: "percona-toolkit"
 654 +  - name: "percona-xtrabackup"
 655 +  - name: "pm-utils"
 656 +  - name: "puppet"
 657 +  - name: "python-lesscpy"
 658 +  - name: "python-pip"
 659 +  - name: "rsyslog-gnutls"
 660 +  - name: "rsyslog-relp"
 661 +  - name: "screen"
 662 +  - name: "squashfs-tools"
 663 +  - name: "swift-plugin-s3"
 664 +  - name: "sysfsutils"
 665 +  - name: "sysstat"
 666 +  - name: "telnet"
 667 +  - name: "tmux"
 668 +  - name: "traceroute"
 669 +  - name: "ubuntu-standard"
 670 +  - name: "vim"
 671 +  - name: "virt-what"
 672 +  - name: "xinetd"
 673 +  - name: "xmlstarlet"
 674 +  - name: "tftpd-hpa"
 675 +  - name: "syslinux"
 676 +
 677 +# OPNFV BLACKLIST
 678 +
 679 +# Packetary's dependency solving mechanism brings in more than one alternative
 680 +# For example, "Depends: upstart | systemd-sysv" brings both packages, which
 681 +# leads to debootstrap using "upstart" (1st option available in local repo).
 682 +
 683 +opnfv_blacklist:
 684 +  - name: "upstart"
 685 diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py b/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
 686 deleted file mode 100755
 687 index 1c7eb59..0000000
 688 --- a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
 689 +++ /dev/null
 690 @@ -1,57 +0,0 @@
 691 -#!/usr/bin/env python
 692 -##############################################################################
 693 -# Copyright (c) 2015 Ericsson AB and others.
 694 -# mskalski@mirantis.com
 695 -# All rights reserved. This program and the accompanying materials
 696 -# are made available under the terms of the Apache License, Version 2.0
 697 -# which accompanies this distribution, and is available at
 698 -# http://www.apache.org/licenses/LICENSE-2.0
 699 -##############################################################################
 700 -import os
 701 -import yaml
 702 -
 703 -current_snapshot =  os.environ["LATEST_TARGET_UBUNTU"]
 704 -mos_version = os.environ['MOS_VERSION']
 705 -openstack_version = os.environ['OPENSTACK_VERSION']
 706 -mos_ubuntu = os.environ['MIRROR_MOS_UBUNTU']
 707 -mos_ubuntu_root = os.environ['MIRROR_MOS_UBUNTU_ROOT']
 708 -mirror_ubuntu = os.environ['MIRROR_UBUNTU_URL']
 709 -if os.environ.get('BUILD_FUEL_PLUGINS'):
 710 -  plugins = os.environ['BUILD_FUEL_PLUGINS']
 711 -else:
 712 -  plugins = os.environ['PLUGINS']
 713 -
 714 -
 715 -configuration_file = open('fuel-mirror/contrib/fuel_mirror/etc/config.yaml').read()
 716 -conf = yaml.load(configuration_file)
 717 -conf['pattern_dir'] = '.'
 718 -conf['openstack_version'] = openstack_version
 719 -conf['mos_version'] = mos_version
 720 -
 721 -with open('opnfv-config.yaml', 'w') as outfile:
 722 -  outfile.write( yaml.dump(conf, default_flow_style=False) )
 723 -
 724 -pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read()
 725 -pattern = yaml.load(pattern_file)
 726 -pattern['mos_baseurl'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root)
 727 -pattern['ubuntu_baseurl'] = mirror_ubuntu
 728 -for group in pattern['groups']['mos']:
 729 -  group['uri'] = pattern['mos_baseurl']
 730 -for group in pattern['groups']['ubuntu']:
 731 -  group['uri'] = pattern['ubuntu_baseurl']
 732 -
 733 -for plugin in plugins.split():
 734 -  path = "../{}/packages.yaml".format(plugin)
 735 -  if os.path.isfile(path):
 736 -    f = open(path).read()
 737 -    plugin_yaml = yaml.load(f)
 738 -    plugin_set = set(plugin_yaml['packages'])
 739 -    main_set = set(pattern['packages'])
 740 -    new_packages = plugin_set - main_set
 741 -    print "Plugin {} require new packages: {}".format(plugin, ', '.join(new_packages))
 742 -    pattern['packages'] = pattern['packages'] + list(new_packages)
 743 -
 744 -pattern['requirements']['ubuntu'] = pattern['packages']
 745 -
 746 -with open('ubuntu.yaml', 'w') as outfile:
 747 -  outfile.write( yaml.safe_dump(pattern, default_flow_style=False) )
 748 diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
 749 new file mode 100755
 750 index 0000000..1a603ee
 751 --- /dev/null
 752 +++ b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
 753 @@ -0,0 +1,279 @@
 754 +#!/usr/bin/env python
 755 +##############################################################################
 756 +# Copyright (c) 2015,2016 Ericsson AB, Mirantis Inc., Enea AB and others.
 757 +# mskalski@mirantis.com
 758 +# Alexandru.Avadanii@enea.com
 759 +# All rights reserved. This program and the accompanying materials
 760 +# are made available under the terms of the Apache License, Version 2.0
 761 +# which accompanies this distribution, and is available at
 762 +# http://www.apache.org/licenses/LICENSE-2.0
 763 +##############################################################################
 764 +
 765 +"""Build multiarch partial local Ubuntu mirror using packetary"""
 766 +
 767 +##############################################################################
 768 +# Design quirks / workarounds:
 769 +# 1. Fuel-agent uses `debootstrap` to build bootstrap and target chroots from
 770 +#    the local mirror; which only uses the "main" component from the first
 771 +#    repository, i.e. does not include "updates"/"security".
 772 +#    In order to fullfill all debootstrap dependencies in "main" repo, we will
 773 +#    do an extra packetary run using a reduced scope:
 774 +#    - only "main" component of the first mirror;
 775 +#    - reduced package dependency list (without MOS/OPNFV plugin deps).
 776 +# 2. If repo structure is not mandatory to be in sync with official mirrors,
 777 +#    we can mitigate the issue by "merging" all repo-components into a single
 778 +#    "main".
 779 +##############################################################################
 780 +# Mirror build steps (for EACH architecture in UBUNTU_ARCH):
 781 +# 1. Collect bootstrap package deps from <fuel_bootstrap_cli.yaml>;
 782 +# 2. Collect all fixture release packages from fuel-web's <openstack.yaml>;
 783 +# 3. Parse new "opnfv_config.yaml" list of packages (from old fuel-mirror);
 784 +# 4. Inherit enviroment variable(s) for mirror URLs, paths etc.
 785 +#    - Allow arch-specific overrides for each env var;
 786 +# 5. Mirror config is defined based on common config + OPNFV overrides;
 787 +#    - Convert old configuration format to packetary style where needed;
 788 +# 6. Package lists are defined based on common config + OPNFV deps;
 789 +#    - Keep track of "main" packages separately, required by debootstrap;
 790 +# 7. Clone/update all mirror components;
 791 +# 8. IF mirror merging is disabled:
 792 +#    - Clone/update "main" mirror component (fix missing debootstrap deps);
 793 +# 9. IF mirror merging is enabled:
 794 +#    - Use `dpkg-scanpackages` to filter out old versions of duplicate pkgs;
 795 +#    - Run `packetary create` on the set of downloaded packages, merging
 796 +#      them on the fly into a single-component mirror;
 797 +##############################################################################
 798 +
 799 +from copy import deepcopy
 800 +import os
 801 +import shutil
 802 +import sys
 803 +import yaml
 804 +from contextlib import contextmanager
 805 +from cStringIO import StringIO
 806 +from packetary.cli.app import main
 807 +
 808 +@contextmanager
 809 +def capture_stdout(output):
 810 +    """Context manager for capturing stdout"""
 811 +    stdout = sys.stdout
 812 +    sys.stdout = output
 813 +    yield
 814 +    sys.stdout = stdout
 815 +
 816 +# FIXME: Find a better approach for eliminating duplicate logs than this
 817 +def force_logger_reload():
 818 +    """Force logger reload (ugly hack to prevent log duplication)"""
 819 +    for mod in sys.modules.keys():
 820 +        if mod.startswith('logging'):
 821 +            try:
 822 +                reload(sys.modules[mod])
 823 +            except:
 824 +                pass
 825 +
 826 +def get_unres_pkgs(architecture, cfg_mirror):
 827 +    """Determine missing package dependecies for a mirror defition"""
 828 +    unresolved_pkgs = list()
 829 +    packetary_output = StringIO()
 830 +    with capture_stdout(packetary_output):
 831 +        main('unresolved -a {0} -r {1} -c name version --sep ;'
 832 +            .format(_ARCH[architecture], cfg_mirror).split(' '))
 833 +    for dep_pkg in packetary_output.getvalue().splitlines():
 834 +        if dep_pkg.startswith('#'):
 835 +            continue
 836 +        dep = dep_pkg.split(';')
 837 +        unresolved_pkgs += [{'name': dep[0], 'version': dep[1]}]
 838 +    force_logger_reload()
 839 +    return unresolved_pkgs
 840 +
 841 +def from_legacy_pkglist(legacy_pkglist):
 842 +    """Package list conversion from `old fuel-mirror` to `packetary` style"""
 843 +    pkglist = list()
 844 +    for pkg in legacy_pkglist:
 845 +        pkglist += [{'name': pkg}]
 846 +    return pkglist
 847 +
 848 +def to_legacy_pkglist(pkglist):
 849 +    """Package list conversion from `packetary` style to `old fuel-mirror`"""
 850 +    legacy_pkglist = list()
 851 +    for pkg in pkglist:
 852 +        legacy_pkglist.append(pkg['name'])
 853 +    return legacy_pkglist
 854 +
 855 +def legacy_diff(base_pkglist, new_pkglist, requester, architecture):
 856 +    """Package list diff (old format)"""
 857 +    diff_set = set(new_pkglist)
 858 +    if base_pkglist:
 859 +        diff_set -= set(base_pkglist)
 860 +    if diff_set:
 861 +        print(' * {0} requires new packages for architecture [{1}]: {2}'
 862 +              .format(requester, architecture, ', '.join(diff_set)))
 863 +    return list(diff_set)
 864 +
 865 +def do_local_repo(architecture, cfg_repo, cfg_packages_paths):
 866 +    """Create single-component local repo (one architecture per call)"""
 867 +    # Packetary does not use a global config file, so pass old settings here.
 868 +    main('create -t deb -a {0} --repository {1} --package-files {2}'
 869 +         ' --ignore-errors-num 2 --retries-num 3 --threads-num 10'
 870 +         .format(_ARCH[architecture], cfg_repo, cfg_packages_paths).split(' '))
 871 +    force_logger_reload()
 872 +
 873 +def do_partial_mirror(architecture, cfg_mirror, cfg_packages):
 874 +    """Clone partial local mirror (one architecture per call)"""
 875 +    # Note: '-d .' is ignored, as each mirror defines its own path.
 876 +    main('clone -t deb -a {0} -r {1} -R {2} -d .'
 877 +         ' --ignore-errors-num 2 --retries-num 3 --threads-num 10'
 878 +         .format(_ARCH[architecture], cfg_mirror, cfg_packages).split(' '))
 879 +    force_logger_reload()
 880 +
 881 +def write_cfg_file(cfg_mirror, data):
 882 +    """Write configuration (yaml) file (package list / mirror defition)"""
 883 +    with open(cfg_mirror, 'w') as outfile:
 884 +        outfile.write(yaml.safe_dump(data, default_flow_style=False))
 885 +
 886 +def get_env(env_var, architecture=None):
 887 +    """Evaluate architecture-specific overrides of env vars"""
 888 +    if architecture:
 889 +        env_var_arch = '{0}_{1}'.format(env_var, architecture)
 890 +        if os.environ.get(env_var_arch):
 891 +            return os.environ[env_var_arch]
 892 +    if os.environ.get(env_var):
 893 +        return os.environ[env_var]
 894 +    return None
 895 +
 896 +# Architecture name mapping (dpkg:packetary) for packetary CLI invocation
 897 +_ARCH = {
 898 +    "i386": "i386",
 899 +    "amd64": "x86_64",
 900 +    "arm64": "aarch64",
 901 +}
 902 +
 903 +# Arch-indepedent configuration (old fuel-mirror + OPNFV extra packages)
 904 +CFG_D = 'opnfv_config'
 905 +CFG_OPNFV = 'opnfv_config.yaml'
 906 +MOS_VERSION = get_env('MOS_VERSION')
 907 +UBUNTU_ARCH = get_env('UBUNTU_ARCH')
 908 +MIRROR_UBUNTU_PATH = get_env('MIRROR_UBUNTU_OPNFV_PATH')
 909 +MIRROR_UBUNTU_TMP_PATH = '{0}.tmp'.format(MIRROR_UBUNTU_PATH)
 910 +MIRROR_UBUNTU_MERGE = get_env('MIRROR_UBUNTU_MERGE')
 911 +CFG_MM_UBUNTU = '{0}/ubuntu_mirror_local.yaml'.format(CFG_D)
 912 +FUEL_BOOTSTRAP_CLI_FILE = open('fuel_bootstrap_cli.yaml').read()
 913 +FUEL_BOOTSTRAP_CLI = yaml.load(FUEL_BOOTSTRAP_CLI_FILE)
 914 +FIXTURE_FILE = open('fuel-web/nailgun/nailgun/fixtures/openstack.yaml').read()
 915 +FIXTURE = yaml.load(FIXTURE_FILE)
 916 +OPNFV_CFG_YAML = open(CFG_OPNFV).read()
 917 +OPNFV_CFG = yaml.load(OPNFV_CFG_YAML)
 918 +
 919 +# Create local partial mirror using packetary, one arch at a time
 920 +for arch in UBUNTU_ARCH.split(' '):
 921 +    # Mirror / Package env vars, arch-overrideable
 922 +    mos_ubuntu = get_env('MIRROR_MOS_UBUNTU', arch)
 923 +    mos_ubuntu_root = get_env('MIRROR_MOS_UBUNTU_ROOT', arch)
 924 +    mirror_ubuntu = get_env('MIRROR_UBUNTU_URL', arch)
 925 +    plugins = get_env('BUILD_FUEL_PLUGINS', arch)
 926 +    if plugins is None:
 927 +        plugins = get_env('PLUGINS', arch)
 928 +
 929 +    # Mirror / Package list configuration files (arch-specific)
 930 +    cfg_m_mos = '{0}/mos_{1}_mirror.yaml'.format(CFG_D, arch)
 931 +    cfg_m_ubuntu = '{0}/ubuntu_{1}_mirror.yaml'.format(CFG_D, arch)
 932 +    cfg_p_ubuntu = '{0}/ubuntu_{1}_packages.yaml'.format(CFG_D, arch)
 933 +    cfg_m_ubuntu_main = '{0}/ubuntu_{1}_mirror_main.yaml'.format(CFG_D, arch)
 934 +    cfg_p_ubuntu_main = '{0}/ubuntu_{1}_packages_main.yaml'.format(CFG_D, arch)
 935 +
 936 +    # Mirror config fork before customizing (arch-specific)
 937 +    arch_mos = 'mos_{0}'.format(arch)
 938 +    arch_ubuntu = 'ubuntu_{0}'.format(arch)
 939 +    arch_packages = 'packages_{0}'.format(arch)
 940 +    OPNFV_CFG['groups'][arch_mos] = deepcopy(OPNFV_CFG['groups']['mos'])
 941 +    OPNFV_CFG['groups'][arch_ubuntu] = deepcopy(OPNFV_CFG['groups']['ubuntu'])
 942 +    OPNFV_CFG[arch_packages] = OPNFV_CFG['packages']
 943 +
 944 +    # Mirror config update & conversion to packetary input
 945 +    group_main_ubuntu = dict()
 946 +    for group in OPNFV_CFG['groups'][arch_mos]:
 947 +        group['uri'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root)
 948 +        group['suite'] = group['suite'].replace('$mos_version', MOS_VERSION)
 949 +    for group in OPNFV_CFG['groups'][arch_ubuntu]:
 950 +        group['uri'] = mirror_ubuntu
 951 +        # FIXME: At `create`, packetary insists on copying all pkgs to dest dir,
 952 +        # so configure it for another dir, which will replace the orig.
 953 +        group['path'] = MIRROR_UBUNTU_TMP_PATH
 954 +        if not group_main_ubuntu and 'main' in group:
 955 +            group_main_ubuntu = [deepcopy(group)]
 956 +            group_main_ubuntu[0]['section'] = ['main']
 957 +
 958 +    # Mirror config dump: MOS (for dep resolution), Ubuntu, Ubuntu[main]
 959 +    write_cfg_file(cfg_m_mos, OPNFV_CFG['groups'][arch_mos])
 960 +    write_cfg_file(cfg_m_ubuntu, OPNFV_CFG['groups'][arch_ubuntu])
 961 +    if MIRROR_UBUNTU_MERGE is None:
 962 +        write_cfg_file(cfg_m_ubuntu_main, group_main_ubuntu)
 963 +    else:
 964 +        # FIXME: For multiarch, only one dump would be enough
 965 +        group_main_ubuntu[0]['origin'] = 'Ubuntu'
 966 +        group_main_ubuntu[0]['path'] = MIRROR_UBUNTU_PATH
 967 +        group_main_ubuntu[0]['uri'] = MIRROR_UBUNTU_PATH
 968 +        write_cfg_file(CFG_MM_UBUNTU, group_main_ubuntu[0])
 969 +
 970 +    # Collect package dependencies from:
 971 +    ## 1. fuel_bootstrap_cli.yaml (bootstrap image additional packages)
 972 +    legacy_unresolved = legacy_diff(None, FUEL_BOOTSTRAP_CLI['packages'] + [
 973 +            FUEL_BOOTSTRAP_CLI['kernel_flavor'],
 974 +            FUEL_BOOTSTRAP_CLI['kernel_flavor'].replace('image', 'headers')],
 975 +        'Bootstrap', arch)
 976 +    ## 2. openstack.yaml FIXTURE definition (default target image packages)
 977 +    for release in FIXTURE:
 978 +        editable = release['fields']['attributes_metadata']['editable']
 979 +        if 'provision' in editable and 'packages' in editable['provision']:
 980 +            release_pkgs = editable['provision']['packages']['value'].split()
 981 +            legacy_unresolved += legacy_diff(legacy_unresolved, release_pkgs,
 982 +                'Release {0}'.format(release['fields']['name']), arch)
 983 +    ## 3. OPNFV additional packages (includes old fuel-mirror ubuntu.yaml pkgs)
 984 +    unresolved = dict()
 985 +    unresolved['mandatory'] = 'exact'
 986 +    unresolved['packages'] = from_legacy_pkglist(legacy_unresolved)
 987 +    if 'packages' in OPNFV_CFG:
 988 +        legacy_diff(legacy_unresolved, to_legacy_pkglist(OPNFV_CFG['packages']),
 989 +            'OPNFV config', arch)
 990 +        unresolved['packages'] += OPNFV_CFG['packages']
 991 +
 992 +    # OPNFV plugins dependency resolution
 993 +    if plugins:
 994 +        for plugin in plugins.split():
 995 +            path = "../{}/packages.yaml".format(plugin)
 996 +            if os.path.isfile(path):
 997 +                f = open(path).read()
 998 +                plugin_yaml = yaml.load(f)
 999 +                new_pkgs = legacy_diff(
1000 +                    to_legacy_pkglist(unresolved['packages']),
1001 +                    plugin_yaml['packages'], 'Plugin {0}'.format(plugin), arch)
1002 +                unresolved['packages'] += from_legacy_pkglist(new_pkgs)
1003 +
1004 +    # Package list (reduced, i.e. no MOS deps, but with OPNFV plugin deps)
1005 +    if MIRROR_UBUNTU_MERGE is None:
1006 +        write_cfg_file(cfg_p_ubuntu_main, unresolved)
1007 +
1008 +    # Mirror package list (full, including MOS/OPNFV plugin deps)
1009 +    unresolved['packages'] += get_unres_pkgs(arch, cfg_m_mos)
1010 +    write_cfg_file(cfg_p_ubuntu, unresolved)
1011 +    do_partial_mirror(arch, cfg_m_ubuntu, cfg_p_ubuntu)
1012 +    if MIRROR_UBUNTU_MERGE is None:
1013 +        # Ubuntu[main] must be evaluated after Ubuntu
1014 +        do_partial_mirror(arch, cfg_m_ubuntu_main, cfg_p_ubuntu_main)
1015 +
1016 +if MIRROR_UBUNTU_MERGE is None:
1017 +    shutil.move(MIRROR_UBUNTU_TMP_PATH, MIRROR_UBUNTU_PATH)
1018 +else:
1019 +    # Construct single-component mirror from all components
1020 +    for arch in UBUNTU_ARCH.split(' '):
1021 +        cfg_pp_ubuntu = '{0}/ubuntu_{1}_packages_paths.yaml'.format(CFG_D, arch)
1022 +        # OPNFV blacklist
1023 +        opnfv_blacklist = to_legacy_pkglist(OPNFV_CFG['opnfv_blacklist'])
1024 +        # FIXME: We need scanpackages to omit older DEBs
1025 +        # Inspired from http://askubuntu.com/questions/198474/
1026 +        os.system('dpkg-scanpackages -a {0} {1} 2>/dev/null | '
1027 +                  'grep -e "^Filename:" | sed "s|Filename: |- file://|g" | '
1028 +                  'grep -v -E "\/({2})_" > {3}'
1029 +                  .format(arch, MIRROR_UBUNTU_TMP_PATH,
1030 +                          '|'.join(opnfv_blacklist), cfg_pp_ubuntu))
1031 +        do_local_repo(arch, CFG_MM_UBUNTU, cfg_pp_ubuntu)
1032 +    shutil.rmtree(MIRROR_UBUNTU_TMP_PATH)
1033 diff --git a/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh b/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh
1034 index 5618eb7..c8c86db 100755
1035 --- a/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh
1036 +++ b/build/f_isoroot/f_repobuild/select_ubuntu_repo.sh
1037 @@ -1,5 +1,6 @@
1038  #!/bin/bash
1039
1040 +UBUNTU_DISTRO="xenial"
1041  BLACKLIST="http://mirrors.se.eu.kernel.org/ubuntu/"
1042  #BLACKLIST+=" http://foo.bar"
1043
1044 @@ -28,26 +29,26 @@ blacklisted () {
1045  check_mirror () {
1046      mirror=$1
1047      status=0
1048 -    for packdir in dists/trusty-updates/main/binary-amd64 \
1049 -        dists/trusty-updates/restricted/binary-amd64 \
1050 -        dists/trusty-updates/universe/binary-amd64 \
1051 -        dists/trusty-updates/multiverse/binary-amd64 \
1052 -        dists/trusty-security/main/binary-amd64 \
1053 -        dists/trusty-security/restricted/binary-amd64 \
1054 -        dists/trusty-security/universe/binary-amd64 \
1055 -        dists/trusty-security/multiverse/binary-amd64 \
1056 -        dists/trusty-proposed/main/binary-amd64 \
1057 -        dists/trusty-proposed/restricted/binary-amd64 \
1058 -        dists/trusty-proposed/universe/binary-amd64 \
1059 -        dists/trusty-proposed/multiverse/binary-amd64 \
1060 -        dists/trusty/main/binary-amd64 \
1061 -        dists/trusty/restricted/binary-amd64 \
1062 -        dists/trusty/universe/binary-amd64 \
1063 -        dists/trusty/multiverse/binary-amd64 \
1064 -        dists/trusty-backports/main/binary-amd64 \
1065 -        dists/trusty-backports/restricted/binary-amd64 \
1066 -        dists/trusty-backports/universe/binary-amd64 \
1067 -        dists/trusty-backports/multiverse/binary-amd64
1068 +    for packdir in dists/${UBUNTU_DISTRO}-updates/main/binary-amd64 \
1069 +        dists/${UBUNTU_DISTRO}-updates/restricted/binary-amd64 \
1070 +        dists/${UBUNTU_DISTRO}-updates/universe/binary-amd64 \
1071 +        dists/${UBUNTU_DISTRO}-updates/multiverse/binary-amd64 \
1072 +        dists/${UBUNTU_DISTRO}-security/main/binary-amd64 \
1073 +        dists/${UBUNTU_DISTRO}-security/restricted/binary-amd64 \
1074 +        dists/${UBUNTU_DISTRO}-security/universe/binary-amd64 \
1075 +        dists/${UBUNTU_DISTRO}-security/multiverse/binary-amd64 \
1076 +        dists/${UBUNTU_DISTRO}-proposed/main/binary-amd64 \
1077 +        dists/${UBUNTU_DISTRO}-proposed/restricted/binary-amd64 \
1078 +        dists/${UBUNTU_DISTRO}-proposed/universe/binary-amd64 \
1079 +        dists/${UBUNTU_DISTRO}-proposed/multiverse/binary-amd64 \
1080 +        dists/${UBUNTU_DISTRO}/main/binary-amd64 \
1081 +        dists/${UBUNTU_DISTRO}/restricted/binary-amd64 \
1082 +        dists/${UBUNTU_DISTRO}/universe/binary-amd64 \
1083 +        dists/${UBUNTU_DISTRO}/multiverse/binary-amd64 \
1084 +        dists/${UBUNTU_DISTRO}-backports/main/binary-amd64 \
1085 +        dists/${UBUNTU_DISTRO}-backports/restricted/binary-amd64 \
1086 +        dists/${UBUNTU_DISTRO}-backports/universe/binary-amd64 \
1087 +        dists/${UBUNTU_DISTRO}-backports/multiverse/binary-amd64
1088      do
1089          for packfile in Release Packages.gz
1090          do
1091 diff --git a/build/install/apt-ftparchive-udeb.conf b/build/install/apt-ftparchive-udeb.conf
1092 index 3b5b239..0cabe94 100644
1093 --- a/build/install/apt-ftparchive-udeb.conf
1094 +++ b/build/install/apt-ftparchive-udeb.conf
1095 @@ -17,8 +17,8 @@ TreeDefault {
1096  };
1097
1098  BinDirectory "pool/debian-installer" {
1099 -  Packages "dists/trusty/main/debian-installer/binary-amd64/Packages";
1100 -  BinOverride "./indices/override.trusty.main.debian-installer";
1101 +  Packages "dists/xenial/main/debian-installer/binary-amd64/Packages";
1102 +  BinOverride "./indices/override.xenial.main.debian-installer";
1103  };
1104
1105  Default {
1106 diff --git a/build/install/install.sh b/build/install/install.sh
1107 index 198f26c..b2f6ccc 100755
1108 --- a/build/install/install.sh
1109 +++ b/build/install/install.sh
1110 @@ -122,17 +122,17 @@ prep_make_live() {
1111      ssh-copy-id root@$FUELHOST
1112      sshfs root@1${FUELHOST}:/ $TMP_HOSTMOUNT
1113
1114 -    if [ -f  $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then
1115 +    if [ -f  $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then
1116          echo "Error - found backup file for Packages!"
1117          exit 1
1118      fi
1119
1120 -    if [ -f  $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then
1121 +    if [ -f  $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then
1122          echo "Error - found backup file for Packages.gz!"
1123          exit 1
1124      fi
1125
1126 -    if [ -f  $REPO/dists/trusty/Release.backup ]; then
1127 +    if [ -f  $REPO/dists/xenial/Release.backup ]; then
1128          echo "Error - found backup file for Release!"
1129          exit 1
1130      fi
1131 @@ -142,9 +142,9 @@ prep_make_live() {
1132          exit 1
1133      fi
1134
1135 -    cp $REPO/dists/trusty/main/binary-amd64/Packages $REPO/dists/trusty/main/binary-amd64/Packages.backup
1136 -    cp $REPO/dists/trusty/main/binary-amd64/Packages.gz $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup
1137 -    cp $REPO/dists/trusty/Release $REPO/dists/trusty/Release.backup
1138 +    cp $REPO/dists/xenial/main/binary-amd64/Packages $REPO/dists/xenial/main/binary-amd64/Packages.backup
1139 +    cp $REPO/dists/xenial/main/binary-amd64/Packages.gz $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup
1140 +    cp $REPO/dists/xenial/Release $REPO/dists/xenial/Release.backup
1141      cp -Rvp $DEST/etc/puppet $DEST/etc/puppet.backup
1142  }
1143
1144 @@ -443,7 +443,7 @@ copy_packages() {
1145      echo Not running apt-ftparchive generate "${APT_UDEB_CONF}"
1146
1147      # Fuel also needs this index file
1148 -    # cat dists/trusty/main/binary-amd64/Packages | \
1149 +    # cat dists/xenial/main/binary-amd64/Packages | \
1150      #    awk '/^Package:/{pkg=$2}
1151      # /^Version:/{print pkg ": \"" $2 "\""}' > ubuntu-versions.yaml
1152      # cp ubuntu-versions.yaml $DEST
1153 diff --git a/build/install/uninstall.sh b/build/install/uninstall.sh
1154 index a9e74bc..5252c30 100755
1155 --- a/build/install/uninstall.sh
1156 +++ b/build/install/uninstall.sh
1157 @@ -31,17 +31,17 @@ DEST=$MOUNT
1158  REPO=$DEST/var/www/nailgun/ubuntu/fuelweb/x86_64
1159
1160  cd $REPO
1161 -if [ ! -f  $REPO/dists/trusty/main/binary-amd64/Packages.backup ]; then
1162 +if [ ! -f  $REPO/dists/xenial/main/binary-amd64/Packages.backup ]; then
1163      echo "Error - didn't find backup file for Packages!"
1164      exit 1
1165  fi
1166
1167 -if [ ! -f  $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup ]; then
1168 +if [ ! -f  $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup ]; then
1169      echo "Error - didn't find backup file for Packages.gz!"
1170      exit 1
1171  fi
1172
1173 -if [ ! -f  $REPO/dists/trusty/Release.backup ]; then
1174 +if [ ! -f  $REPO/dists/xenial/Release.backup ]; then
1175      echo "Error - didn't find backup file for Release!"
1176      exit 1
1177  fi
1178 @@ -71,9 +71,9 @@ cd $REPO
1179
1180  echo "Restoring backups of datafiles"
1181
1182 -rm -f $REPO/dists/trusty/main/binary-amd64/Packages $REPO/dists/trusty/main/binary-amd64/Packages.gz
1183 -rm -f $REPO/dists/trusty/Release $DEST/etc/puppet/manifests/site.pp
1184 -mv $REPO/dists/trusty/main/binary-amd64/Packages.backup $REPO/dists/trusty/main/binary-amd64/Packages
1185 -mv $REPO/dists/trusty/main/binary-amd64/Packages.gz.backup $REPO/dists/trusty/main/binary-amd64/Packages.gz
1186 -mv $REPO/dists/trusty/Release.backup $REPO/dists/trusty/Release
1187 +rm -f $REPO/dists/xenial/main/binary-amd64/Packages $REPO/dists/xenial/main/binary-amd64/Packages.gz
1188 +rm -f $REPO/dists/xenial/Release $DEST/etc/puppet/manifests/site.pp
1189 +mv $REPO/dists/xenial/main/binary-amd64/Packages.backup $REPO/dists/xenial/main/binary-amd64/Packages
1190 +mv $REPO/dists/xenial/main/binary-amd64/Packages.gz.backup $REPO/dists/xenial/main/binary-amd64/Packages.gz
1191 +mv $REPO/dists/xenial/Release.backup $REPO/dists/xenial/Release
1192  mv $DEST/etc/puppet/manifests/site.pp.backup $DEST/etc/puppet/manifests/site.pp
1193 diff --git a/deploy/config/dea_base.yaml b/deploy/config/dea_base.yaml
1194 index ce81994..2520cf5 100644
1195 --- a/deploy/config/dea_base.yaml
1196 +++ b/deploy/config/dea_base.yaml
1197 @@ -722,22 +722,10 @@ settings:
1198          value:
1199          - name: ubuntu
1200            priority: null
1201 -          section: main universe multiverse
1202 +          section: main
1203            suite: xenial
1204            type: deb
1205 -          uri: http://archive.ubuntu.com/ubuntu/
1206 -        - name: ubuntu-updates
1207 -          priority: null
1208 -          section: main universe multiverse
1209 -          suite: xenial-updates
1210 -          type: deb
1211 -          uri: http://archive.ubuntu.com/ubuntu/
1212 -        - name: ubuntu-security
1213 -          priority: null
1214 -          section: main universe multiverse
1215 -          suite: xenial-security
1216 -          type: deb
1217 -          uri: http://archive.ubuntu.com/ubuntu/
1218 +          uri: http://10.20.0.2:8080/mirrors/ubuntu/
1219          - name: mos
1220            priority: 1050
1221            section: main restricted