1 From: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2 Date: Wed, 6 Jul 2016 17:03:49 +0200
3 Subject: [PATCH] f_repobuild: Use packetary to build partial mirror
5 FIXME: DO NOT MERGE until [2, 3] are merged upstream & included here!
6 TODO: drop ALL fuel-mirror dependencies, for Fuel Newton compatibility.
8 Packetary [1] will replace fuel-mirror in upstream Fuel.
9 fuel-mirror is using packetary under the hood already, so the
10 transition should not be too hard.
12 Adapt OPNFV Ubuntu partial mirror build:
13 - obsolete opnfv-config.yaml (packetary uses CLI args for this);
14 - split old ubuntu.yaml into:
15 * mos_<arch>_mirror.yaml - MOS mirror cfg, only for deps analysis;
16 * ubuntu_<arch>_mirror.yaml - Ubuntu mirror cfg, for pkgs download;
17 * ubuntu_<arch>_packages.yaml - Ubuntu packages to download;
19 While the consumers of our mirror still use `debootstrap`, which only
20 supports the "main" repo component, make sure the "main" package set
21 is complete, in the scope of:
22 * ubuntu_<arch>_mirror_main.yaml - Ubuntu [main only] mirror;
23 * ubuntu_<arch>_packages_main.yaml - Ubuntu packages for [main];
25 New workflow (executed for EACH architecture):
26 1. Parse "fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml"
27 and write correponding sections to new yaml files described above;
29 NOTE: Since fuel-mirror is going to be made obsolete, there are a
30 number of new ways we can gather the data currently passed via
31 fuel-mirror's "ubuntu.yaml" (e.g. use "mos-repos" from Fuel ISO
32 to determine MOS dependencies).
34 2. Add OPNFV extra package dependencies (NEW)
35 Currently, this is empty / not used, but might be used to add
36 extra packages to the local mirror, like additional kernels,
39 NOTE: Dependencies up to and including this group should also
40 be satisfiable from Ubuntu "main" repo component only.
42 3. Append plugin dependencies to the package list;
44 4. Run `packetary unresolved` for MOS repos, gathering MOS deps that
45 should be fetched from Ubuntu. The new packages will be appended
46 to the predefined list from old "ubuntu.yaml";
48 5. Run `packetary clone` to download all required pkgs for the
49 partial Ubuntu mirror.
51 6. IF repo component merging is disabled, run `packetary clone` only
52 on Ubuntu[main] repo components to download any (still) missing
53 dependencies for `debootstrap`, which only looks at this component,
54 and not the whole mirror.
56 7. IF repo component merging is enabled, run first `dpkg-scanpackages`
57 to filter out old duplicate versions of packages, then run
58 `packetary create` to merge all repo comps into a single "main".
61 - multiarch support (activated via UBUNTU_ARCH envvar);
62 - fixed debootstrap missing deps in "main" component of Ubuntu mirror;
63 - factored out some hardcodes into variables/functions;
64 - add .gitignore file;
65 - move generated config files to "opnfv_config" subdir;
66 - added arm64 Ubuntu repo URL (ports.ubuntu.com is separated from the
67 main Ubuntu mirror, archive.ubuntu.com, which only holds x86 pkgs);
70 - introduce env var "MIRROR_UBUNTU_MERGE" to control local mirror
71 repo-component merging into a single "main";
72 - enable mirror repo-component merging by default, edit
73 fuel_bootstrap_cli.yaml accordingly;
76 - edit dea_base to use only main mirror;
77 - fix duplicated logs;
78 - remove `-q` flag for `create`;
80 NOTE: Without filtering out old package verisons using
81 `dpkg-scanpackages`, bootstrap build cannot solve all deps.
83 FIXME: Packetary solves missing dependecies by also accepting
84 different packages that provide the same package name (e.g. "ifupdown"
85 is satisfied by "netscript" package).
87 FIXME: Repo component merging is sort of slow, since packetary insists
88 on copying the source files to the destination dir even if they are
89 pointing to the same repo.
91 FIXME: Packetary `create` uses a different directory scheme for the
92 created mirror than the upstream Ubuntu, e.g.:
93 [p] mirrors/ubuntu/pool/main/l/lvm2_2.02.98-6ubuntu2_amd64.deb
94 [U] mirrors/ubuntu/pool/main/l/lvm2/lvm2_2.02.98-6ubuntu2_amd64.deb
96 This disencourages creating the "merged" mirror in the same location
97 as the source partial mirror.
99 [1] https://wiki.openstack.org/wiki/Packetary
100 [2] https://review.openstack.org/#/c/392936/
101 [3] https://review.openstack.org/#/c/392937/
107 Change-Id: If2ee86f348b7683c83bfaf686baba4f1b8f555f0
108 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
110 build/config.mk | 5 +
111 build/f_isoroot/f_repobuild/.gitignore | 4 +
112 build/f_isoroot/f_repobuild/Makefile | 34 ++-
113 build/f_isoroot/f_repobuild/config.mk | 26 +++
114 .../f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml | 14 +-
115 build/f_isoroot/f_repobuild/opnfv_mirror_conf.py | 57 -----
116 build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py | 247 +++++++++++++++++++++
117 build/f_isoroot/f_repobuild/opnfv_packages.yaml | 14 ++
118 deploy/config/dea_base.yaml | 14 +-
119 9 files changed, 313 insertions(+), 102 deletions(-)
120 create mode 100644 build/f_isoroot/f_repobuild/.gitignore
121 create mode 100644 build/f_isoroot/f_repobuild/config.mk
122 delete mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
123 create mode 100755 build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
124 create mode 100644 build/f_isoroot/f_repobuild/opnfv_packages.yaml
126 diff --git a/build/config.mk b/build/config.mk
127 index ab27858..a62b7e9 100644
128 --- a/build/config.mk
129 +++ b/build/config.mk
130 @@ -17,6 +17,11 @@ FUEL_MAIN_TAG = 9.0.1
132 OPENSTACK_VERSION = mitaka-9.0
134 +# List of space-separated Ubuntu architectures supported with current build
135 +# Format: same as `dpkg-architecture -qDEB_HOST_ARCH1`
136 +# NOTE: Currently only amd64 is supported by Fuel@OPNFV. Armband adds arm64.
137 +export UBUNTU_ARCH ?= amd64
139 # FIXME(alav): Disable remote tracking for now, stick to submodule commits
142 diff --git a/build/f_isoroot/f_repobuild/.gitignore b/build/f_isoroot/f_repobuild/.gitignore
144 index 0000000..cfedefb
146 +++ b/build/f_isoroot/f_repobuild/.gitignore
152 diff --git a/build/f_isoroot/f_repobuild/Makefile b/build/f_isoroot/f_repobuild/Makefile
153 index ad1a934..5ab2650 100644
154 --- a/build/f_isoroot/f_repobuild/Makefile
155 +++ b/build/f_isoroot/f_repobuild/Makefile
157 ##############################################################################
158 -# Copyright (c) 2015 Ericsson AB and others.
159 +# Copyright (c) 2015,2016 Ericsson AB, Enea AB and others.
160 # stefan.k.berg@ericsson.com
161 # jonas.bjurel@ericsson.com
162 +# Alexandru.Avadanii@enea.com
163 # All rights reserved. This program and the accompanying materials
164 # are made available under the terms of the Apache License, Version 2.0
165 # which accompanies this distribution, and is available at
170 -TMP_ROOT_DIR := $(shell echo "$(MIRROR_UBUNTU_ROOT)" | cut -d "/" -f2)
172 include ../../config.mk
176 -export OPENSTACK_VERSION
177 +export MIRROR_UBUNTU_OPNFV_PATH:=$(TOP)/nailgun/mirrors/ubuntu
183 - sudo apt-get install -y git libxml2-dev libxslt-dev python-dev python-pip libz-dev libyaml-dev createrepo python-yaml
185 - sudo mkdir -p /var/www/nailgun
186 + sudo apt-get install -y createrepo git libxml2-dev libxslt1-dev \
187 + python-dev zlib1g-dev
188 + rm -Rf nailgun packetary opnfv_config && mkdir opnfv_config
189 ln -sf ${F_SUBMOD_DIR}/fuel-mirror fuel-mirror
190 - sudo pip install -U -r ./fuel-mirror/requirements.txt
191 - sudo pip install ./fuel-mirror
192 - sudo pip install ./fuel-mirror/contrib/fuel_mirror
193 - ./opnfv_mirror_conf.py
194 - sudo fuel-mirror --debug --config ./opnfv-config.yaml create --group ubuntu --pattern=ubuntu
195 - sudo chmod -R 755 /var/www/nailgun
196 - cp -Rp /var/www/nailgun .
197 - # In the end we want to have ubuntu repository in mirrors/ubuntu directory
198 - -if [ "$(MIRROR_UBUNTU_ROOT)" != "/ubuntu/" ]; then \
199 - mkdir -p nailgun/mirrors/ubuntu;\
200 - mv nailgun/mirrors$(MIRROR_UBUNTU_ROOT)* nailgun/mirrors/ubuntu;\
201 - [ "$(MIRROR_UBUNTU_ROOT)" != "/" ] && rm -rf nailgun/mirrors/$(TMP_ROOT_DIR);\
202 + git clone --quiet $(PACKETARY_REPO)
203 + if [ -n $(PACKETARY_COMMIT) ]; then \
204 + git -C packetary checkout $(PACKETARY_COMMIT); \
206 + sudo pip install -U -r ./packetary/requirements.txt
207 + sudo pip install ./packetary
208 + # Handle config and mirror build in one place
209 + ./opnfv_mirror_ubuntu.py
210 # Store artifact in cache straight away if caching is enabled
211 # (no .cacheid will be present unless this is a cached build)
212 test -f .cacheid && $(MAKE) -f Makefile put-cache || exit 0
216 - @rm -rf ../release/opnfv/nailgun nailgun fuel-mirror opnfv-config.yaml ubuntu.yaml
217 + @rm -rf ../release/opnfv/nailgun nailgun packetary fuel-mirror opnfv_config
221 diff --git a/build/f_isoroot/f_repobuild/config.mk b/build/f_isoroot/f_repobuild/config.mk
223 index 0000000..79e7d1a
225 +++ b/build/f_isoroot/f_repobuild/config.mk
227 +##############################################################################
228 +# Copyright (c) 2016 Ericsson AB, Enea AB and others.
229 +# stefan.k.berg@ericsson.com
230 +# jonas.bjurel@ericsson.com
231 +# Alexandru.Avadanii@enea.com
232 +# All rights reserved. This program and the accompanying materials
233 +# are made available under the terms of the Apache License, Version 2.0
234 +# which accompanies this distribution, and is available at
235 +# http://www.apache.org/licenses/LICENSE-2.0
236 +##############################################################################
238 +# Use a recent master commit, since tags/branches are not yet mature
239 +# FIXME(armband): Update upstream commit ref once [1, 2] are merged
240 +# [1] https://review.openstack.org/#/c/392937/
241 +# [2] https://review.openstack.org/#/c/392936/
242 +export PACKETARY_REPO?=https://github.com/openstack/packetary
243 +export PACKETARY_COMMIT?=c46465c3255a9f5e59a05b8701e06054df39f32f
245 +# arm64 Ubuntu mirror is separated from archive.ubuntu.com
246 +export MIRROR_UBUNTU_URL_arm64=http://ports.ubuntu.com/ubuntu-ports/
247 +export MIRROR_UBUNTU_ROOT_arm64=ubuntu-ports
249 +# Merge all local mirror repo components/section into single "main"
250 +# NOTE: When changing this, make sure to also update all consumer config, like:
251 +# - fuel_bootstrap_cli.yaml
252 +export MIRROR_UBUNTU_MERGE=true
253 diff --git a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
254 index 19dea1c..52786fb 100644
255 --- a/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
256 +++ b/build/f_isoroot/f_repobuild/fuel_bootstrap_cli.yaml
261 - section: "main universe multiverse"
263 uri: "http://127.0.0.1:8080/mirrors/ubuntu"
267 - - name: ubuntu-updates
268 - section: "main universe multiverse"
269 - uri: "http://127.0.0.1:8080/mirrors/ubuntu"
271 - suite: trusty-updates
273 - - name: ubuntu-security
274 - section: "main universe multiverse"
275 - uri: "http://127.0.0.1:8080/mirrors/ubuntu"
277 - suite: trusty-security
280 section: "main restricted"
281 uri: "http://127.0.0.1:8080/ubuntu/x86_64"
282 diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py b/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
283 deleted file mode 100755
284 index 1c7eb59..0000000
285 --- a/build/f_isoroot/f_repobuild/opnfv_mirror_conf.py
288 -#!/usr/bin/env python
289 -##############################################################################
290 -# Copyright (c) 2015 Ericsson AB and others.
291 -# mskalski@mirantis.com
292 -# All rights reserved. This program and the accompanying materials
293 -# are made available under the terms of the Apache License, Version 2.0
294 -# which accompanies this distribution, and is available at
295 -# http://www.apache.org/licenses/LICENSE-2.0
296 -##############################################################################
300 -current_snapshot = os.environ["LATEST_TARGET_UBUNTU"]
301 -mos_version = os.environ['MOS_VERSION']
302 -openstack_version = os.environ['OPENSTACK_VERSION']
303 -mos_ubuntu = os.environ['MIRROR_MOS_UBUNTU']
304 -mos_ubuntu_root = os.environ['MIRROR_MOS_UBUNTU_ROOT']
305 -mirror_ubuntu = os.environ['MIRROR_UBUNTU_URL']
306 -if os.environ.get('BUILD_FUEL_PLUGINS'):
307 - plugins = os.environ['BUILD_FUEL_PLUGINS']
309 - plugins = os.environ['PLUGINS']
312 -configuration_file = open('fuel-mirror/contrib/fuel_mirror/etc/config.yaml').read()
313 -conf = yaml.load(configuration_file)
314 -conf['pattern_dir'] = '.'
315 -conf['openstack_version'] = openstack_version
316 -conf['mos_version'] = mos_version
318 -with open('opnfv-config.yaml', 'w') as outfile:
319 - outfile.write( yaml.dump(conf, default_flow_style=False) )
321 -pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read()
322 -pattern = yaml.load(pattern_file)
323 -pattern['mos_baseurl'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root)
324 -pattern['ubuntu_baseurl'] = mirror_ubuntu
325 -for group in pattern['groups']['mos']:
326 - group['uri'] = pattern['mos_baseurl']
327 -for group in pattern['groups']['ubuntu']:
328 - group['uri'] = pattern['ubuntu_baseurl']
330 -for plugin in plugins.split():
331 - path = "../{}/packages.yaml".format(plugin)
332 - if os.path.isfile(path):
333 - f = open(path).read()
334 - plugin_yaml = yaml.load(f)
335 - plugin_set = set(plugin_yaml['packages'])
336 - main_set = set(pattern['packages'])
337 - new_packages = plugin_set - main_set
338 - print "Plugin {} require new packages: {}".format(plugin, ', '.join(new_packages))
339 - pattern['packages'] = pattern['packages'] + list(new_packages)
341 -pattern['requirements']['ubuntu'] = pattern['packages']
343 -with open('ubuntu.yaml', 'w') as outfile:
344 - outfile.write( yaml.safe_dump(pattern, default_flow_style=False) )
345 diff --git a/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
347 index 0000000..da88445
349 +++ b/build/f_isoroot/f_repobuild/opnfv_mirror_ubuntu.py
351 +#!/usr/bin/env python
352 +##############################################################################
353 +# Copyright (c) 2015,2016 Ericsson AB, Mirantis Inc., Enea AB and others.
354 +# mskalski@mirantis.com
355 +# Alexandru.Avadanii@enea.com
356 +# All rights reserved. This program and the accompanying materials
357 +# are made available under the terms of the Apache License, Version 2.0
358 +# which accompanies this distribution, and is available at
359 +# http://www.apache.org/licenses/LICENSE-2.0
360 +##############################################################################
362 +##############################################################################
363 +# Build multiarch partial local Ubuntu mirror
364 +##############################################################################
365 +# Design quirks / workarounds:
366 +# 1. Fuel-agent uses `debootstrap` to build bootstrap and target chroots from
367 +# the local mirror; which only uses the "main" component from the first
368 +# repository, i.e. does not include "updates"/"security".
369 +# In order to fullfill all debootstrap dependencies in "main" repo, we will
370 +# do an extra packetary run using a reduced scope:
371 +# - only "main" component of the first mirror;
372 +# - reduced package dependency list (without MOS/OPNFV plugin deps).
373 +# 2. If repo structure is not mandatory to be in sync with official mirrors,
374 +# we can mitigate the issue by "merging" all repo-components into a single
376 +##############################################################################
377 +# Mirror build steps:
378 +# 1. Parse old <fuel-mirror> package list ("ubuntu.yaml");
379 +# 2. Parse new "opnfv_packages.yaml" list of additional packages;
380 +# 3. Inherit enviroment variable(s) for mirror URLs, paths etc.
381 +# - Allow arch-specific overrides for each env var;
382 +# 4. For each architecture in UBUNTU_ARCH:
383 +# 4.1. Mirror config is defined based on common config + OPNFV overrides;
384 +# - Convert old configuration format to packetary style where needed;
385 +# 4.2. Package lists are defined based on common config + OPNFV deps;
386 +# - Keep track of "main" packages separately, required by debootstrap;
387 +# 4.3. Clone/update all mirror components;
388 +# 4.4. IF mirror merging is disabled OR workaround for ifupdown (see below):
389 +# - Clone/update "main" mirror component (fix missing debootstrap deps);
390 +# 5. IF mirror merging is enabled:
391 +# - Use `dpkg-scanpackages` to filter out old versions of duplicate pkgs;
392 +# - Run `packetary create` on the set of downloaded packages, merging
393 +# them on the fly into a single-component mirror;
394 +##############################################################################
401 +from contextlib import contextmanager
402 +from cStringIO import StringIO
403 +from packetary.cli.app import main
406 +def captureStdOut(output):
407 + stdout = sys.stdout
408 + sys.stdout = output
410 + sys.stdout = stdout
412 +# FIXME: Find a better approach for eliminating duplicate logs than this
413 +def force_logger_reload():
414 + for mod in sys.modules.keys():
415 + if mod.startswith('logging'):
417 + reload(sys.modules[mod])
421 +# Determine missing package dependecies for a mirror defition
422 +def get_unres_pkgs(arch, cfg_mirror):
423 + unresolved = dict()
424 + unresolved['packages'] = list()
425 + packetary_output = StringIO()
426 + with captureStdOut(packetary_output):
427 + main('unresolved -a {0} -r {1} -c name version --sep ;'
428 + .format(_ARCHITECTURES[arch], cfg_mirror).split(' '))
429 + for dep_pkg in packetary_output.getvalue().splitlines():
430 + if dep_pkg.startswith('#'):
432 + dep = dep_pkg.split(';')
433 + unresolved['packages'] += [ {'name': dep[0], 'version': dep[1]} ]
434 + force_logger_reload()
437 +# Create single-component local repo (one arch per call)
438 +def do_local_repo(arch, cfg_repo, cfg_packages_paths):
439 + # Packetary does not use a global config file, so pass old settings here.
440 + main('create -t deb -a {0} --repository {1} --package-files {2}'
441 + ' --ignore-errors-num 2 --retries-num 3 --threads-num 10'
442 + .format(_ARCHITECTURES[arch], cfg_repo, cfg_packages_paths).split(' '))
443 + force_logger_reload()
445 +# Clone partial local mirror (one arch per call)
446 +def do_partial_mirror(arch, cfg_mirror, cfg_packages):
447 + # Note: '-d .' is ignored, as each mirror defines its own path.
448 + main('clone -t deb -a {0} -r {1} -R {2} -d .'
449 + ' --ignore-errors-num 2 --retries-num 3 --threads-num 10'
450 + .format(_ARCHITECTURES[arch], cfg_mirror, cfg_packages).split(' '))
451 + force_logger_reload()
453 +# Write configuration (yaml) file (package list / mirror defition)
454 +def write_cfg_file(cfg_mirror, data):
455 + with open(cfg_mirror, 'w') as outfile:
456 + outfile.write( yaml.safe_dump(data, default_flow_style=False) )
458 +# Allow arch-specific overrides of env vars
459 +def get_env(env_var, arch=None):
461 + env_var_arch = '{0}_{1}'.format(env_var, arch)
462 + if os.environ.get(env_var_arch):
463 + return os.environ[env_var_arch]
464 + if os.environ.get(env_var):
465 + return os.environ[env_var]
468 +# Architecture name mapping (dpkg:packetary) for packetary CLI invocation
472 + "arm64": "aarch64",
475 +# Arch-indepedent configuration (old fuel-mirror + OPNFV extra packages)
476 +cfg_dir = 'opnfv_config'
477 +cfg_p_opnfv = 'opnfv_packages.yaml'
478 +mos_version = get_env('MOS_VERSION')
479 +ubuntu_arch = get_env('UBUNTU_ARCH')
480 +mirror_ubuntu_path = get_env('MIRROR_UBUNTU_OPNFV_PATH')
481 +mirror_ubuntu_tmp_path = '{0}.tmp'.format(mirror_ubuntu_path)
482 +mirror_ubuntu_merge = get_env('MIRROR_UBUNTU_MERGE')
483 +cfg_mm_ubuntu = '{0}/ubuntu_mirror_local.yaml'.format(cfg_dir)
484 +pattern_file = open('fuel-mirror/contrib/fuel_mirror/data/ubuntu.yaml').read()
485 +pattern = yaml.load(pattern_file)
486 +opnfv_pkgs_yaml = open(cfg_p_opnfv).read()
487 +opnfv_pkgs = yaml.load(opnfv_pkgs_yaml)
489 +# FIXME: Packetary solves missing dependecies by also accepting
490 +# different packages that provide the same name (e.g. "ifupdown" dependency
491 +# is satisfied by "netscript" package from "universe" repo-component).
492 +# Work around this by resolving all deps in "main" repo-component,
493 +# then scan and keep only latest debs for the whole <merged> repo.
494 +mirror_ubuntu_resolve_main_deps = True
496 +# Create local partial mirror using packetary, one arch at a time
497 +for arch in ubuntu_arch.split(' '):
498 + # Mirror / Package env vars, arch-overrideable
499 + mos_ubuntu = get_env('MIRROR_MOS_UBUNTU', arch)
500 + mos_ubuntu_root = get_env('MIRROR_MOS_UBUNTU_ROOT', arch)
501 + mirror_ubuntu = get_env('MIRROR_UBUNTU_URL', arch)
502 + plugins = get_env('BUILD_FUEL_PLUGINS', arch)
503 + if plugins is None:
504 + plugins = get_env('PLUGINS', arch)
506 + # Mirror / Package list configuration files (arch-specific)
507 + cfg_m_mos = '{0}/mos_{1}_mirror.yaml'.format(cfg_dir, arch)
508 + cfg_m_ubuntu = '{0}/ubuntu_{1}_mirror.yaml'.format(cfg_dir, arch)
509 + cfg_p_ubuntu = '{0}/ubuntu_{1}_packages.yaml'.format(cfg_dir, arch)
510 + cfg_m_ubuntu_main = '{0}/ubuntu_{1}_mirror_main.yaml'.format(cfg_dir, arch)
511 + cfg_p_ubuntu_main = '{0}/ubuntu_{1}_packages_main.yaml'.format(cfg_dir, arch)
513 + # Mirror config fork before customizing (arch-specific)
514 + arch_group_mos = 'mos_{0}'.format(arch)
515 + arch_group_ubuntu = 'ubuntu_{0}'.format(arch)
516 + arch_packages = 'packages_{0}'.format(arch)
517 + pattern['groups'][arch_group_mos] = copy.deepcopy(pattern['groups']['mos'])
518 + pattern['groups'][arch_group_ubuntu] = copy.deepcopy(pattern['groups']['ubuntu'])
519 + pattern[arch_packages] = pattern['packages']
521 + # Mirror config update & conversion to packetary input
522 + group_main_ubuntu = dict()
523 + for group in pattern['groups'][arch_group_mos]:
524 + group['uri'] = "http://{}{}".format(mos_ubuntu, mos_ubuntu_root)
525 + group['suite'] = group['suite'].replace('$mos_version', mos_version)
526 + group['section'] = group['section'].split()
527 + for group in pattern['groups'][arch_group_ubuntu]:
528 + group['uri'] = mirror_ubuntu
529 + # FIXME: At `create`, packetary insists on copying all pkgs to dest dir,
530 + # so configure it for another dir, which will replace the orig at the end.
531 + group['path'] = mirror_ubuntu_tmp_path
532 + group['section'] = group['section'].split()
533 + if not group_main_ubuntu and 'main' in group:
534 + group_main_ubuntu = [ copy.deepcopy(group) ]
535 + group_main_ubuntu[0]['section'] = [ 'main' ]
537 + # Mirror config dump: MOS (for dep resolution), Ubuntu, Ubuntu[main]
538 + write_cfg_file(cfg_m_mos, pattern['groups'][arch_group_mos])
539 + write_cfg_file(cfg_m_ubuntu, pattern['groups'][arch_group_ubuntu])
540 + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None:
541 + write_cfg_file(cfg_m_ubuntu_main, group_main_ubuntu)
542 + if mirror_ubuntu_merge is not None:
543 + # FIXME: For multiarch, only one dump would be enough
544 + group_main_ubuntu[0]['origin'] = 'Ubuntu'
545 + group_main_ubuntu[0]['path'] = mirror_ubuntu_path
546 + group_main_ubuntu[0]['uri'] = mirror_ubuntu_path
547 + write_cfg_file(cfg_mm_ubuntu, group_main_ubuntu[0])
549 + # Package list conversion from `old fuel-mirror` to `packetary` style + OPNFV
550 + unresolved_pkgs = dict()
551 + unresolved_pkgs['packages'] = list()
552 + unresolved_pkgs['mandatory'] = 'exact'
553 + if opnfv_pkgs['packages'] is not None:
554 + unresolved_pkgs['packages'] = opnfv_pkgs['packages']
555 + for pkg in pattern['packages']:
556 + unresolved_pkgs['packages'] += [ {'name': pkg} ]
558 + # OPNFV plugins dependency resolution
559 + for plugin in plugins.split():
560 + path = "../{}/packages.yaml".format(plugin)
561 + if os.path.isfile(path):
562 + f = open(path).read()
563 + plugin_yaml = yaml.load(f)
564 + plugin_set = set(plugin_yaml['packages'])
565 + main_set = set(pattern['packages'])
566 + new_packages = plugin_set - main_set
567 + print('Plugin {0} requires new packages for arch [{1}]: {2}'
568 + .format(plugin, arch, ', '.join(new_packages)))
569 + for pkg in new_packages:
570 + unresolved_pkgs['packages'] += [ {'name': pkg} ]
572 + # Package list (reduced, i.e. no MOS deps, but with OPNFV plugin deps)
573 + # FIXME: This helps work around packetary solving main deps from universe
574 + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None:
575 + write_cfg_file(cfg_p_ubuntu_main, unresolved_pkgs)
577 + # Mirror package list (full, including MOS/OPNFV plugin deps)
578 + unresolved_pkgs['packages'] += get_unres_pkgs(arch, cfg_m_mos)['packages']
579 + write_cfg_file(cfg_p_ubuntu, unresolved_pkgs)
580 + do_partial_mirror(arch, cfg_m_ubuntu, cfg_p_ubuntu)
581 + if mirror_ubuntu_resolve_main_deps or mirror_ubuntu_merge is None:
582 + # Ubuntu[main] must be evaluated after Ubuntu
583 + do_partial_mirror(arch, cfg_m_ubuntu_main, cfg_p_ubuntu_main)
585 +if mirror_ubuntu_merge is None:
586 + shutil.move(mirror_ubuntu_tmp_path, mirror_ubuntu_path)
588 + # Construct single-component mirror from all components
589 + for arch in ubuntu_arch.split(' '):
590 + cfg_pp_ubuntu = '{0}/ubuntu_{1}_packages_paths.yaml'.format(cfg_dir, arch)
591 + # FIXME: We need scanpackages to omit older DEBs
592 + # Inspired from http://askubuntu.com/questions/198474/
593 + os.system('dpkg-scanpackages -a {0} {1} 2>/dev/null | '
594 + 'grep -e "^Filename:" | sed "s|Filename: |- file://|g" > {2}'
595 + .format(arch, mirror_ubuntu_tmp_path, cfg_pp_ubuntu))
596 + do_local_repo(arch, cfg_mm_ubuntu, cfg_pp_ubuntu)
597 + shutil.rmtree(mirror_ubuntu_tmp_path)
598 diff --git a/build/f_isoroot/f_repobuild/opnfv_packages.yaml b/build/f_isoroot/f_repobuild/opnfv_packages.yaml
600 index 0000000..3f5c59f
602 +++ b/build/f_isoroot/f_repobuild/opnfv_packages.yaml
604 +##############################################################################
605 +# Copyright (c) 2016 Enea AB and others.
606 +# Alexandru.Avadanii@enea.com
607 +# All rights reserved. This program and the accompanying materials
608 +# are made available under the terms of the Apache License, Version 2.0
609 +# which accompanies this distribution, and is available at
610 +# http://www.apache.org/licenses/LICENSE-2.0
611 +##############################################################################
613 +# This file lists packages that should be present in the local Ubuntu mirror,
614 +# but are not direct dependencies of any other packages from MOS or Ubuntu.
615 +# e.g.: additional kernels, bootloaders etc.
619 diff --git a/deploy/config/dea_base.yaml b/deploy/config/dea_base.yaml
620 index f5a03e2..370a75a 100644
621 --- a/deploy/config/dea_base.yaml
622 +++ b/deploy/config/dea_base.yaml
623 @@ -633,22 +633,10 @@ settings:
627 - section: main universe multiverse
631 uri: http://10.20.0.2:8080/mirrors/ubuntu/
632 - - name: ubuntu-updates
634 - section: main universe multiverse
635 - suite: trusty-updates
637 - uri: http://10.20.0.2:8080/mirrors/ubuntu/
638 - - name: ubuntu-security
640 - section: main universe multiverse
641 - suite: trusty-security
643 - uri: http://10.20.0.2:8080/mirrors/ubuntu/
646 section: main restricted