:maxdepth: 2
release-process
+ release-automation
stable-branch-guide
versioning
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+.. (c) Open Platform for NFV Project, Inc. and its contributors
+
+.. _release-automation:
+
+==================
+Release Automation
+==================
+
+This page describes how projects can take advantage of the release
+automation introduced in Fraser for creating their stable branch, and
+stable branch Jenkins jobs.
+
+It also describes the structures of the ``releases`` directory and the
+associated scripts.
+
+Stable Branch Creation
+----------------------
+
+If your project participated in the last release (beginning with
+Euphrates), perform the following steps:
+
+#. Copy your project's release file to the new release directory. For
+ example::
+
+ cp releases/euphrates/apex.yaml releases/fraser/apex.yaml
+
+#. For projects who are participating the in the stable release process for
+ the first time, you can either copy a different project's file and
+ changing the values to match your project, or use the following
+ template, replacing values marked with ``<`` and ``>``:
+
+ .. code-block:: yaml
+
+ ---
+ project: <opnfv-project-name>
+ project-type: <opnfv-project-type>
+ release-model: stable
+
+ branches:
+ - name: stable/<release>
+ location:
+ <project-repo>: <git-sha1>
+
+#. Modify the file, replacing the previous stable branch name with the
+ new release name, and the commit the branch will start at. For
+ example:
+
+ .. code-block:: yaml
+
+ branches:
+ - name: stable/fraser
+ location:
+ apex: <git-full-sha1>
+
+#. If your project contains multiple repositories, add them to the list
+ of branches. They can also be added later if more time is needed
+ before the stable branch window closes.
+
+ .. code-block:: yaml
+
+ branches:
+ - name: stable/fraser
+ location:
+ apex: <git-sha1>
+ - name: stable/fraser
+ location:
+ apex-puppet-tripleo: <git-sha1>
+
+#. Git add, commit, and git-review the changes. A job will be triggered
+ to verify the commit exists on the branch, and the yaml file follows
+ the scheme listed in ``releases/schema.yaml``
+
+#. Once the commit has been reviewed and merged by Releng, a job will
+ be triggered to create the stable branch Jenkins jobs under
+ ``jjb/``.
+
+
+Stable Release Tagging
+----------------------
+
+TBD
+
+Release File Fields
+-------------------
+
+The following is a description of fields in the Release file, which are
+verified by the scheme file at ``releases/schema.yaml``
+
+project
+ Project team in charge of the release.
+
+release-model
+ Release model the project follows.
+
+ One of: stable, non-release
+
+project-type
+ Classification of project within OPNFV.
+
+ One of: installer, feature, testing, tools, infra
+
+upstream
+ (Optional) Upstream OpenStack project assocated with this project.
+
+releases
+ List of released versions for the project.
+
+ version
+ Version of the release, must be in the format ``opnfv-X.Y.Z``.
+
+ location
+ Combination of repository and git hash to locate the release
+ version.
+
+ Example::
+
+ opnfv-project: f15d50c2009f1f865ac6f4171347940313727547
+
+branches
+ List of stable branches for projects following the ``stable`` release-model.
+
+ name
+ Stable branch name. Must start with the string ``stable/``
+
+ location
+ Same syntax as ``location`` under ``releases``
+
+release-notes
+ Link to release notes for the projects per-release.
+
+
+Scripts
+-------
+
+* ``create_branch.py -f <RELEASE_FILE>``
+
+ Create branches in Gerrit listed in the release file.
+
+ Must be ran from the root directory of the releng repository as the
+ release name is extracted from the subdirectory under ``releases/``
+
+ The Gerrit server can be changed by creating a ``~/releases.cfg``
+ file with the following content::
+
+ [gerrit]
+ url=http://gerrit.example.com
+
+ This will override the default configuration of using the OPNFV
+ Gerrit server at https://gerrit.opnfv.org, and is primarily used for
+ testing.
+
+* ``create_jobs.py -f <RELEASE_FILE>``
+
+ Modifies the jenkins job files for a project to add the stable branch
+ stream. Assumes the jenkins jobs are found in the releng repository
+ under ``jjb/<project>/``
+
+* ``verify_schema -s <SCHEMA_FILE> -y <YAML_FILE>``
+
+ Verifies the yaml file matches the specified jsonschema formatted
+ file. Used to verify the release files under ``releases/``
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+set -xe
+
+# Configure the git user/email as we'll be pushing up changes
+git config user.name "jenkins-ci"
+git config user.email "jenkins-opnfv-ci@opnfv.org"
+
+# Ensure we are able to generate Commit-IDs for new patchsets
+curl -kLo .git/hooks/commit-msg https://gerrit.opnfv.org/gerrit/tools/hooks/commit-msg
+chmod +x .git/hooks/commit-msg
+
+# Activate virtualenv, supressing shellcheck warning
+# shellcheck source=/dev/null
+. $WORKSPACE/venv/bin/activate
+pip install -r releases/scripts/requirements.txt
+
+STREAM=${STREAM:-'nostream'}
+RELEASE_FILES=$(git diff HEAD^1 --name-only -- "releases/$STREAM")
+
+for release_file in $RELEASE_FILES; do
+ python releases/scripts/create_branch.py -f $release_file
+ python releases/scripts/create_jobs.py -f $release_file
+ NEW_FILES=$(git status --porcelain --untracked=no | cut -c4-)
+ if [ -n "$NEW_FILES" ]; then
+ git add $NEW_FILES
+ git commit -m "Create Stable Branch Jobs for $(basename $release_file .yaml)"
+ git push origin HEAD:refs/for/master
+ fi
+done
--- /dev/null
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+---
+- project:
+ name: releng-release-jobs
+
+ stream:
+ - fraser
+
+ jobs:
+ - 'releng-release-{stream}-verify'
+ - 'releng-release-{stream}-merge'
+
+ project: 'releng'
+
+- job-template:
+ name: 'releng-release-{stream}-verify'
+
+ parameters:
+ - stream-parameter:
+ stream: '{stream}'
+ - project-parameter:
+ project: '{project}'
+ branch: 'master'
+
+ scm:
+ - git-scm-gerrit
+
+ triggers:
+ - gerrit:
+ server-name: 'gerrit.opnfv.org'
+ trigger-on:
+ - patchset-created-event:
+ exclude-drafts: 'false'
+ exclude-trivial-rebase: 'false'
+ exclude-no-code-change: 'false'
+ - comment-added-contains-event:
+ comment-contains-value: 'recheck'
+ - comment-added-contains-event:
+ comment-contains-value: 'reverify'
+ projects:
+ - project-compare-type: 'ANT'
+ project-pattern: 'releng'
+ branches:
+ - branch-compare-type: 'ANT'
+ branch-pattern: '**/master'
+ file-paths:
+ - compare-type: ANT
+ pattern: 'releases/{stream}/**'
+ - compare-type: ANT
+ pattern: 'releases/schema.yaml'
+ - compare-type: ANT
+ pattern: 'releases/scripts/verify_schema.py'
+
+ builders:
+ - create-virtualenv
+ - shell:
+ !include-raw-escape: releng-release-verify.sh
+
+ publishers:
+ - email-jenkins-admins-on-failure
+
+- job-template:
+ name: 'releng-release-{stream}-merge'
+
+ parameters:
+ - node:
+ name: SLAVE_NAME
+ description: 'Only run merge job on build1'
+ default-slaves:
+ - lf-build1
+ allowed-multiselect: false
+ ignore-offline-nodes: true
+ - stream-parameter:
+ stream: '{stream}'
+ - project-parameter:
+ project: '{project}'
+ branch: 'master'
+
+ scm:
+ - git-scm-gerrit
+
+ triggers:
+ - gerrit-trigger-change-merged:
+ project: '{project}'
+ branch: 'master'
+ files: 'releases/**'
+
+ builders:
+ - create-virtualenv
+ - shell:
+ !include-raw-escape: releng-release-create-branch.sh
+
+ publishers:
+ - email-jenkins-admins-on-failure
+
+- parameter:
+ name: stream-parameter
+ parameters:
+ - string:
+ name: STREAM
+ default: '{stream}'
+ description: "OPNFV Stable Stream"
+
+- builder:
+ name: create-virtualenv
+ builders:
+ - shell: |
+ #!/bin/bash
+ sudo pip install virtualenv
+ virtualenv $WORKSPACE/venv
+ . $WORKSPACE/venv/bin/activate
+ pip install --upgrade pip
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+set -xe
+
+# Activate virtualenv, supressing shellcheck warning
+# shellcheck source=/dev/null
+. $WORKSPACE/venv/bin/activate
+pip install -r releases/scripts/requirements.txt
+
+STREAM=${STREAM:-'nostream'}
+RELEASE_FILES=$(git diff HEAD^1 --name-only -- "releases/$STREAM")
+
+# TODO: The create_branch.py should be refactored so it can be used here
+# to verify the commit exists that is being added, along with
+# jjb/<project>
+for release_file in $RELEASE_FILES; do
+ python releases/scripts/verify_schema.py \
+ -s releases/schema.yaml \
+ -y $release_file
+done
--- /dev/null
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+---
+project: apex
+project-type: installer
+release-model: stable
+upstream: https://wiki.openstack.org/wiki/TripleO
+
+releases:
+ - version: opnfv-5.0.0
+ location:
+ apex: 2f1c99daeee9cf0e89a8e833e034e7a5979ae894
+ - version: opnfv-5.1.0
+ location:
+ apex: f15d50c2009f1f865ac6f4171347940313727547
+
+branches:
+ - name: stable/euphrates
+ location:
+ apex: f27da77b87837e025907f689890b413c8f183c59
+ - name: stable/euphrates
+ location:
+ apex-tripleo-heat-templates: 676db53c4423693441112640cf362e93931161ae
+ - name: stable/euphrates
+ location:
+ apex-puppet-tripleo: 14bc31f54ea943547a3319b479ea7b8cd9661e85
+ - name: stable/euphrates
+ location:
+ apex-os-net-config: a6c3f2a2c853ca489cceff959a52d7f75bf4ffe0
+
+release-notes: http://docs.opnfv.org/en/stable-euphrates/submodules/apex/docs/release/release-notes/release-notes.html
--- /dev/null
+---
+project: compass4nfv
+project-type: installer
+release-model: stable
+
+branches:
+ - name: stable/euphrates
+ location:
+ compass4nfv: 435cd3756a833db0515eb70c1d8ec4adca90950f
--- /dev/null
+##############################################################################
+# Copyright (c) 2018 Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+---
+$schema: 'http://json-schema.org/schema#'
+$id: 'https://github.com/opnfv/releng/blob/master/releases/schema.yaml'
+
+additionalProperties: false
+
+required:
+ - 'project'
+ - 'project-type'
+
+properties:
+ project:
+ type: 'string'
+ release-model:
+ type: 'string'
+ enum: ['stable', 'non-release']
+ project-type:
+ type: 'string'
+ enum: ['installer', 'testing', 'feature', 'tools', 'infra']
+ upstream:
+ type: 'string'
+ releases:
+ type: 'array'
+ items:
+ type: 'object'
+ properties:
+ version:
+ type: 'string'
+ # Matches semantic versioning (X.Y.Z)
+ pattern: '^opnfv-([0-9]+\.){2}[0-9]+$'
+ location:
+ type: 'object'
+ required: ['version', 'location']
+ additionalProperties: false
+ branches:
+ type: 'array'
+ items:
+ type: 'object'
+ properties:
+ name:
+ type: 'string'
+ pattern: '^stable/[a-z]+$'
+ location:
+ type: 'object'
+ required: ['name', 'location']
+ additionalProperties: false
+ release-notes:
+ type: 'string'
+ format: 'uri'
--- /dev/null
+#!/usr/bin/env python2
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+"""
+Create Gerrit Branchs
+"""
+
+import argparse
+import ConfigParser
+import logging
+import os
+import yaml
+
+from requests.compat import quote
+from requests.exceptions import RequestException
+
+from pygerrit2.rest import GerritRestAPI
+from pygerrit2.rest.auth import HTTPDigestAuthFromNetrc, HTTPBasicAuthFromNetrc
+
+
+logging.basicConfig(level=logging.INFO)
+
+
+def quote_branch(arguments):
+ """
+ Quote is used here to escape the '/' in branch name. By
+ default '/' is listed in 'safe' characters which aren't escaped.
+ quote is not used in the data of the PUT request, as quoting for
+ arguments is handled by the request library
+ """
+ new_args = arguments.copy()
+ new_args['branch'] = quote(new_args['branch'], '')
+ return new_args
+
+
+def create_branch(api, arguments):
+ """
+ Create a branch using the Gerrit REST API
+ """
+ logger = logging.getLogger(__file__)
+
+ branch_data = """
+ {
+ "ref": "%(branch)s"
+ "revision": "%(commit)s"
+ }""" % arguments
+
+ # First verify the commit exists, otherwise the branch will be
+ # created at HEAD
+ try:
+ request = api.get("/projects/%(project)s/commits/%(commit)s" %
+ arguments)
+ logger.debug(request)
+ logger.debug("Commit exists: %(commit)s", arguments)
+ except RequestException as err:
+ if hasattr(err, 'response') and err.response.status_code in [404]:
+ logger.warn("Commit %(commit)s for %(project)s:%(branch)s does"
+ " not exist. Not creating branch.", arguments)
+ else:
+ logger.error("Error: %s", str(err))
+ # Skip trying to create the branch
+ return
+
+ # Try to create the branch and let us know if it already exist.
+ try:
+ request = api.put("/projects/%(project)s/branches/%(branch)s" %
+ quote_branch(arguments), branch_data)
+ logger.info("Branch %(branch)s for %(project)s successfully created",
+ arguments)
+ except RequestException as err:
+ if hasattr(err, 'response') and err.response.status_code in [412, 409]:
+ logger.info("Branch %(branch)s already created for %(project)s",
+ arguments)
+ else:
+ logger.error("Error: %s", str(err))
+
+
+def main():
+ """Given a yamlfile that follows the release syntax, create branches
+ in Gerrit listed under branches"""
+
+ config = ConfigParser.ConfigParser()
+ config.read(os.path.join(os.path.abspath(os.path.dirname(__file__)),
+ 'defaults.cfg'))
+ config.read([os.path.expanduser('~/releases.cfg'), 'releases.cfg'])
+
+ gerrit_url = config.get('gerrit', 'url')
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--file', '-f',
+ type=argparse.FileType('r'),
+ required=True)
+ parser.add_argument('--basicauth', '-b', action='store_true')
+ args = parser.parse_args()
+
+ GerritAuth = HTTPDigestAuthFromNetrc
+ if args.basicauth:
+ GerritAuth = HTTPBasicAuthFromNetrc
+
+ try:
+ auth = GerritAuth(url=gerrit_url)
+ except ValueError, err:
+ logging.error("%s for %s", err, gerrit_url)
+ quit(1)
+ restapi = GerritRestAPI(url=gerrit_url, auth=auth)
+
+ project = yaml.safe_load(args.file)
+
+ create_branches(restapi, project)
+
+
+def create_branches(restapi, project):
+ """Create branches for a specific project defined in the release
+ file"""
+
+ branches = []
+ for branch in project['branches']:
+ repo, ref = next(iter(branch['location'].items()))
+ branches.append({
+ 'project': repo,
+ 'branch': branch['name'],
+ 'commit': ref
+ })
+
+ for branch in branches:
+ create_branch(restapi, branch)
+
+
+if __name__ == "__main__":
+ main()
--- /dev/null
+#!/usr/bin/env python2
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+"""
+Create Gerrit Branches
+"""
+
+import argparse
+import logging
+import os
+import re
+import yaml
+import subprocess
+
+# import ruamel
+from ruamel.yaml import YAML
+
+
+logging.basicConfig(level=logging.INFO)
+
+
+def has_string(filepath, string):
+ """
+ Return True if the given filepath contains the regex string
+ """
+ with open(filepath) as yaml_file:
+ for line in yaml_file:
+ if string.search(line):
+ return True
+ return False
+
+
+def jjb_files(project, release):
+ """
+ Return sets of YAML file names that contain 'stream' for a given
+ project, and file that already contain the stream.
+ """
+ files, skipped = set(), set()
+ file_ending = re.compile(r'ya?ml$')
+ search_string = re.compile(r'^\s+stream:')
+ release_string = re.compile(r'- %s:' % release)
+ jjb_path = os.path.join('jjb', project)
+
+ if not os.path.isdir(jjb_path):
+ logging.warn("JJB directory does not exist at %s, skipping job "
+ "creation", jjb_path)
+ return (files, skipped)
+
+ for file_name in os.listdir(jjb_path):
+ file_path = os.path.join(jjb_path, file_name)
+ if os.path.isfile(file_path) and file_ending.search(file_path):
+ if has_string(file_path, release_string):
+ skipped.add(file_path)
+ elif has_string(file_path, search_string):
+ files.add(file_path)
+ return (files, skipped)
+
+
+def main():
+ """
+ Create Jenkins Jobs for stable branches in Release File
+ """
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--file', '-f',
+ type=argparse.FileType('r'),
+ required=True)
+ args = parser.parse_args()
+
+ project_yaml = yaml.safe_load(args.file)
+
+ # Get the release name from the file path
+ release = os.path.split(os.path.dirname(args.file.name))[1]
+
+ create_jobs(release, project_yaml)
+
+
+def create_jobs(release, project_yaml):
+ """Add YAML to JJB files for release stream"""
+ logger = logging.getLogger(__file__)
+
+ # We assume here project keep their subrepo jobs under the part
+ # project name. Otherwise we'll have to look for jjb/<repo> for each
+ # branch listed.
+ project, _ = next(iter(project_yaml['branches'][0]['location'].items()))
+
+ yaml_parser = YAML()
+ yaml_parser.preserve_quotes = True
+ yaml_parser.explicit_start = True
+ # yaml_parser.indent(mapping=4, sequence=0, offset=0)
+ # These are some esoteric values that produce indentation matching our jjb
+ # configs
+ # yaml_parser.indent(mapping=3, sequence=3, offset=2)
+ # yaml_parser.indent(sequence=4, offset=2)
+ yaml_parser.indent(mapping=2, sequence=4, offset=2)
+
+ (job_files, skipped_files) = jjb_files(project, release)
+
+ if skipped_files:
+ logger.info("Jobs already exists for %s in files: %s",
+ project, ', '.join(skipped_files))
+ # Exit if there are not jobs to create
+ if not job_files:
+ return
+ logger.info("Creating Jenkins Jobs for %s in files: %s",
+ project, ', '.join(job_files))
+
+ stable_branch_stream = """\
+ %s:
+ branch: 'stable/{stream}'
+ gs-pathname: '/{stream}'
+ disabled: false
+ """ % release
+
+ stable_branch_yaml = yaml_parser.load(stable_branch_stream)
+ stable_branch_yaml[release].yaml_set_anchor(release, always_dump=True)
+
+ for job_file in job_files:
+ yaml_jjb = yaml_parser.load(open(job_file))
+ if 'stream' not in yaml_jjb[0]['project']:
+ continue
+
+ # TODO: Some JJB files don't have 'stream'
+ project_config = yaml_jjb[0]['project']['stream']
+ # There is an odd issue where just appending adds a newline before the
+ # branch config, so we append (presumably after master) instead.
+ project_config.insert(1, stable_branch_yaml)
+
+ # NOTE: In the future, we may need to override one or multiple of the
+ # following ruamal Emitter methods:
+ # * ruamel.yaml.emitter.Emitter.expect_block_sequence_item
+ # * ruamel.yaml.emitter.Emitter.write_indent
+ # To hopefully replace the need to shell out to sed...
+ yaml_parser.dump(yaml_jjb, open(job_file, 'w'))
+ args = ['sed', '-i', 's/^ //', job_file]
+ subprocess.Popen(args, stdout=subprocess.PIPE, shell=False)
+
+
+if __name__ == "__main__":
+ main()
--- /dev/null
+[gerrit]
+url=https://gerrit.opnfv.org/
--- /dev/null
+pygerrit2 < 2.1.0
+PyYAML < 4.0
+jsonschema < 2.7.0
+rfc3987
+ruamel.yaml
--- /dev/null
+#!/usr/bin/env python2
+# SPDX-License-Identifier: Apache-2.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Apache License, Version 2.0
+# which accompanies this distribution, and is available at
+# http://www.apache.org/licenses/LICENSE-2.0
+##############################################################################
+"""
+Verify YAML Schema
+"""
+import argparse
+import logging
+import jsonschema
+import yaml
+
+LOADER = yaml.CSafeLoader if yaml.__with_libyaml__ else yaml.SafeLoader
+
+
+def main():
+ """
+ Parse arguments and verify YAML
+ """
+ logging.basicConfig(level=logging.INFO)
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--yaml', '-y', type=str, required=True)
+ parser.add_argument('--schema', '-s', type=str, required=True)
+
+ args = parser.parse_args()
+
+ with open(args.yaml) as _:
+ yaml_file = yaml.load(_, Loader=LOADER)
+
+ with open(args.schema) as _:
+ schema_file = yaml.load(_, Loader=LOADER)
+
+ # Load the schema
+ validation = jsonschema.Draft4Validator(
+ schema_file,
+ format_checker=jsonschema.FormatChecker()
+ )
+
+ # Look for errors
+ errors = 0
+ for error in validation.iter_errors(yaml_file):
+ errors += 1
+ logging.error(error)
+ if errors > 0:
+ raise RuntimeError("%d issues invalidate the release schema" % errors)
+
+
+if __name__ == "__main__":
+ main()
Code Review / releng.git / commitdiff