Add --detailed-exitcodes when running puppet via ansible

puppet run on never fails, even when it should, since we moved
to the ansible way of applying it. The reason is the current following code:

    - name: Run puppet host configuration for step {{step}}
      command: >-
        puppet apply
        --modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules
        --logdest syslog --logdest console --color=false
        /var/lib/tripleo-config/puppet_step_config.pp

The above is missing the --detailed-exitcodes switch and so puppet will never
really error out on us and the deployment will keep on running all the
steps even though a previous puppet manifest might have failed. This
cause extra hard-to-debug failures.

Initially the issue was observed on the puppet host runs, but this
parameter is missing also from docker-puppet.py, so let's add it there
as well as it makes sense to return proper error codes whenever we call
puppet.

Besides this being a good idea in general, we actually *have* to do it
because puppet does not fail correctly without this option due to the
following puppet bug:
https://tickets.puppetlabs.com/browse/PUP-2754

Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
Change-Id: Ie9df4f520645404560a9635fb66e3af42b966f54
Closes-Bug: #1723163
(cherry picked from commit 11e599d116cfbf7df4dcd0e7670c3405a4224c1a)

diff --git a/common/deploy-steps-tasks.yaml b/common/deploy-steps-tasks.yaml
index 785095b..79a8bc8 100644 (file)
--- a/common/deploy-steps-tasks.yaml
+++ b/common/deploy-steps-tasks.yaml
@@ -14,16 +14,17 @@
       command: >-
         puppet apply {{ host_puppet_config_debug|default('') }}
         --modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules
+        --detailed-exitcodes
         --logdest syslog --logdest console --color=false
         /var/lib/tripleo-config/puppet_step_config.pp
-      changed_when: false
+      changed_when: outputs.rc == 2
       check_mode: no
       register: outputs
       failed_when: false
       no_log: true
     - debug: var=(outputs.stderr|default('')).split('\n')|union(outputs.stdout_lines|default([]))
       when: outputs is defined
-      failed_when: outputs|failed
+      failed_when: outputs.rc not in [0, 2]
     ######################################
     # Generate config via docker-puppet.py
     ######################################

diff --git a/docker/docker-puppet.py b/docker/docker-puppet.py
index 533ed07..d12e055 100755 (executable)
--- a/docker/docker-puppet.py
+++ b/docker/docker-puppet.py
@@ -225,8 +225,14 @@ def mp_puppet_config((config_volume, puppet_tags, manifest, config_image, volume
         touch /tmp/the_origin_of_time
         sync
 
+        set +e
         FACTER_hostname=$HOSTNAME FACTER_uuid=docker /usr/bin/puppet apply \
-        --color=false --logdest syslog --logdest console $TAGS /etc/config.pp
+        --detailed-exitcodes --color=false --logdest syslog --logdest console $TAGS /etc/config.pp
+        rc=$?
+        set -e
+        if [ $rc -ne 2 -a $rc -ne 0 ]; then
+            exit $rc
+        fi
 
         # Disables archiving
         if [ -z "$NO_ARCHIVE" ]; then
@@ -307,7 +313,9 @@ def mp_puppet_config((config_volume, puppet_tags, manifest, config_image, volume
         subproc = subprocess.Popen(dcmd, stdout=subprocess.PIPE,
                                    stderr=subprocess.PIPE, env=env)
         cmd_stdout, cmd_stderr = subproc.communicate()
-        if subproc.returncode != 0:
+        # puppet with --detailed-exitcodes will return 0 for success and no changes
+        # and 2 for success and resource changes. Other numbers are failures
+        if subproc.returncode not in [0, 2]:
             log.error('Failed running docker-puppet.py for %s' % config_volume)
             if cmd_stdout:
                 log.error(cmd_stdout)
@@ -355,7 +363,7 @@ returncodes = list(p.map(mp_puppet_config, process_map))
 config_volumes = [pm[0] for pm in process_map]
 success = True
 for returncode, config_volume in zip(returncodes, config_volumes):
-    if returncode != 0:
+    if returncode not in [0, 2]:
         log.error('ERROR configuring %s' % config_volume)
         success = False