Retry ceilometer-upgrade

When the ceilometer-upgrade command is run in step5, it talks to gnocchi
and keystone on all the controllers. Since these other nodes might have
httpd restarted mid-upgrade we should retry if we get a failure.

Change-Id: I874cf9c34b41d055a258704dabe9150eab0f7968
Closes-Bug: #1703444

diff --git a/manifests/profile/base/ceilometer/upgrade.pp b/manifests/profile/base/ceilometer/upgrade.pp
index f192b48..0031b79 100644 (file)
--- a/manifests/profile/base/ceilometer/upgrade.pp
+++ b/manifests/profile/base/ceilometer/upgrade.pp
@@ -42,8 +42,16 @@ class tripleo::profile::base::ceilometer::upgrade (
   # are created safely.
   if $step >= 5 and $sync_db {
     exec {'ceilometer-db-upgrade':
-      command => 'ceilometer-upgrade --skip-metering-database',
-      path    => ['/usr/bin', '/usr/sbin'],
+      command   => 'ceilometer-upgrade --skip-metering-database',
+      path      => ['/usr/bin', '/usr/sbin'],
+      # LP#1703444 - When this runs, it talks to gnocchi on all controllers
+      # which then reaches out to keystone via haproxy. Since the deployment
+      # may restart httpd on these other nodes it can result in an intermittent
+      # 503 which fails this command. We should retry the upgrade in case of
+      # error since we cannot ensure that there might not be some other deploy
+      # process running on the other nodes.
+      try_sleep => 5,
+      tries     => 10
     }
   }
 }