Workaround VCP minions timeout post pkg.upgrade

pkg.upgrade was enabled for all salt minions, including VCP VMs,
which take longer to perform the operation, probably due to an
older set of packages in the Ubuntu disk image we use.
One way to work around this is to switch to UCA Xenial image, and
let Salt pre-provising salt minion on it, but that adds deploy
time delay and has caused issues in the past (should be ok now).

Alternatively, we can retry the pkg.upgrade until all minions
respond, before moving on with the state execution. This prevents
silently skipping the next salt calls (e.g. installing keepalived).

Note that the issue did not manifest for OVD-DPDK, where after
pkg.upgrade, DPDK is installed, giving VCP VMs enough time to return.

While at it, retry 'salt.control' state apply too (non-critical,
but it fails every once in a while).

Fixes: 87310fb

Change-Id: I97acc2b23206a55d72f7e6583ca42127fdbacc16
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
(cherry picked from commit a6feb158fa5bc16b8d496c785c461c73ca715322)

diff --git a/mcp/config/states/maas b/mcp/config/states/maas
index e1ffa87..de9d699 100755 (executable)
--- a/mcp/config/states/maas
+++ b/mcp/config/states/maas
@@ -91,7 +91,8 @@ salt -C 'kvm*' state.sls libvirt
 
 salt -C '* and not cfg01* and not mas01*' state.apply salt
 salt -C 'kvm*' saltutil.sync_all
-salt -C 'kvm*' state.sls salt.control
+wait_for 10 "! salt -C 'kvm*' state.sls salt.control | " \
+  "tee /dev/stderr | fgrep -q 'Not connected'"
 
 vcp_nodes=$(salt --out yaml 'kvm01*' pillar.get salt:control:cluster:internal:node | \
             awk '/\s+\w+:$/ {gsub(/:$/, "*"); print $1}')
@@ -115,4 +116,5 @@ wait_for 10 "salt -C 'E@^(?!cfg01|mas01|kvm|cmp00).*' ssh.set_auth_key ${SUDO_US
   $(awk 'NR==1{print $2}' "$(eval echo "~${SUDO_USER}/.ssh/authorized_keys")")"
 
 # Get the latest packages
-salt '*' pkg.upgrade refresh=False
+wait_for 10 "! salt '*' pkg.upgrade refresh=False | " \
+  "tee /dev/stderr | fgrep -q 'Not connected'"