Wait for MongoDB connections before creating replset

Sometimes the mongodb_replset resource fails with:

Could not evaluate: Can't find master host for replicaset tripleo.

This issue is intermittent so the fix cannot be perfectly verified, but
the assumption is that if we wait for MongoDB to be reachable on all
nodes, it will assure that the members will appear to the puppet module
as alive when creating or verifying the replset. If the validation
fails, it should help us uncover which of the members was causing
trouble.

Change-Id: I0bcd0d063a7a766483426fdd5ea81cbe1dfaa348
Closes-Bug: #1624420

diff --git a/manifests/profile/base/database/mongodb.pp b/manifests/profile/base/database/mongodb.pp
index 1a19bb6..8967f5b 100644 (file)
--- a/manifests/profile/base/database/mongodb.pp
+++ b/manifests/profile/base/database/mongodb.pp
@@ -44,6 +44,13 @@ class tripleo::profile::base::database::mongodb (
     include ::tripleo::profile::base::database::mongodbcommon
 
     if $bootstrap_node == $::hostname {
+      # make sure we can connect to all servers before forming the replset
+      tripleo::profile::pacemaker::database::mongodbvalidator {
+        $tripleo::profile::base::database::mongodbcommon::mongodb_node_ips :
+        port    => $tripleo::profile::base::database::mongodbcommon::port,
+        require => Service['mongodb'],
+        before  => Mongodb_replset[$mongodb_replset],
+      }
       mongodb_replset { $mongodb_replset :
         members => $tripleo::profile::base::database::mongodbcommon::mongo_node_ips_with_port_nobr,
       }