Prove that clustered devices have identical applications and are patched correctly
The Customer
A financial company running multiple trading applications on 40 discrete clusters (pairs of devices) needs to prove that each pair has identical applications and that patches get rolled out appropriately.
The Situation
This company runs 40 clusters in transparent failover pairs, where one of the pair is up, doing the work, carrying the load, while the other is quiescent, waiting either for failure or for the switchover. The two sides swap every week. When patching, they always patch the quiescent side, so they apply patches roughly every week or so. If it’s a critical patch, of course, it moves much faster.
The Challenge
The company faces the same issue everyone using Microsoft has: Patch Tuesday, so named because Tuesday is the day Microsoft bundles up all the patches and ships them out. The challenge when those patches arrive is how to quality test them and how to deploy them. Typically what people do is: apply it to production or put it in a lab. This customer updates its development machines with the patch and lets it “soak” for 10 days. If it’s a critical patch, of course, they fast-track it. The 10-day soak and the 7-day switchover in the clusters do not jibe, so it is not always easy to figure out which server has which patches.
The Old Way: Manual and time-consuming
It is important to be sure the patch has been applied, but the customer couldn’t be sure which side of the cluster pairs was operating at which level without looking at the clusters manually – a high-human-cost look. An admin had to dig through 80 to 90 machines until they find the patch. When managing an environment as dynamic as this customer’s, the patch job compounded an already difficult problem of simply managing the servers.
The company also needed certainty that the environment they were running in was not affected by shifts in the underlying platform, and that they could identify any problems related to the applications they were adding.
The SignaCert Solution #1: Automate Patch Verification
The company first wanted to know what patch levels they were at on both sides of the cluster. SignaCert’s Enterprise Trust Server (ETS) was installed and within hours it easily validated that each of the devices in the cluster had the correct patch. Now, signatures for patches are regularly published to the ETS appliance, which uses them to identify patch files on the devices. The SignaCert client monitors each device in the pair for the patches to identify whether or not they have the correct set of patches. This allows them to effectively manage each device in the cluster, even if they are not identical matches.
The SignaCert Solution #2: Verify Devices are Identical
SignaCert provided the company an important second solution: verifying that the clusters are consistent, and if they’re not, showing exactly why. ETS shows whether the inconsistency is a result of a known, expected change or the result of something inadvertent. That way, if anything is inconsistent, the customer knows immediately whether that was the result of something expected or not.
Proving that devices have identical applications requires capturing a snapshot for one of the devices in the cluster and comparing it to the other device to prove they are identical or to identify deviations. When specific deviations are identified, the customer can very rapidly apply the appropriate remediation to bring the pairs into synch. Anecdotally, on initial deployment, the customer was confident that paired devices were identical, but the ETS appliance quickly identified deviations in software versions between them.
Results
The company is extremely happy to have gained the assurance that both pairs of clusters are always patched correctly and that the paired devices have identical applications.
For customers who have many more servers in their clusters, it may be useful to look at the case study for Grids.