Operations Manager 2012 R2 – Post-Disaster Recovery

Recently we performed an OS upgrade on our OpsMgr environment. We originally were running on Server 2008 R2, but we wanted to flatten our System Center environment to just one OS platform (Server 2012 R2).

Officially, Microsoft supports two methods of upgrading the base OS when running OpsMgr 2012 R2–you can do an in-place upgrade of Windows (very quick), or you can stand up another machine (of the same name) and perform a disaster recovery on the server. In my scenario, while the thought of an in-place upgrade sounded alluring, I personally still have this tiny speck of hesitance whenever I hear the words “Windows Server” and “in-place upgrade”–so we went with standing up a new machine.

The documentation for this is quite good, and I was happily surprised to see how quickly one could bring back an OpsMgr environment (when the database hasn’t gone anywhere). After running the recovery setup, re-entering the RunAs passwords, patching to UR5, and rebooting the server, I thought everything was perfect. The server was up, the Management Server status was Healthy, and there were no alerts.

..wait, no alerts? My environment is quiet. Too quiet.

After rebooting a server to provoke some OpsMgr updates (that server needed to update anyway), I still saw no alerts. OK, now it’s time to dig into the event viewer.

I saw multiple Event ID 21023 alerts:

OpsMgr has no configuration for management group MGMT_GROUP and is requesting new configuration from the Configuration Service.

I also saw occasional Event ID 29120 alerts:

OpsMgr Management Configuration Service failed to process configuration request (Xml configuration file or management pack request) due to the following exception
Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.HealthServicePublicKeyNotRegisteredException: Padding is invalid and cannot be removed.

Some quick searching suggested that I check the credentials for my RunAs accounts, or delete the Health Service cache. I tried both of these with no success.

At least I thought I did.

When they say “update the RunAs account information,” it means update BOTH the username and the password. Just updating the password isn’t enough. Even if you open that RunAs window and it shows the user account listed, just paste the account name on top of it. After entering both the username and the password, OpsMgr started processing alerts almost immediately and my Inbox was full of alerts.

So now we can ponder: when is a user account not a user account? 🙂

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.