Saturday, October 22, 2016

Upgrading To SCOM 2016 GA – 02 – Let’s Upgrade!

In this posting I’ll write about my upgrade experiences of one of my SCOM 2012 R2 UR#11 environments to SCOM 2016 GA (SCOM 2016 + UR#01). The SCOM 2012 R2 environment is rather small, but still representative for many SCOM 2012 R2 environments, since it exists out of two SCOM Management Servers and a few SCOM Agents.

Because the upgrade process of a SCOM Agent to SCOM 2016 is the same, no matter the amount of them, IMHO the upgrade of my SCOM 2012 R2 UR#11 environment is applicable to many SCOM environments. The only thing lacking here is at least one SCOM Gateway Server. Since a Gateway Servers is in essence a SCOM Agent with some additional features, upgrading such a server is a walk in the park, especially compared to upgrading a SCOM Management Server.

This is my test lab:

  1. 2x SCOM Management Servers;
  2. No SCOM Gateway Servers;
  3. 1x dedicated SQL instance, hosting both SCOM SQL databases;
  4. 4x SCOM Agent.

Side note: Yes, it’s a small environment but it runs locally on my laptop, besides a SCCM environment providing FEP functionality for all VMs, an additional SQL server and a DC. So I think it’s still quite something for an average notebook Smile.

All servers involved run Windows Server 2012 R2. SQL Server 2012 SP1 x64 is used for the SQL instance hosting the SCOM SQL databases.

There is much to tell, so let’s start.

A 01: Pre-Upgrade Tasks
NEVER EVER skip the Pre-Upgrade Tasks! Preparation is key, otherwise the upgrade is most likely to fail which is bad. And something else as well:

Before you start BACKUP! ALL SCOM Management Servers to be more specific. When on VMs, make snapshots or clones. And for the SCOM SQL databases: make VALID backups!

Microsoft recently published a TechNet article all about the Pre-Upgrade Tasks. So I won’t repeat them but only highlight some steps here. STILL TAKE CARE TO COVER ALL STEPS AS MENTIONED IN THIS TECHNET ARTICLE!!!

Steps I want to give additional attention are highlighted in yellow:

  • Step 2: Clean the Database (ETL Table)
    This is serious business. In any real life SCOM environment this table may become quite big. In every day life one doesn’t notice this. But when upgrading this table will become the culprit of slow (ever lasting) upgrades and may eventually fail. So RUN the query in order to clean up that ETL table.

  • Step 7: Stop & disable all related SCOM services on the Management Servers when not being upgraded
    This step makes perfect sense. Simply because when running the upgrade on a given SCOM Management Server, the underlying SCOM databases are also ‘touched’.

    Even more when it’s the first SCOM Management Server to be upgraded. In such a case the SCOM databases are upgraded as well, and when done, a flag is set so when the next SCOM Management Server is upgraded, the SCOM databases aren’t upgraded AGAIN…

    However, even when the SCOM databases are already upgraded, and another SCOM Management Server is upgraded, the SCOM databases are touched. In cases like that it’s better not having other SCOM Management Server running write/read actions on those very same databases in the same time.

    Therefore, only have the related SCOM services in a RUNNING state on the SCOM Management Server which is being upgraded, and on all other SCOM Management Servers, stopped & disabled.

  • Step 8: OpsMgr SQL DB must have 50% or more free space
    This is very important. Check it in order to know for sure the OpsMgr SQL database has ENOUGH free space to run the upgrade.

  • Step 11: Disable the Operations Manager website in IIS
    Since SCOM 2016 is in the process to ditch the Silverlight dependency in the SCOM Web Console, the SCOM 2012 R2 UR#11 Web Console gets an overhaul as well. Therefore it’s better to disable this website in IIS, so the upgrade process won’t find any locked handles or processes on it. Also enhancing the change of success of the upgrade.

A 02: Pre-Upgrade Steps I like to add
In addition to the earlier mentioned TechNet article there are a few additional pre-upgrade steps I would like to add as well here.

  1. Install the SCOM 2016 Console prereqs on the SCOM Management Servers where the SCOM 2012 R2 UR#11 Console is also installed.

    Even though the SCOM 2016 installation/upgrade wizard checks for these prereqs:
    It’s better to avoid those Alerts and to install these two prereqs yourself in advance. So install Microsoft CLR Types for SQL Server 2014 and the SQL 2015 Report Viewer Controls in advance.

  2. DOUBLE check the environment
    Meaning: Is your current SCOM 2012 R2 UR#11 environment upgradable? Do the underlying Windows Server OS’s AND SQL instances support SCOM 2016?

    Use this TechNet article to verify it. When some components aren’t supported in SCOM 2016, address those issues first before upgrading to SCOM 2016.

  3. SCOM 2016 UR#01
    As stated earlier, UR#01 for SCOM 2016 brings it on GA level. So there is NO doubt whether or not to install those bits as well. SIMPLY INSTALL IT! So download the required bits and when your SCOM 2012 R2 UR#11 environment is upgraded to SCOM 2016, install UR#01 right after it.

    In order to save time and work, you could skip the SCOM Agents. Simply upgrade the SCOM 2012 R2 UR#11 Agents to SCOM 2016 UR#1 right away, since the SCOM 2012 R2 UR#11 Agents communicate with the SCOM 2016 UR#01 environment as well, thus buying you more time Smile.

    So download the SCOM 2016 UR#01 bits before starting the upgrade to SCOM 2016 RTM.

    Also good to know: SCOM 2016 UR#01 only touches the SCOM Management Server, Console and Web Console. Not the SCOM Gateway Servers. The SCOM Agent is touched as well since the Agent staging folders on the SCOM Management Servers do get a MSP update package as well (KB3190029-amd64-Agent.msp for x64 and KB3190029-i386-Agent.msp for x86).

  4. SCOM SDK Account
    When upgrading SCOM 2016 will ask for the SCOM SDK (Data Access) credentials. So make sure you’ve got them at hand.

  5. Permissions
    Run the upgrade with an account which is local admin on the SCOM Management Servers to be upgraded, has SCOM admin permissions in SCOM and has SQL sysadmin permissions on the SQL instances hosting the SCOM SQL databases. Otherwise the upgrade will fail!

B 01: Upgrade – First SCOM Management Server
Now it’s time to start the upgrade and I start with the first SCOM 2012 R2 UR#11 Management Server, also hosting the RSME role. it also hosts the SCOM Web Console and the Console.

Additional information regarding the required UR level:
For the upgrade itself Microsoft has also recently published an updated TechNet article, to be found here. And as you can see, SCOM 2012 R2 doesn’t have to be on UR#11 level, since the upgrade can also be run from UR#9. My guess is however, that most SCOM 2012 R2 environments will be on UR#11 already. When it’s not, but on UR#9 level you don’t have to roll out UR#11 first. Just make sure you meet all requirements and go through all pre-upgrade tasks successfully. Then you’re ready to upgrade to SCOM 2016 just as well.

I won’t post all screenshots, but only the most important ones. And apologies for the lack of quality of those very same screenshots. I recorded the upgrade with the built-in Steps Recorder, not knowing the screens are saved in a lower quality Sad smile.

And YES I’ve stopped & disabled all SCOM services on all other SCOM Management Servers which aren’t upgraded at that moment.

  • SCOM 2016 recognizes the presence of multiple SCOM 2012 R2 components and will update them accordingly to SCOM 2016:
  • Upgrade is running:
  • Upgrade ran successfully on all SCOM 2012 R2 components. Please know that the upgrade of the SCOM 2012 databases takes a long time. And also know that the new license isn’t set, so SCOM 2016 is en eval mode. Therefore a warning is shown for the SCOM Management Server, highlighted in yellow:

B 02: Upgrade – Second SCOM Management Server
This upgrade runs much faster since the related SCOM databases are already successfully upgraded (as such the upgrade flags are set in those databases, telling the upgrade to skip them) and this second SCOM Management Server only runs the Console, NOT the Web Console.

Since I stopped & disabled the SCOM services here before upgrading the first SCOM Management Server, I start and set them to start automatically them BEFORE upgrading this SCOM Management Server and STOP & DISABLE them on the FIRST SCOM Management Server, which is already upgraded. DON’T FORGET THIS!!!

  • SCOM 2016 recognizes all SCOM 2012 R2 components to be upgraded:
  • Upgrade ran successfully:

Now I start the SCOM services on the first SCOM Management Server and set them to start automatically. Also I start the SCOM website in IIS. Now the most crucial SCOM components are upgraded to SCOM 2016 RTM. There are no SCOM Gateway Servers in my environment to upgrade Smile.

B 03: Upgrade – Install SCOM 2016 UR#01
Now it’s time to install SCOM 2016 UR#01. The order of it is like all other URs for SCOM 2012 R2:

  1. Management Servers;
  2. Gateway Servers (not applicable, since SCOM 2016 UR#01 doesn’t touch them);
  3. SCOM Console(s);
  4. SCOM Web Console;
  5. SQL query (SCOM 2016 UR#01 only touches the OpsMgr database);
  6. SCOM Management Packs;
  7. SCOM Agents (when running Gateways, copy the MSP files to their Agent staging folders!).

So I start with the first SCOM Management Server which also hosts the SCOM Web Console and Console. In this case these SCOM 2016 components are touched:

  • SCOM Management Server;
  • SCOM Web Console;
  • SCOM Console.

After this I upgrade the second SCOM Management Server, also hosting the Console.

In this case these SCOM 2016 components are touched:

  • SCOM Management Server;
  • SCOM Console.

Then I run the SQL query (as stated earlier, only the OpsMgr database is touched):

After that I import the SCOM 2016 core MPs:

And now – except for the SCOM Agents that is (still on SCOM 2012 R2 UR#11 level) – SCOM is on 2016 UR#01 level.

C 01 – Upgrade – The Aftermath
Now it’s time to wrap it all up with these steps:

  1. Upgrade SCOM Agents to SCOM 2016 UR#01
    Upgrade them in batches. When installed from the Console upgrade them from there as well. For all other manually installed Agents, update them manually or automated by GPO, SCCM or really manually.

  2. Install the license
    Go from eval to retail by installing the SCOM license. Fun fact: The same SCOM 2012 R2 key is used by SCOM 2016 as well…

  3. Upgrade Helper MP
    I used Wei H Lim’s excellent Upgrade Helper MP in order to ascertain myself EVERYTHING is on SCOM 2016 level. Both dashboards are viewed, so I am 100% sure all is okay:


    Some Agents have issues, but they are on SCOM 2016 UR#01 level none the less.

  4. Remove the Upgrade Helper MP
    When you’re 100% sure all components are on SCOM 2016 level, you can safely remove the Upgrade Helper MP.

Wrap up
Since SCOM 2016 isn’t that much of a change compared to SCOM 2012 R2, the upgrade is less likely to fail, moreover when you’re sure all components (underlying Windows Server OS and SQL instances included) meet the SCOM 2016 requirements AND you respect the pre-upgrade tasks.

Compared to all other SCOM upgrades I have done before this was the easiest one. None the less: PREPARATION is KEY!!!

Also: when your current SCOM 2012 R2 environment comes from SCOM 2012 SP1 or even older, THINK TWICE before upgrading to SCOM 2016. Changes are things will break during the upgrade, so seriously consider the along side scenario in which a new SCOM 2016 environment is rolled out alongside your current SCOM 2012 R2 environment and monitoring is gradually moved to the new SCOM 2016 UR#01 environment.

Thursday, October 20, 2016

Extensible Network Monitoring MP Generator Tool

Yesterday Microsoft released the Extensible Network Monitoring MP Generator Tool.

As the name implies this tool allows users to generate custom Management Packs in order to extend monitoring for network enabled devices.

It’s compatible with SCOM 2012 R2 and SCOM 2016.

Want to know more or to download the tool? Go here.

Wednesday, October 19, 2016

SCOM 2012x/2016 Console Crash Hotfix Available

As stated before, the Windows Cumulative Update October 2016 breaks the SCOM 2012x/2016 Console. Until now the only way to solve it was to remove those very samen CUs.

Gladly Microsoft has released a hotfix addressing this issue as stated in KB3200006. From Microsoft Update Catalog the required Hotfix (per Windows product) can be downloaded.

Tuesday, October 18, 2016

OM16: UR#01 Available

2016-10-18 Update: Just got word this UR#01 is actually the ‘GA’ release, thus it’s strongly advised to apply it to any SCOM 2016 RTM environment. Also the list of fixes and updates is huge. As such this kind of list wouldn’t help at all.

In other words: Simply install this UR#01 for SCOM 2016 RTM and be done with it.

Since a few days UR#01 for SCOM 2016 RTM is already out, see KB3190029.

As far as I can see this UR touches the SCOM Management Server, Console and Web Console. Also the updates are rather small:

And when downloaded from the Microsoft Update Catalog, the download works differently meaning when you select the update you want to download (server, console, web console)

You’ll be shown the required download files in a new screen. Select the one you require for your language and the download will start.



Web Console:

Monday, October 17, 2016

!!!WARNING!!! October 2016 Cumulative Windows Updates Causes SCOM 2012/2016 Console Crashes On All Windows OS Versions (Client & Server)!!!

2016-10-19 Update: Microsoft has released a hotfix which addresses this issue. Read this posting for more detailed information.

Just got word that the October 2016 cumulative Windows updates (KB3194798, KB3192392, KB3185330 & KB3185331) breaks the SCOM 2012/2016 Console on all Windows versions (Windows Server: 2008 R2 up to 2016 and Windows Client 7 up to 10).

Dirk Brinkmann (SCOM PFE for Microsoft Germany) blogged about it, to be found here.

Cross Post: Step-By-Step Guide Installing SCOM 2016 On WS 2016 & SQL 2016

Fellow MVP Ravi Yadav wrote an excellent posting, all about the latest & greatest: Installing SCOM 2016 on Windows Server 2016 and SQL Server 2016.

Have fun!

Saturday, October 15, 2016

Upgrading To SCOM 2016 RTM - 01 - Preparation Is Key!!!

As stated before I would write a new posting about upgrading from SCOM 2012 UR#11 to SCOM 2016 RTM. However as it turns out, only ONE posting isn’t enough. Hence a whole new series about this upgrade.

So this is the first posting of the new series. And yes it start’s with PREPARATION. Without it, changes are the upgrade will fail and there’s is no way back. Ouch! So make sure everything is okay before you start the upgrade.

Preparation = Success?
Wish that was true. But in fact it isn’t. Even when thorough preparations, I’ve seen quite a few upgrades turn sour. So always think twice before upgrading. Sometimes it’s just better to run an alongside migration scenario, where you rollout a whole new SCOM 2016 RTM environment, make the SCOM Agents double homed, a migrate to the new SCOM 2016 RTM MG step by step.

However, when upgrading ALWAYS know what your starting point is. Meaning when the current SCOM 2012 R2 environment is already an upgraded SCOM MG, like SCOM 2012 RTM > SCOM 2012 SP1 > SCOM 2012 R2 (or even SCOM 2012 SP1 > SCOM 2012 R2), my advice is NOT TO UPGRADE but to choose for the alongside migration scenario instead.

Simply because the upgrades which turned sour on me were for 95% SCOM MGs which were upgraded previously. So SCOM MGs like that are simply to be scrapped along the way and to be replaced by a brand new SCOM 2016 RTM MG.

Another reason for this approach is also the underlying Server operating systems, used SQL Server instances and so on. Many times those require upgrades as well, making the SCOM 2016 RTM upgrade an even bigger challenge, with cost and resource requirements which aren’t to be accepted easily.

And on top of that, you can’t give (or you shouldn’t) the guarantee the upgrade itself will run smoothly. Ouch again!

Things to do
Luckily I already blogged about that, so check out this posting of mine: Resources & Tips About Upgrading To SCOM 2016.

Also good steps to take are:

  • Install UR#11 for SCOM 2012 R2 before upgrading;
  • Check the SCOM 2016 RTM requirements in order to know for sure your SCOM 2012 R2 UR#11 environment fully supports SCOM 2016 RTM;
  • On the SCOM MS servers: Install the SCOM 2016 RTM Console requirements BEFORE upgrading to SCOM 2016 RTM;
  • Import Wei H Lim’s Upgrade Helper MP for SCOM 2012 R2 to 2016 and USE it! Smile
  • READ and EXECUTE the Pre-Upgrade Tasks as stated on the TechNet pages (which are down now?);
  • Make backups of the VMs running the SCOM Management Servers, Gateway Servers, SCOM SQL databases so there is a way back;
  • Make sure your SCOM 2012 R2 UR#11 environment is healthy before upgrading to SCOM 2016 RTM. Upgrades WON’T fix issues! So fix your environment first before upgrading.

In the next posting in this series I’ll take a deeper dive into some aspects of the preparations. However, I want to do that with the proper TechNet pages being available so I have some official resources to refer to.