EM Cloud Control – Applying system patches

Patching system software is a  common undertaking, that should be smooth and easy. Software projects delivering  artifacts, (should) have some kind of deployment pipeline to get to production. The same holds true for 3rd party/vendor software patches. Also with today’s ever expanding interconnected systems, and thus possible exposure to bugs/undocumented features, it’s crucial to have an active patching/lifecycle strategy.

Let’s look at how EM Cloud Control facilitates patching itself.

First make sure EM is connected to Oracle support.

Add proxy, if needed.

Add MOS credentials. Do not use your personal EM & MOS account for the catalog refresh job. I’ve created a separate account on MOS just for downloading metadata and patches.

Force an update.

On the EM home page, there’s a patch widget, which will bring us to the available patches for our monitored targets.

Select only agent target type to view all targets, needing patching.

Create  new patchplan from the selected patch/agent.

Next open the new plan just follow all the steps in the wizard. In the analyze step Cloud Control will validate all necessary prerequisites to be able to succesfully patch the target.

Last step is to execute the patchplan. A job will be created.

Next step would be to deploy the patches to whole group of agents. This is simple to achieve with a patch template.

First create a patch template from the patch plan, executed earlier.

From this patch template, we can create a new patchplan with all the targets we want to patch at one. The rest of the steps are the same as described earlier.

Now all agents should have their latest core patches applied.

But I actually missed the plug-in patches in the ‘patches & updates’ overview. They are not visible for some reason. But these so-called ‘system’ patches (or opatch) were needed to fix some annoying bugs in our Coud Control environment.

This requires some additional work and we will patch the following components:

• OPatch of the Oracle Management Server (OMS)
• Apply latest PSU to the OMS
• Apply latest patches to plug-ins OMS-side
• Apply latest patches to plug-ins agent-side
• Upgrade  JVMD/ADP engines with latest patches
• UPgrade JVMD/ADP agents with latest patches

Patch OPatch

unzip –d $ORACLE_HOME p6880880_111000_Linux-x86-64.zip

Apply latest PSU to the OMS

rm –rf /tmp/orapatch/*
unzip -d /tmp/orapatch/ p18090046_121030_Generic_OMS_PSU_121032.zip
cd /tmp/orapatch/*

emctl stop oms

opatchauto apply -analyze -property_file $ORACLE_HOME/OPatch/oms_patch.properties
opatchauto apply -property_file $ORACLE_HOME/OPatch/oms_patch.properties

Make sure you sync all emcli clients after patching the OMS.

emcli login -username='t3'
emcli sync

Next we will patch the plug-ins on the OMS-side. This involves the same steps, but each plug-in has its own patch.

# repeat the following steps for all OMS side plugin patches
rm -rf /tmp/orapatch/*
unzip -d /tmp/orapatch/ Your_plugin_patch.zip
emctl stop oms

cd /tmp/orapatch/*
opatchauto apply -analyze -property_file $ORACLE_HOME/OPatch/oms_patch.properties
opatchauto apply -property_file $ORACLE_HOME/OPatch/oms_patch.properties

After all patches are done, we can start the OMS again.

emctl start oms

Next step is to patch all agent plug-ins. We need to manually patch a plug-in and then create a ‘custom plugin update’ from that image. The custom plugin update can be used to ‘redeploy’ the patched plugin on all your other agents.

Each agent can have a different set of plug-ins, depending on the type of components running on the managed node. Plug-ins are patched separately. Patches are applied to the separate plug-in installations. Be aware that plug-ins are sometimes split into a monitoring and a discovery installation (PLUGIN_HOME). The custom plugin update will encompass both monitoring and discovery patches.

export ORACLE_BASE=/apps/oracle
export AGENT_BASE=$ORACLE_BASE/agent12c
export ORACLE_HOME=$AGENT_BASE/core/12.1.0.3.0
export PATH=$ORACLE_HOME/OPatch:$PATH
mkdir /tmp/orapatch/$(whoami)/

Blackout and stop the ‘test’ agent

$ORACLE_HOME/bin/emctl start blackout Patch_agent_$(hostname) –nodeLevel
$ORACLE_HOME/bin/emctl status blackout
$ORACLE_HOME/bin/emctl stop agent

Patch the Oracle database discovery plugin (version 12.1.0.5)

export PLUGIN_HOME=$AGENT_BASE/plugins/oracle.sysman.db.discovery.plugin_12.1.0.5.0

rm -rf /tmp/orapatch/$(whoami)/*
unzip -d /tmp/orapatch/$(whoami)/ the_plugin_patch.zip
cd /tmp/orapatch/$(whoami)/*

opatch apply -analyze -oh $PLUGIN_HOME
opatch apply -oh $PLUGIN_HOME

You can repeat the steps for all other plug-ins. Just change the PLUGIN_HOME and unzip the accompanying patch file.

After finishing patching, start the agent and end the blackout.

$ORACLE_HOME/bin/emctl start agent
$ORACLE_HOME/bin/emctl stop blackout Patch_agent_$(hostname)

So now we patched one agent. Next step is to deploy these patched plugins to other agents. This can be done entirely through ’emcli’.

emcli
login(username='t3')
Enter password  ********

Login successful

help('create_custom_plugin_update')
  emcli create_custom_plugin_update
        -agent_name="agent_name"
        -plugin_id="plugin_id"
        [-overwrite]

create_custom_plugin_update(agent_name="testcop.whrocks.nl:3872" ,plugin_id="oracle.sysman.db" ,overwrite=True)
create_custom_plugin_update(agent_name="testcop.whrocks.nl:3872" ,plugin_id="oracle.sysman.oh")
create_custom_plugin_update(agent_name="testcop.whrocks.nl:3872" ,plugin_id="oracle.sysman.emas" ,overwrite=True)

I now have create custom plugin updates for the Oracle database, Fusion middleware and the ‘Oracle Home’ plug-in.

All information can be queried through ’emcli’.

list_custom_plugin_updates()
Plug-in Name                            Plug-in ID               Version [revision]       Created on                    Source

Oracle Fusion Middleware                oracle.sysman.emas       12.1.0.5.0               Apr 30, 2014 11:13:18 AM CEST Agent testcop.whrocks.nl:3872
Oracle Database                         oracle.sysman.db         12.1.0.5.0               Apr 29, 2014 4:19:42 PM CEST  Agent testcop.whrocks.nl:3872
Oracle Home                             oracle.sysman.oh         12.1.0.3.0               Apr 29, 2014 4:21:34 PM CEST  Agent testcop.whrocks.nl:3872

Querying the specific patch contents (for discovery patch add ,discovery=True  to the command)

list_patches_in_custom_plugin_update(plugin="oracle.sysman.db:12.1.0.5.0")
Oracle Interim Patch Installer version 11.1.0.10.3
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /apps/oracle/product/agent12c/plugins/oracle.sysman.db.agent.plugin_12.1.0.5.0
Central Inventory : /apps/oracle/11.0.2/oraInventory
   from           : /apps/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
OPatch version    : 11.1.0.10.3
OUI version       : 11.1.0.11.0
Log file location : /apps/oracle/product/agent12c/plugins/oracle.sysman.db.agent.plugin_12.1.0.5.0/cfgtoollogs/opatch/opatch2014-04-30_08-20-29AM_1.log

Lsinventory Output file location : /apps/oracle/product/agent12c/plugins/oracle.sysman.db.agent.plugin_12.1.0.5.0/cfgtoollogs/opatch/lsinv/lsinventory2014-04-30_08-20-29AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Enterprise Manager plug-in for Oracle Database                       12.1.0.5.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  18356442     : applied on Tue Apr 29 15:55:58 CEST 2014
Unique Patch ID:  17475043
Patch description:  "EM DB PLUGIN BUNDLE PATCH 12.1.0.5.4 (AGENT SIDE)"
   Created on 12 Mar 2014, 04:30:30 hrs PST8PDT
   Bugs fixed:
     18136919, 17652710, 17971467, 18026077, 18336314, 18336315, 17594934
     18012965, 17902923, 18356442, 18104700, 16006270, 18307270
--------------------------------------------------------------------------------
OPatch succeeded.

Now that we have created the necessary custom plugin updates, we can redeploy them to other agents.

help('redeploy_plugin_on_agent')
  emcli redeploy_plugin_on_agent
        {-agent_names="agent1[;agent2...]" | -group_name="group1"}
        -plugin="plug-in_id:version"
        [-redeploy_noprompt]
        [-include_dependent_agents]

redeploy_plugin_on_agent(agent_names="agentx.whrocks.nl:3872; agenty.whrocks.nl:3872", plugin="oracle.sysman.db:12.1.0.5.0", redeploy_noprompt=True)

get_plugin_deployment_status(plugin_id="oracle.sysman.db")

A job is started from the OMS that will patch all the provided targets.

After patching all agents, we can upgrade the APM components (JVMD,ADP). This can be done through the UI.

Make sure you blackout and shutdown the engines, then go to Middleware Management > APM.

You will see a blue sign indicating that there is an upgrade available.

Next apply the correct credentials and a job is submitted to upgrade the engines. When the job has succeeded the engines will be live again.

Patching the APM agent follows a similar procedure. From the APM homepage click the ‘Manage Diagnostics agents’ button.

Select the agents you want to upgrade.

Make sure you apply the correct credentials to each agent and submit the upgrade.

The patch management of Enterprise Manager Cloud Control is quite robust. There are still some parts that need some work (automatic system patching of plug-ins), but all in all it works really well.