2.1.1 OPatch Utility Information 
每个节点的gi home和db home执行,version 11.2.0.3.6 or later. 
$ unzip -d  
$ /OPatch/opatch version 

#su - root 
chmod 777 /tmp/p6880880_112000_Linux-x86-64.zip 
chown grid:oinstall /tmp/p6880880_112000_Linux-x86-64.zip 
su - grid 
unzip /tmp/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.0/grid/ 
replace /u01/app/11.2.0/grid/OPatch/jlib/oracle.opatch.classpath.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: A 

#su - root 
chown oracle:oinstall /tmp/p6880880_112000_Linux-x86-64.zip 
su - oracle 
unzip /tmp/p6880880_112000_Linux-x86-64.zip -d /u01/app/oracle/product/11.2.0/dbhome_1/ 
rm /tmp/p6880880_112000_Linux-x86-64.zip 



2.1.2 OCM(Oracle Configuration Manager) Configuration 
创建ocm README没直接给出，而是引用了一篇文档 
示例中ORACLE_HOME是db_home，此次主要是gi PSU所以安装用gi_home

./emocmrsp后可不加参数，将在在当前目录下生成ocm.rsp
节点2同样执行：

[root@rac-node1 ~]# su - grid 
[grid@rac-node1 ~]$ /u01/app/11.2.0/grid/OPatch/ocm/bin/emocmrsp -no_banner -output /tmp/ocm_my.rsp 
Provide your email address to be informed of security issues, install and 
initiate Oracle Configuration Manager. Easier for you if you use your My 
Oracle Support Email address/User Name. 
Visit http://www.oracle.com/support/policies.html for details. 
Email address/User Name: 

You have not provided an email address for notification of security issues. 
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y 
The OCM configuration response file (/tmp/ocm_my.rsp) was successfully created. 
[grid@rac-node1 ~]$ ll /tmp/ocm_my.rsp 
-rw-r--r-- 1 grid oinstall 621 Oct 31 07:59 /tmp/ocm_my.rsp 
[grid@rac-node1 ~]$ chmod 777 /tmp/ocm_my.rsp 
[grid@rac-node1 ~]$ ll /tmp/ocm_my.rsp 
-rwxrwxrwx 1 grid oinstall 621 Oct 31 07:59 /tmp/ocm_my.rsp

2.1.3 Validation of Oracle Inventory 
每个节点的gi home和db home执行： 
$ /OPatch/opatch lsinventory -detail -oh  

su - grid 
/u01/app/11.2.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/11.2.0/grid/ 

su - oracle 
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/11.2.0/dbhome_1/ 




2.1.4 Download and Unzip the Patch 
所有节点都online可以访问 
grid用户解压到共享目录（可以非共享） 
目录不能有其他文件，（不能是/tmp根目录下，/tmp下建子目录可以） 
ORA_INSTALL group需要有读的权限 
su - grid

mkdir -p /tmp/patch

cd /tmp/patch

unzip p19380115_112040_Linux-x86-64

chmod -R 777 /tmp/patch

2.1.5 Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch 
oracle用户停止em: 
$ /bin/emctl stop dbconsole 



2.3 OPatch auto for GI 
root运行会同时patch gi_home和db_home。 
如果GI home or Oracle RAC database home is in non-shared storage.，要在每个节点手动执行，并且不能同时执行。 


2.4 Patch Installation 

补丁安装的2种情形: 
Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured. 
Case 2: GI Home is not shared, Database Home is shared, ACFS may be used. 

Patching Oracle RAC Database Homes and GI Together 
Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured. 

As root user, execute the following command on each node of the cluster: 
# opatch auto /19380115 -ocmrf  
同时打gi_home和db_home 需读取ocm.rsp权限 
chmod 777 /tmp/ocm_my.rsp 
/u01/app/11.2.0/grid/OPatch/opatch auto /tmp/patch/19380115 -ocmrf /tmp/ocm.rsp 


对于auto patch失败的部分，无需rollback，再次auto即可 
Verifying environment and performing prerequisite checks... 
All of the constituents of the composite patch are already installed in the Oracle Home. No need to apply this patch. 
Log file location: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch3014-10-30_23-06-06PM_1.log 


--其他节点执行同样的操作 



2.5 Patch Post-Installation Instructions安装后期指令 

2.5.2 Loading Modified SQL Files into the Database 
1.在任一节点执行 

SQL> startup

SQL> @?/rdbms/admin/catbundle.sql psu apply
更新dba_registry_history view以便使用psu 
--没提示编辑无效对象，psu的过程应该包括 


2.检查日志： $ORACLE_BASE/cfgtoollogs/catbundle for any errors: 
catbundle_PSU__APPLY_.log 
catbundle_PSU__GENERATE_.log 


2.5.3 更新rman Catalog 
If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it: 
$ rman catalog username/password@alias 
RMAN> UPGRADE CATALOG; 



2.6 Patch Post-Installation Instructions for Databases Created or Upgraded after Installation of Patch in the Oracle Home 
安装db_home补丁后,使用模版创建的database需要执行"Loading Modified SQL Files into the Database" 

These instructions are for a database that is created or upgraded after the installation of the patch. 

You must execute the steps in Section 2.5.2, "Loading Modified SQL Files into the Database" for any new database only if it was created by any of the following methods: 

Using DBCA (Database Configuration Assistant) to select a sample database (General, Data Warehouse, Transaction Processing) 

Using a script that was created by DBCA that creates a database from a sample database 

There are no actions required for databases that have been upgraded. 

查询DB补丁记录 
set linesize 300 
col action_time for a30 
col version for a15 
col id for 99 
col action for a10 
col namespace for a10 
col COMMENTS for a20 
col BUNDLE_SERIES for a15 
select * from registry$history; 
查询补丁状态grid、oracle皆可 
$ $ORACLE_HOME/OPatch/opatch lspatches 
$ $ORACLE_HOME/OPatch/opatch lsinventory 

2.7 Patch Deinstallation 

Roll Back the Oracle RAC Database Homes and GI Together 

Case 1: GI Home and Database Homes that are not shared and ACFS file system is not configured. 

As root user, execute the following command on each node of the cluster. 

# opatch auto /19380115 -rollback -ocmrf  

/u01/app/11.2.0/grid/OPatch/opatch auto /tmp/patch/19380115 -rollback -ocmrf /tmp/ocm.rsp

节点2同样执行

If the message, "A system reboot is recommended before using ACFS" is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

Case 2: GI Home is not shared, Database Home is shared and ACFS may be used. 

From the Oracle database home, make sure to stop the Oracle RAC databases running on all nodes. As the database home owner execute: 

$ /bin/srvctl stop database –d  
On the 1st node, unmount the ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. 

On the 1st node, roll back the patch from the GI Home using the opatch auto command. As root user, execute the following command: 

# opatch auto /19380115 -oh -rollback -ocmrf  
If the message, "A system reboot is recommended before using ACFS” is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

On the 1st node, remount ACFS file systems. See My Oracle Support Document 1494652.1 for mounting ACFS file systems. 

On the 1st node, roll back the patch to the Database home using the opatch auto command. This operation will rollback the patch to the Database home across the cluster given that it is a shared ACFS home. Note that a USM only patch cannot be applied to a Database home. As root user, execute the following command: 

# opatch auto /19380115 -oh -rollback -ocmrf  
On the 1st node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute: 

$ /bin/srvctl start instance –d -n  
On the 2nd (next) node, unmount the ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. 

On the 2nd node, roll back the patch to GI Home using the opatch auto command. As root user, execute the following command: 

# opatch auto /19380115 -oh -rollback -ocmrf  
If the message, "A system reboot is recommended before using ACFS” is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

On the 2nd node, running the opatch auto command in Step 9 will restart the stack. 

On the 2nd node, remount ACFS file systems. See My Oracle Support Document 1494652.1 for mounting ACFS file systems. 

On the 2nd node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute: 

$ /bin/srvctl start instance –d -n  
Repeat Steps 8 through 13 for all remaining nodes of the cluster. 

2.8 Patch Post-Deinstallation Instructions for an Oracle RAC Environment 

Follow these steps only on the node for which the steps in Section 2.5.2, "Loading Modified SQL Files into the Database" were executed during the patch application.: 

只需在一个节点上执行，升级时执行了"Loading Modified SQL Files into the Database"的节点上

Start all database instances running from the Oracle home. (For more information, see Oracle Database Administrator's Guide.) 
启动所有的instance

For each database instance running out of the ORACLE_HOME, connect to the database using SQL*Plus as SYSDBA and run the rollback script as follows: 

sqlplus / AS SYSDBA 
SQL> STARTUP 
SQL> @?/rdbms/admin/catbundle_PSU__ROLLBACK.sql 

 SQL> @?/rdbms/admin/catbundle_PSU_CLUSDB_ROLLBACK.sql

In an Oracle RAC environment, the name of the rollback script will have the format catbundle_PSU__ROLLBACK.sql. 

SQL> set linesize 300

SQL> col action_time for a30

SQL> col version for a15

SQL> col id for 99

SQL> col action for a10

SQL> col namespace for a10

SQL> col COMMENTS for a20

SQL> col BUNDLE_SERIES for a15

SQL> select * from registry$history;

ACTION_TIME ACTION NAMESPACE VERSION ID COMMENTS BUNDLE_SERIES

------------------------------ ---------- ---------- --------------- --- -------------------- ---------------

30-OCT-14 04.02.28.027746 PM APPLY SERVER 11.2.0.4 0 Patchset 11.2.0.2.0 PSU

31-OCT-14 08.44.46.147995 AM APPLY SERVER 11.2.0.4 4 PSU 11.2.0.4.4 PSU

31-OCT-14 02.01.00.767507 PM ROLLBACK SERVER 11.2.0.4 4 PSU 11.2.0.4.4 PSU

31-OCT-14 02.01.09.369049 PM APPLY SERVER 11.2.0.4 0 Patchset 11.2.0.2.0 PSU

Check the log file for any errors. The log file is found in $ORACLE_BASE/cfgtoollogs/catbundle and is named catbundle_PSU__ROLLBACK_.log where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS. If there are errors, see Section 3, "Known Issues". 

Ensure that you verify the Oracle Inventory and compare the output with the one you ran in Section 2.1.3, "Validation of Oracle Inventory" and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command: 

$ opatch lsinventory 
All other instances can be started and accessed as usual while you are executing the deinstallation steps. 

/u01/app/11.2.0/grid/OPatch/opatch lsinventory

/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory

3 Known Issues

This section includes the following known issues: 

Issue 1

When Trace File Analyzer (TFA) is patched as part of a PSU, any rollback of the PSU will not roll back any patch applied to TFA. This is intended behavior, but if there are any issues with patching TFA itself, then manual steps will be required to recover.

回退psu不会回退Trace File Analyzer (TFA)，这是有意的行为。

If a PSU has been rolled back and due to some issue with patching TFA it has to be reinstalled, then you should:

Remove TFA by executing the following command on each node as the root user:

GIHOME/tfa//tfa_home/bin/uninstalltfa.sh -local 

Reinstall the previous TFA version by executing the following command on each node as the root user:

GIHOME/crs/install/tfa_setup.sh -crshome -silent 

Issue 2

After rolling back using opatch auto from the Oracle Grid infrastructure Patch Set Update 11.2.0.4.4, database and listener resources are going to be offline.

You should start the database and listener resources using the srvctl command.

When performing any further patching actions, databases and listeners may not be started up automatically for the home being patched on a node. Start up these explicitly by executing srvctl start database and srvctl start listener commands.

排错：

2014-10-30 23:03:10: status of apply patch is 18688 
2014-10-30 23:03:10: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.6 
Copyright (c) 2013, Oracle Corporation. All rights reserved. 


Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1 
Central Inventory : /u01/app/oraInventory 
from : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc 
OPatch version : 11.2.0.3.6 
OUI version : 11.2.0.4.0 
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch3014-10-30_23-03-00PM_1.log 

Verifying environment and performing prerequisite checks... 
Prerequisite check "CheckActiveFilesAndExecutables" failed. 
The details are: 


Following executables are active : 
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed. 
Log file location: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch3014-10-30_23-03-00PM_1.log 

OPatch failed with error code 73 



libclntsh.so.11.1文件busy，刚开始没想到杀进程，重启了系统，之后fuser 文件没提示有人用，再次auto opatch成功。 
fuser /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 


如果有则执行如下： 
oracle@mydb1>/sbin/fuser /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: 12195m 
oracle@mydb1> 
oracle@mydb1>ps -ef |grep 12195 
oracle 12195 11944 0 Jan19 pts/7 00:00:00 adrci 
oracle 22813 3717 0 21:18 pts/13 00:00:00 grep 12195 
oracle@mydb1>kill -9 12195