I am working for a customer on a quite big implementation of Cold Failover Cluster with Oracle Grid Infrastructure on Linux. I hope to have some material to publish soon about it! However, in this post I will be talking about patching the database in a cold-failover environment.
DISCLAIMER: I use massively scripts provided in this great blog post by Simon Pane:
https://www.pythian.com/blog/oracle-database-12c-patching-dbms_qopatch-opatch_xml_inv-and-datapatch/
Thank you Simon for sharing this 
Intro
We are not yet in the process of doing out-of-place patching; at the moment the customer prefers to do in-place patching:
- evacuate a node by relocating all the databases on other nodes
- patching the node binaries
- move back the databases and patch them with datapatch
- do the same for the remaining nodes
I beg to disagree with this method, being a fan of having many patched golden copies distributed on all servers and patching the databases by just changing the ORACLE_HOME and running datapatch (like Rapid Home Provisioning does). But, this is the situation today, and we have to live with it.
Initial situation
- Server 1, 2 and 3: one-off 20139391 applied
- New database created
When the DBCA creates a new database, in 12.1.0.2, it does not run datapatch by default, thus, the database does not have any patches installed.
However, this specific one-off patch does not modify anything in the database (sql_patch=false)
SQL> -- Patches installed in the oracle home
SQL> r
1 with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
2 select x.patch_id, x.patch_uid, x.description
3 from a,
4 xmltable('InventoryInstance/patches/*'
5 passing a.patch_output
6 columns
7 patch_id number path 'patchID',
8 patch_uid number path 'uniquePatchID',
9 description varchar2(80) path 'patchDescription',
10 sql_patch varchar2(8) path 'sqlPatch'
10 ) x
11 *
PATCH_ID PATCH_UID DESCRIPTION SQL_PATCH
---------- ---------- ------------------------- ---------
20139391 18466820 false
SQL> -- Patches installed in the database
SQL> select s.patch_id, s.patch_uid, s.description from dba_registry_sqlpatch s;
no rows selected
SQL>and the datapatch runs without touching the db:
oracle1> $ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 12.2.0.0.0 on Wed Nov 2 13:34:10 2016 Copyright (c) 2014, Oracle. All rights reserved. Connecting to database...OK Determining current state...done Current state of SQL patches: Adding patches to installation queue and performing prereq checks... Installation queue: Nothing to roll back Nothing to apply SQL Patching tool complete on Wed Nov 2 13:34:13 2016 oracle1>
Next step: I evacuate the server 2 and patch it, then I relocate my database on it
oracle2> $ORACLE_HOME/OPatch/opatch lspatches 24340679;DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) OPatch succeeded. oracle2> oracle2> crsctl relocate res theludot.db -n oracle2 CRS-2673: Attempting to stop 'theludot.db' on 'oracle1' CRS-2677: Stop of 'theludot.db' on 'oracle1' succeeded CRS-2672: Attempting to start 'theludot.db' on 'oracle2' CRS-2676: Start of 'theludot.db' on 'oracle2' succeeded oracle2>
Now the database is not at the same level of the binaries and need to be patched:
SQL> -- Patches installed in the oracle home
SQL> r
1 with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
2 select x.*
3 from a,
4 xmltable('InventoryInstance/patches/*'
5 passing a.patch_output
6 columns
7 patch_id number path 'patchID',
8 patch_uid number path 'uniquePatchID',
9 description varchar2(80) path 'patchDescription',
10 constituent number path 'constituent',
11 patch_type varchar2(20) path 'patchType',
12 rollbackable varchar2(20) path 'rollbackable',
13 sql_patch varchar2(8) path 'sqlPatch',
14 DBStartMode varchar2(10) path 'sqlPatchDatabaseStartupMode'
15* ) x
PATCH_ID PATCH_UID DESCRIPTION CONSTITUENT PATCH_TYPE ROLLBACKABLE SQL_PATC DBSTARTMOD
---------- ---------- -------------------------------------------------- ----------- -------------------- ------------ -------- ----------
24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 24340679 singleton true true normal
23144544 20247727 DATABASE BUNDLE PATCH: 12.1.0.2.160719 (23144544) 24340679 singleton true true normal
22806133 19983161 DATABASE BUNDLE PATCH: 12.1.0.2.160419 (22806133) 24340679 singleton true true normal
21949015 19576071 DATABASE BUNDLE PATCH: 12.1.0.2.160119 (21949015) 24340679 singleton true true normal
21694919 19338504 DATABASE BUNDLE PATCH: 12.1.0.2.13 (21694919) 24340679 singleton true true normal
21527488 19238856 DATABASE BUNDLE PATCH: 12.1.0.2.12 (21527488) 24340679 singleton true true normal
21359749 19147148 DATABASE BUNDLE PATCH: 12.1.0.2.11 (21359749) 24340679 singleton true true normal
21125181 18992109 DATABASE BUNDLE PATCH: 12.1.0.2.10 (21125181) 24340679 singleton true true normal
20950328 18903184 DATABASE BUNDLE PATCH: 12.1.0.2.9 (20950328) 24340679 singleton true true normal
20788771 18810992 DATABASE BUNDLE PATCH: 12.1.0.2.8 (20788771) 24340679 singleton true true normal
20594149 18687526 DATABASE BUNDLE PATCH: 12.1.0.2.7 (20594149) 24340679 singleton true true normal
20415006 18565812 DATABASE BUNDLE PATCH: 12.1.0.2.6 (20415006) 24340679 singleton true true normal
20243804 18468778 DATABASE BUNDLE PATCH: 12.1.0.2.5 (20243804) 24340679 singleton true true normalThe column CONSTITUENT is important here because it tells us what the parent patch_id is. This is the column that we have to check when we want to know if the patch has been applied on the database.
oracle2> $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Wed Nov 2 13:47:49 2016
Copyright (c) 2016, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_63956_2016_11_02_13_47_49/sqlpatch_invocation.log
Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of SQL patches:
Bundle series DBBP:
ID 161018 in the binary registry and not installed in the SQL registry
Adding patches to installation queue and performing prereq checks...
Installation queue:
Nothing to roll back
The following patches will be applied:
24340679 (DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679))
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...
Patch 24340679 apply: SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/24340679/20646358/24340679_apply_THELUDOT_2016Nov02_13_48_03.log (no errors)
SQL Patching tool complete on Wed Nov 2 13:49:51 2016
oracle2>Now the patch is visible inside the dba_registry_sqlpatch:
SQL> r 1* select patch_id, patch_uid, description, action_time, action, status, bundle_series, bundle_id from dba_registry_sqlpatch PATCH_ID PATCH_UID DESCRIPTION ACTION_TIME ACTION STATUS BUNDLE_SERIES BUNDLE_ID ---------- ---------- -------------------------------------------------- ------------------------------ --------------- -------- ------------- ---------- 24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 02-NOV-16 01.49.51.664800 PM APPLY SUCCESS DBBP 161018
Notice that the child patches are not listed in thie view.
Rolling back
Now, one node is patched, but the others are not. What happen if I relocate the patched database to a non-patched node?
oracle1> crsctl relocate res theludot.db -n oracle1 CRS-2673: Attempting to stop 'theludot.db' on 'oracle2' CRS-2677: Stop of 'theludot.db' on 'oracle2' succeeded CRS-2672: Attempting to start 'theludot.db' on 'oracle1' CRS-2676: Start of 'theludot.db' on 'oracle1' succeeded oracle1>
The patch is applied inside the database but not in the binaries!
SQL> select patch_id, patch_uid, description, action_time, action, status, bundle_series, bundle_id
2 from dba_registry_sqlpatch;
PATCH_ID PATCH_UID DESCRIPTION ACTION_TIME ACTION STATUS BUNDLE_SERIES BUNDLE_ID
---------- ---------- -------------------------------------------------- ------------------------------ --------------- -------- ------------- ----------
24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 02.11.16 13:49:51.664800 APPLY SUCCESS DBBP 161018
SQL> r
1 with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
2 select x.*
3 from a,
4 xmltable('InventoryInstance/patches/*'
5 passing a.patch_output
6 columns
7 patch_id number path 'patchID',
8 patch_uid number path 'uniquePatchID',
9 description varchar2(80) path 'patchDescription',
10 constituent number path 'constituent',
11 patch_type varchar2(20) path 'patchType',
12 rollbackable varchar2(20) path 'rollbackable',
13 sql_patch varchar2(8) path 'sqlPatch',
14 DBStartMode varchar2(10) path 'sqlPatchDatabaseStartupMode'
15* ) x
PATCH_ID PATCH_UID DESCRIPTION CONSTITUENT PATCH_TYPE ROLLBACKABLE SQL_PATC DBSTARTMOD
---------- ---------- -------------------------------------------------- ----------- -------------------- ------------ -------- ----------
20139391 18466820 singleton true falseIf I run datapatch again, the patch is rolled back:
oracle1> $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.0.0 on Wed Nov 2 14:48:50 2016
Copyright (c) 2014, Oracle. All rights reserved.
Connecting to database...OK
Determining current state...done
Current state of SQL patches:
Adding patches to installation queue and performing prereq checks...
Installation queue:
The following patches will be rolled back:
24340679 (DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679))
Nothing to apply
catcon: ALL catcon-related output will be written to /tmp/sqlpatch_catcon__catcon_24776.lst
catcon: See /tmp/sqlpatch_catcon_*.log files for output generated by scripts
catcon: See /tmp/sqlpatch_catcon__*.lst files for spool files, if any
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...
Patch 24340679 rollback: SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/24340679/20646358/24340679_rollback_THELUDOT_2016Nov. 02_14_48_53.log (no errors)
SQL Patching tool complete on Wed Nov 2 14:48:53 2016
oracle1>The patch has been rolled back according to the datapatch, and the action is shown in the dba_registry_sqlpatch:
SQL> r 1 select patch_id, patch_uid, description, action_time, action, status, bundle_series, bundle_id 2* from dba_registry_sqlpatch PATCH_ID PATCH_UID DESCRIPTION ACTION_TIME ACTION STATUS BUNDLE_SERIES BUNDLE_ID ---------- ---------- -------------------------------------------------- ------------------------------ --------------- -------- ------------- ---------- 24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 02.11.16 13:49:51.664800 APPLY SUCCESS DBBP 161018 24340679 20646358 02.11.16 14:48:53.760632 ROLLBACK SUCCESS
But if I look at the logfile, the patch had some errors:
oracle1> grep "ORA-\|PLS-" /tmp/sqlpatch_catcon_0.log ORA-20001: set_patch_metadata not called ORA-06512: a "SYS.DBMS_SQLPATCH", ligne 621 ORA-06512: a ligne 2 IGNORABLE ERRORS: ORA-02303 IGNORABLE ERRORS: ORA-01418 IGNORABLE ERRORS: ORA-01435 IGNORABLE ERRORS: ORA-01435 IGNORABLE ERRORS: ORA-01435 IGNORABLE ERRORS: ORA-01435 IGNORABLE ERRORS: ORA-01435 IGNORABLE ERRORS: ORA-01435 ORA-01555: cliches trop vieux : rollback segment no , nomme "", trop petit ORA-22924: cliche trop ancien ORA-06512: a "SYS.DBMS_SQLPATCH", ligne 102 ORA-06512: a "SYS.DBMS_SQLPATCH", ligne 663 ORA-06512: a ligne 1
Indeed, the patch looks still there:
SQL> r 1 SELECT dbms_sqlpatch.sql_registry_state 2* FROM dual SQL_REGISTRY_STATE -------------------------------------------------------------------------------- <sql_registry_state> <!-- Non bundle patches --> <!-- Bundle patches --> <patch bundle="yes" id="24340679" uid="20646358" action="APPLY" status="SUCCES S" bundle_series="DBBP" bundle_id="161018">DBBP bundle patch 161018 (DATABASE BU NDLE PATCH: 12.1.0.2.161018 (24340679))</patch> </sql_registry_state>
If I try to run it again, it does nothing/it fails saying the patch is not there:
oracle1> $ORACLE_HOME/OPatch/datapatch -rollback 24340679 SQL Patching tool version 12.2.0.0.0 on Wed Nov 2 16:10:49 2016 Copyright (c) 2014, Oracle. All rights reserved. Connecting to database...OK Determining current state...done Adding patches to installation queue and performing prereq checks...done Installation queue: Nothing to roll back Nothing to apply SQL Patching tool complete on Wed Nov 2 16:10:51 2016 oracle1> $ORACLE_HOME/OPatch/datapatch -rollback 24340679 -force SQL Patching tool version 12.2.0.0.0 on Wed Nov 2 16:11:01 2016 Copyright (c) 2014, Oracle. All rights reserved. Connecting to database...OK Determining current state...done Error: prereq checks failed! patch 24340679: Could not determine unique patch ID for patch 24340679 because it is not present in the SQL registry Prereq check failed, exiting without installing any patches. Please refer to MOS Note 1609718.1 for information on how to resolve the above errors. SQL Patching tool complete on Wed Nov 2 16:11:01 2016
What does it say on the patched node?
oracle2> crsctl relocate res theludot.db -n oracle2 CRS-2673: Attempting to stop 'theludot.db' on 'oracle1' CRS-2677: Stop of 'theludot.db' on 'oracle1' succeeded CRS-2672: Attempting to start 'theludot.db' on 'oracle2' CRS-2676: Start of 'theludot.db' on 'oracle2' succeeded oracle2> oracle2> $ORACLE_HOME/OPatch/datapatch -verbose SQL Patching tool version 12.1.0.2.0 on Wed Nov 2 16:15:36 2016 Copyright (c) 2016, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_7878_2016_11_02_16_15_36/sqlpatch_invocation.log Connecting to database...OK Bootstrapping registry and package to current versions...done Determining current state...done Current state of SQL patches: Bundle series DBBP: ID 161018 in the binary registry and ID 161018 in the SQL registry Adding patches to installation queue and performing prereq checks... Installation queue: Nothing to roll back Nothing to apply SQL Patching tool complete on Wed Nov 2 16:15:49 2016
Whaaat? datapatch there says that the patch IS in the registry and there’s nothing to do. Let’s try to force its apply again:
oracle2> $ORACLE_HOME/OPatch/datapatch -verbose -apply 24340679 -force
SQL Patching tool version 12.1.0.2.0 on Wed Nov 2 16:17:40 2016
Copyright (c) 2016, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_12726_2016_11_02_16_17_40/sqlpatch_invocation.log
Connecting to database...OK
Determining current state...done
Current state of SQL patches:
Bundle series DBBP:
ID 161018 in the binary registry and ID 161018 in the SQL registry
Adding patches to installation queue and performing prereq checks...
Installation queue:
Nothing to roll back
The following patches will be applied:
24340679 (DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679))
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...
Patch 24340679 apply: SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/24340679/20646358/24340679_apply_THELUDOT_2016Nov02_16_17_40.log (no errors)
SQL Patching tool complete on Wed Nov 2 16:18:50 2016SQL> r 1 select patch_id, patch_uid, description, action_time, action, status, bundle_series, bundle_id 2* from dba_registry_sqlpatch PATCH_ID PATCH_UID DESCRIPTION ACTION_TIME ACTION STATUS BUNDLE_SERIES BUNDLE_ID ---------- ---------- -------------------------------------------------- ------------------------------ --------------- -------- ------------- ---------- 24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 02-NOV-16 01.49.51.664800 PM APPLY SUCCESS DBBP 161018 24340679 20646358 02-NOV-16 02.48.53.760632 PM ROLLBACK SUCCESS 24340679 20646358 DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679) 02-NOV-16 04.18.50.320745 PM APPLY SUCCESS DBBP 161018
Conclusion
I’m not sure whether it is safe to run the patched database in a non-patched Oracle Home. I guess it is time for a new SR
Meanwhile, we will try hard not to relocate the databases once they have been patched.
Cheers
—
Ludo
