Quantcast
Channel: Ludovico – DBA survival BLOG
Viewing all 119 articles
Browse latest View live

The new era of spam? Technical-related comments that defeat anti-spam filters

$
0
0

Part of the tasks of every blogger is to update the anti-spam filters (a WordPress plugin in my case, akismet), read the comments in quarantine and either approve them, delete them or mark them as spam (to that the anti-spam plugin can enrich the spam detection algorithms).

So far, the main factor to me to decide to approve or not, is to read the content of the message and find it appropriate or not for the blog post.

Until today.

Some time ago I have written this blog post about diagnosing problems in deleting the Audit Trail in Oracle 12c:

DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL not working on 12c? Here’s why…

Last week I have got this comment:

Technical. Talking about Oracle Audit (which is relevant to my blog post) and with  technical details that only an Oracle professional might know.

So I have approved it and read my blog post again to find a proper answer.

Except that my blog post was not talking about Unified Audit, relink of binaries or exports!

I have realized then that the “link” of the blogger was in fact a website selling sex toys… (yeah, writing this here will bring even more spam… I take the risk 🙂 )

So where does it come from? Actually, the comment has been copied from another blog:

https://petesdbablog.wordpress.com/2013/07/20/12c-new-feature-unified-auditing/

A new type of spam (maybe it exists since a while? I am not a spam expert).

The spammer (bot?) copies technical comments from blog posts with similar keywords and just changes the name and the link of the author.

Beware, bloggers.

Ludo


First draft of a Common Oracle Environment… for the Cloud Database (and not only)

$
0
0

I have just published on GitHub a draft of a common Oracle environment scripts that make the shell environment a little bit smarter than what it is by default. It uses some function and aliases that I have published during the past years.

You can start playing with:

# Connect as oracle
sudo su - oracle

# Clone this repository
git clone https://github.com/ludovicocaldara/COE.git

# Enable the profile scripts
echo ". ~/COE/profile.sh" >> $HOME/.bash_profile

# Load the new profile
. ~/.bash_profile

Ideal for the Oracle Cloud Infrastructure

If you are new to the Oracle Cloud, probably you do not have environment scripts that makes it easy to interact with the database.

The environment scripts that I have published work out-of the box in the cloud (just make sure that you have rlwrap installed so that you can have a better CLI experience).

Actually, they work great as well on-premises, but I assume that you already have something automatic there.

Some examples

  • My famous Smart Prompt 😉 (including version, edition, exit code, etc)

# [ oracle@ludodb01:/home/oracle [22:18:59] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
#

  • u : gets the status of the databases

# [ oracle@ludodb01:/home/oracle [22:18:59] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# u
DB_Unique_Name           DB_Name  ludodb01       Oracle_Home
------------------------ -------- -------------- --------------------------------------------------
CDB_fra1cw               CDB      CDB            /u01/app/oracle/product/18.0.0.0/dbhome_1

  • pmon: just displays the running pmon processes

# [ oracle@ludodb01:/home/oracle [22:27:17] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# pmon
grid      8093     1  0 Mar25 ?        00:01:39 asm_pmon_+ASM1
grid     10293     1  0 Mar25 ?        00:01:43 apx_pmon_+APX1
oracle   11077     1  0 Mar25 ?        00:01:47 ora_pmon_CDB

  • db : sets the environment for a specific DB_NAME, DB_UNIQUE_NAME or SID

# [ oracle@ludodb01:/u01/app/oracle/diag/rdbms/cdb_fra1cw/CDB/trace [22:33:53] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# db CDB
DB_UNIQUE_NAME  = CDB_fra1cw
ORACLE_SID      = CDB
ROLE            = PRIMARY
VERSION         = 18.4.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/18.0.0.0/dbhome_1
NLS_LANG        = AMERICAN_AMERICA.AL32UTF8

  • svcstat : shows the running services (and the corresponding pdb, host, etc) as I described in my previous post

# [ oracle@ludodb01:/home/oracle [22:28:03] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# svcstat
DB_Unique_Name           Service_Name                   PDB                            ludodb01
------------------------ ------------------------------ ------------------------------ --------------
cdb_fra1cw               pdb_service_test               PDB1                           ONLINE

  • s_ : smart alias for sqlplus: connects as sysdba/sysasm by default, or with any arguments that you pass:

# [ oracle@ludodb01:/home/oracle [22:29:14] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# s_

SQL*Plus: Release 18.0.0.0.0 - Production on Mon Apr 15 22:30:22 2019
Version 18.4.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.


Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.4.0.0.0

SQL> show user
USER is "SYS"
SQL> Disconnected from Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.4.0.0.0

# [ oracle@ludodb01:/home/oracle [22:30:30] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# s_ pippo/pippo

SQL*Plus: Release 18.0.0.0.0 - Production on Mon Apr 15 22:30:34 2019
Version 18.4.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.

ERROR:
ORA-01017: invalid username/password; logon denied


Enter user-name:

  • adr_, dg_ rman_, cm_, lsn_ : aliases for common oracle binaries
  • genpasswd : generates random passwords (default length 30)

# [ oracle@ludodb01:/home/oracle [22:32:35] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# genpasswd
+gagDCqVSgqHqsU+-IdeA0nx_-HVZ1

# [ oracle@ludodb01:/home/oracle [22:33:00] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# genpasswd 12
DiU9nHiwPB9y

  • lsoh: lists the Oracle Homes attached to the inventory

# [ oracle@ludodb01:/u01/app/oracle/diag/rdbms/cdb_fra1cw/CDB/trace [22:33:53] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# lsoh

HOME                        LOCATION                                                VERSION      EDITION
--------------------------- ------------------------------------------------------- ------------ ---------
OraGrid180                  /u01/app/18.0.0.0/grid                                  18.4.0.0.0   GRID
OraDB18000_home1            /u01/app/oracle/product/18.0.0.0/dbhome_1               18.4.0.0.0   DBMS EE

  • setoh: sets the Oracle Home given its name in the inventory

# [ oracle@ludodb01:/u01/app/oracle/diag/rdbms/cdb_fra1cw/CDB/trace [22:35:38] [18.4.0.0.0 [DBMS EE] SID=CDB] 0 ] #
# setoh OraGrid180
VERSION         = 18.4.0.0.0
ORACLE_HOME     = /u01/app/18.0.0.0/grid

 

You might want to install the same environment for oracle, grid (if you have role separation, it should be the case for Cloud DB Systems) and (eventually) root.

I am curious to know if it works well for your environment.

Cheers

Ludo

(unsupported) DST_UPGRADE_STATE is DATAPUMP(1) but no data pump jobs are running. How to fix?

$
0
0

This blog post contains unsupported commands. Do not try to use them without the supervision of Oracle Support!

I have just run across an Oracle Database who had a broken configuration in database_properties.

The database was in process of being upgraded to 18c, but the DST upgrade step was not working because of wrong entries in the view DATABASE_PROPERTIES:

SQL> SELECT version FROM v$timezone_file;

       VERSION
--------------
            14


SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
2 FROM DATABASE_PROPERTIES
3 WHERE PROPERTY_NAME LIKE 'DST_%'
4 ORDER BY PROPERTY_NAME;

PROPERTY_NAME               VALUE
--------------------------- ------------------------------
DST_PRIMARY_TT_VERSION      14
DST_SECONDARY_TT_VERSION    4
DST_UPGRADE_STATE           DATAPUMP(1)

The MOS note Updating the RDBMS DST version in 12c Release 1 (12.1.0.1 and up) using DBMS_DST (Doc ID 1509653.1) states that I had to check note How To Cleanup Orphaned DataPump Jobs In DBA_DATAPUMP_JOBS ? (Doc ID 336014.1) to solve the problem.

In fact, there should have  been an orphan data pump job trying to import the timezone file. But in my case, no jobs at all, do data pump job tables.

Second, the secondary time zone being lower than the primary one was, to me, sign of an old upgrade went wrong.

Trying to begin a new prepare phase was failing with:

SQL> exec DBMS_DST.BEGIN_PREPARE(31);
BEGIN DBMS_DST.BEGIN_PREPARE(31); END;

*
ERROR at line 1:
ORA-56920: a prepare or upgrade window or an on-demand or datapump-job loading of a secondary time zone data file is in an active state
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.DBMS_DST", line 1390
ORA-06512: at line 1

Trying to end the old one was failing as well:

SQL> EXEC DBMS_DST.END_PREPARE;
BEGIN DBMS_DST.END_PREPARE; END;

*
ERROR at line 1:
ORA-56924: prepare window does not exist
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.DBMS_DST", line 1470
ORA-06512: at line 1

Trying to unload the secondary was failing as well:

SQL> exec dbms_dst.UNLOAD_SECONDARY
BEGIN dbms_dst.UNLOAD_SECONDARY; END;

*
ERROR at line 1:
ORA-56938: no secondary time zone data file being loaded by on-demand or a datapump job
ORA-06512: at "SYS.DBMS_DST", line 1975
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.DBMS_DST", line 1950
ORA-06512: at line 1

I double-checked ALL the notes to clean-up the situation and made sure that there was nothing actually running regarding a DST upgrade.

I am pretty evil trying unsupported stuff. So I have decided to check the underlying table:

sys@ACCINT:SQL> select text from dba_views where view_name='DATABASE_PROPERTIES';

TEXT
--------------------------------------------------------------------------------
select name, value$, comment$
  from x$props

Fixed tables are not writable, but sys.props$ is, and it was containing the same bad data:

NAME                                          
------------------------------------------
VALUE$                                    
------------------------------------------
COMMENT$                                  
------------------------------------------
DST_UPGRADE_STATE                         
DATAPUMP(1)                               
State of Day Light Saving Time Upgrade    
                                          
DST_PRIMARY_TT_VERSION                    
14                                        
Version of primary timezone data file     
                                          
DST_SECONDARY_TT_VERSION                  
4                                         
Version of secondary timezone data file

So I did what I knew was wrong, after taking a guaranteed restore point. Do not try this at home without the supervision of Oracle Support!

SQL> update props$ set value$=0 where name='DST_SECONDARY_TT_VERSION';

1 row updated.

SQL> update props$ set value$='NONE' where name='DST_UPGRADE_STATE';

1 row updated.

SQL> select * from props$ where name like 'DST%';

NAME                                     
-----------------------------------------
VALUE$                                   
-----------------------------------------
COMMENT$                                 
-----------------------------------------
DST_UPGRADE_STATE                        
NONE                                     
State of Day Light Saving Time Upgrade   
                                         
DST_PRIMARY_TT_VERSION                   
14                                       
Version of primary timezone data file    
                                         
DST_SECONDARY_TT_VERSION                 
0                                        
Version of secondary timezone data file  


3 rows selected.

SQL> commit;

Commit complete.

Trying again:

SQL> exec DBMS_DST.BEGIN_PREPARE(31);
A prepare window has been successfully started.

PL/SQL procedure successfully completed.

The rest of the upgrade procedure went smoothly.

Ludovico

Oracle Grid Infrastructure 19c does not configure the local-mode automaton by default. How to add it?

$
0
0

I have been installing Grid Infrastructure 18c for a while, then switched to 19c when it became GA.

At the beginning I have been overly enthusiast by the shorter installation time:

The GIMR is now optional, that means that deciding to install it is a choice of the customer, and a customer might like to keep it or not, depending on its practices.

Not having the GIMR by default means not having the local-mode automaton. This is also not a problem at all. The default configuration is good for most customers and works really well.

This new simplified configuration reduces some maintenance effort at the beginning, but personally I use a lot the local-mode automaton for out-of-place patching of Grid Infrastructure (read my blog posts to know why I really love the local-mode automaton), so it is something that I definitely need in my clusters.

A choice that makes sense for Oracle and most customers

Oracle vision regarding Grid Infrastructure consists of a central management of clusters, using the Oracle Domain Services Cluster. In this kind of deployment, the Management Repository, TFA, and many other services, are centralized. All the clusters use those services remotely instead of having them configured locally. The local-mode automaton is no exception: the full, enterprise-grade version of Fleet Patching and Provisioning (FPP, formerly Rapid home provisioning or RHP) allows much more than just out-of-place patching of Grid Infrastructure, so it makes perfectly sense to avoid those configurations everywhere, if you use a Domain Cluster architecture. Read more here.

Again, as I said many times in the past, doing out-of-place patching is the best approach in my opinion, but if you keep doing in-place patching, not having the local-mode automaton is not a problem at all and the default behavior in 19c is a good thing for you.

I need local-mode automaton on 19c, what I need to do at install time?

If you have many clusters, you are not installing them by hand with the graphic interface (hopefully!). In the responseFile for 19c Grid Infrastructure installation, this is all you need to change comparing to a 18c:

$ diff grid_install_template_18.rsp grid_install_template_19.rsp
1c1
< oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v18.0.0
---
> oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
25a26
> oracle.install.crs.configureGIMR=true
27c28
< oracle.install.crs.config.storageOption=
---
> oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE

as you can see, also Flex ASM is not part of the game by default in 19c.

Once you specify in the responseFile that you want GIMR, then the local-mode automaton is installed  as well by default.

I installed GI 19c without GIMR and local-mode automaton. How can I add them to my new cluster?

First, recreate the empty MGMTDB CDB by hand:

$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true \
 -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb \
 -storageType ASM -diskGroupName +MGMT \
 -datafileJarLocation $OH/assistants/dbca/templates \
 -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck

Prepare for db operation
10% complete
Registering database with Oracle Grid Infrastructure
14% complete
Copying database files
43% complete
Creating and starting Oracle instance
45% complete
49% complete
54% complete
58% complete
62% complete
Completing Database Creation
66% complete
69% complete
71% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
 /u01/app/oracle/cfgtoollogs/dbca/_mgmtdb.
Database Information:
Global Database Name:_mgmtdb
System Identifier(SID):-MGMTDB
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb2.log" for further details.

Then, configure the PDB for the cluster. Pay attention to the -local switch that is not documented (or at least it does not appear in the inline help):

$ mgmtca -local

After that, you might check that you have the PDB for your cluster inside the MGMTDB, I’ll skip this step.

Before creating the rhpserver (local-mode automaton resource), we need the volume and filesystem to make it work (read here for more information).

The volume:

ASMCMD> volcreate -G MGMT -s 1536M --column 8 --width 1024k --redundancy unprotected GHCHKPT

ASMCMD> volinfo --all
Diskgroup Name: MGMT

         Volume Name: GHCHKPT
         Volume Device: /dev/asm/ghchkpt-303
         State: ENABLED
         Size (MB): 1536
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

The filesystem:

(oracle)$ mkfs -t acfs /dev/asm/ghchkpt-303

(root)# $CRS_HOME/bin/srvctl add filesystem -d /dev/asm/ghchkpt-303 -m /opt/oracle/rhp_images/chkbase -u oracle -fstype ACFS
(root)# $CRS_HOME/bin/srvctl enable filesystem -volume ghchkpt -diskgroup MGMT
(root)# $CRS_HOME/bin/srvctl start filesystem -volume ghchkpt -diskgroup MGMT

Finally, create the local-mode automaton resource:

(root)# $CRS_HOME/bin/srvctl add rhpserver -local -storage /opt/oracle/rhp_images

Again, note that there is a -local switch that is not documented. Specifying it will create the resource as a local-mode automaton and not as a full FPP Server (or RHP Server, damn, this change of name gets me mad when I write blog posts about it 🙂 ).

HTH

Ludovico

Oracle SW_ONLY install leads to relink with rac_off at every attachHome

$
0
0

OK, I really do not know what other title I should use for this post.

I have developed and presented a few times my personal approach to Oracle Home provisioning and patching. You can read more in this series.

With this approach:

  • I install the software (either GI or RDBMS) with the option SW_ONLY once
  • I patch it to the last version
  • I create a golden image that I evolve for the rest of the release lifecycle

When I need to install it, I just unzip the golden image and attach it to the Central Inventory.

I have discovered quite longtime ago that, every time I was attaching the home to the inventory, the binaries were relinked with rac_off, disregarding the fact that the home that I zipped actually had RAC enabled. This is quite annoying at my work at CERN, as all our databases are RAC.

So my solution to the problem is to detect if the server is on a cluster, and relink on the fly:

### EARLIER, IN THE ENVIRONMENT SCRIPTS
if [ -f /etc/oracle/olr.loc ] ; then
        export CRS_EXISTS=1
else
        export CRS_EXISTS=0
fi

### LATER, AFTER ATTACHING THE ORACLE_HOME:
pushd $ORACLE_HOME/rdbms/lib
if [ $CRS_EXISTS -eq 1 ] ; then
	make -f ins_rdbms.mk rac_on
else
	make -f ins_rdbms.mk rac_off
fi
make -f ins_rdbms.mk ioracle

This is a simplified snippet of my actual code, but it gives the idea.

What causes the relink with rac_off?

I have discovered recently that the steps used by the runInstaller process to attach the Oracle Home are described in this file:

$ORACLE_HOME/inventory/make/makeorder.xml

and in my case, for all my golden images, it contains:

<ohmd:MAKE
MAKEPATH="/usr/bin/make" FILENAME="rdbms/lib/ins_rdbms.mk" >
<ohmd:TARGET ACTIONTYPE="INSTALL" TARGETNAME="rac_off" >
<ohmd:INPUT_LIST>
<ohmd:INPUT VAL="ORACLE_HOME=%ORACLE_HOME%"/>
</ohmd:INPUT_LIST>
<ohmd:COMP_LIST>
<ohmd:COMP NAME="oracle.rdbms" VERSION="18.0.0.0.0"/>
</ohmd:COMP_LIST>
</ohmd:TARGET>
</ohmd:MAKE>

So, it does not matter how I prepare my images: unless I change this file and put rac_on, the runInstaller keeps relinking with rac_off.

I have thought about changing the file, but then realized that I prefer to check and recompile at runtime, so I can reuse my images also for standalone servers (in case we need them).

Just to avoid surprises, it is convenient to check if a ORACLE_HOME is linked with RAC with this small function:

$ type isRACoh
isRACoh is a function
isRACoh ()
{
    OH2CHECK=${1:-$ORACLE_HOME};
    ar -t $OH2CHECK/rdbms/lib/libknlopt.a | grep --color=auto kcsm.o > /dev/null;
    if [ $? -eq 0 ]; then
        echo "Enabled";
    else
        echo "Disabled";
        false;
    fi
}

This is true especially for Grid Infrastructure golden images, as they have the very same behavior of RDBMS homes, with the exception that they might break out-of-place patching if RAC is not enabled: the second ASM instance will not mount because the first will be exclusively mounted without the RAC option.

 

HTH.

Ludovico

How to install and access Oracle Weblogic 12.2 in the Oracle Cloud Infrastructure

$
0
0

I put here the steps required to install and access Weblogic in the OCI (mostly for me in case I need to do it again 😉 ). The assumptions are:

  • you already have an account for the Oracle Cloud Infrastructure and you can access the OCI console
  • you already have a Compartment with a VCN and a subnet configured (for test purposes, a VCN created with the default values will be just fine)
  • you already have a keypair for your SSH client (id_rsa, id_rsa.pub)
  • you have an X server on your laptop (if you have Windows, I recommend MobaXTerm, but Xming or other servers are just fine)

Create the compute instance

  • Menu -> Core Infrastructure -> Compute -> Instances -> Create Instance
  • Choose a name for the Instance, all the other fields defaults are fine for test (Oracle Linux 7.6, VM.Standard2.1, etc.)
  • Paste your SSH public key
  • Optionally, under advanced/network, specify a different name for the VM
  • Click on Create to complete the creation

At some point you will have an instance “Green” ready to access:

Click on it and get the public address:

Using your SSH keypair, you can now access the instance with:

$ ssh opc@{public_ip}

 

Setup sshd for SSH tunneling and X11 forwarding

Edit as root the sshd_config:

$ sudo vi /etc/ssh/sshd_config

Modify it so that the following lines are present with these values:

AllowTcpForwarding yes
PermitOpen any
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no

Those values are required for X11 forwarding (required for the graphical installation) and for SSH tunneling (required to access  the Weblogic ports without exposing them over internet).

Then restart sshd:

$ sudo systemctl restart sshd

Install the packages for X11 

# sudo yum install xorg-x11-xauth.x86_64
# sudo yum install libXtst
# # optional, to test if X11 forwarding works
# sudo yum install xterm

At this point, it should be possible to forward X11. You can test by reconnecting with:

$ ssh -XC opc@{public_ip}

and then:

$ xterm

Create the oracle user

$ sudo su - root
# groupadd -g 54321 oinstall
# useradd -u 54321 -g oinstall oracle
# passwd oracle

At this point, you can reconnect using oracle directly, so X11 forward will work for the oracle user without any additional setup:

$ ssh -XC oracle@{public_ip}

 

Follow the canonical steps to install weblogic

If you do not know how to do that, follow this good tutorial by Tim Hall (oracle-base):

Oracle WebLogic Server (WLS) 12cR2 (12.2.1) Installation on Oracle Linux 6 and 7

 

Access the Weblogic console from outside Oracle Cloud

If you configured correctly sshd, once the Oracle Weblogic instance is configured and started, you can tunnel to the port (it should be 7001):

$ ssh -L 7001:{vm_name}:7001 oracle@{public_ip}

And be able to browse from your laptop using localhost:7001:

HTH

Ludovico

Steps to remove/add node from a cluster if RHP fails to move gihome

$
0
0

I am getting more and more experience patching clusters with the local-mode automaton. The whole process would be very complex, but the local-mode automaton makes it really easy.

I have had nevertheless a couple of clusters where the process did not work:

#1: The very first cluster that I installed in 18c

This cluster has “kind of failed” patching the first node. Actually, the rhpctl command exited with an error:

$ rhpctl move gihome -sourcehome /u01/crs/crs1830 -desthome /u01/crs/crs1860 -node server1
server1.cern.ch: Audit ID: 2
server1.cern.ch: verifying versions of Oracle homes ...
server1.cern.ch: verifying owners of Oracle homes ...
server1.cern.ch: verifying groups of Oracle homes ...
server1.cern.ch: starting to move the Oracle Grid Infrastructure home from "/u01/crs/crs1830" to "/u01/crs/crs1860" on server cluster "AISTEST-RAC16"
[...]
2019/07/08 09:45:06 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
PRCG-1239 : failed to close a proxy connection
Connection refused to host: server1.cern.ch; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
PRCG-1079 : Internal error: ClientFactoryImpl-submitAction-error1
PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]

But actually, the helper lept running and configured everything properly:

$ tail -f /ORA/dbs01/oracle/crsdata/server1/crsconfig/crs_postpatch_server1_2019-07-08_09-41-36AM.log
2019-07-08 09:55:25:
2019-07-08 09:55:25: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:SUCCESS
2019-07-08 09:55:25: Executing cmd: /u01/crs/crs1860/bin/clsecho -p has -f clsrsc -m 672
2019-07-08 09:55:25: Executing cmd: /u01/crs/crs1860/bin/clsecho -p has -f clsrsc -m 672
2019-07-08 09:55:25: Command output:
>  CLSRSC-672: Post-patch steps for patching GI home successfully completed.
>End Command output
2019-07-08 09:55:25: CLSRSC-672: Post-patch steps for patching GI home successfully completed.

The cluster was OK on the first node, with the correct patch level. The second node, however, was filing with:

$  rhpctl move gihome -sourcehome /u01/crs/crs1830 -desthome /u01/crs/crs1860 -node server2
server1.cern.ch: retrieving status of databases ...
server1.cern.ch: retrieving status of services of databases ...
PRCT-1011 : Failed to run "rhphelper". Detailed error: <HLP_EMSG>,RHPHELP_procCmdLine-05,</HLP_EMSG>,<HLP_VRES>3</HLP_VRES>,<HLP_IEEMSG>,PRCG-1079 : Internal error: RHPHELP122_main-01,</HLP_IEEMSG>,<HLP_ERES>1</HLP_ERES>

I am not sure about the cause, but let’s assume it is irrelevant for the moment.

#2: A cluster with new GI home not properly linked with RAC

This was another funny case, where the first node patched successfully, but the second one failed upgrading in the middle of the process with a java NullPointer exception. We did a few bad tries of prePatch and postPatch to solve, but after that the second node of the cluster was in an inconsistent state: in ROLLING_UPGRADE mode and not possible to patch anymore.

Common solution: removing the node from the cluster and adding it back

In both cases we were in the following situation:

  • one node was successfully patched to 18.6
  • one node was not patched and was not possible to patch it anymore (at least without heavy interventions)

So, for me, the easiest solution has been removing the failing node and adding it back with the new patched version.

Steps to remove the node

Although the steps are described here: https://docs.oracle.com/en/database/oracle/oracle-database/18/cwadd/adding-and-deleting-cluster-nodes.html#GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A, there are a few differences that I will highlight:

Stop of the cluster:

(root)# crsctl stop crs

The actual procedure to remove a node asks to deconfigure the databases and managed homes from the active cluster version. But as we manage our homes with golden images, we do not need this; we rather want to keep all the entries in the OCR so that when we add it back, everything is in place.

Once stopped the CRS, we have deinstalled the CRS home on the failing node:

(oracle)$ $OH/deinstall/deinstall -local

This  complained about the CRS that was down, but continued and ask for this script to be executed:

/u01/crs/crs1830/crs/install/rootcrs.sh -force  -deconfig -paramfile "/tmp/deinstall2019-07-08_11-37-20AM/response/deinstall_1830.rsp"

We’ve got errors also for this script, but the remove process was OK afterall.

Then, from the surviving node:

root # crsctl delete node -n server2
oracle $ srvctl stop vip -vip server2
root $ srvctl remove vip -vip server2

Adding the node back

From the surviving node, we ran gridSetup.sh and followed the steps to ad the node.

Wait before running root.sh.

In our case, we have originally installed the cluster starting with a SW_ONLY install. This type of installation keeps some leftovers in the configuration files that prevent the root.sh from configuring the cluster…we have had to modify rootconfig.sh:

check/modify /u01/crs/crs1860/crs/config/rootconfig.sh and change this:
# before:
# SW_ONLY=true
# after:
SW_ONLY=false

then, after running root.sh and the config tools, everything was back as before removing the node form the cluster.

For one of the clusters , both nodes were at the same patch level, but the cluster was still in ROLLING_PATCH mode. So we have had to do a

(root) # crsctl stop rollingpatch

Ludo

Install and configure CMAN 19c in the Oracle Cloud, step by step

$
0
0

Installing and configuring CMAN is a trivial activity, but having the steps in one place is better than reinventing the wheel.

Prepare for the install

Download the Oracle Client 19.3.0.0 in the Oracle Database 19c download page.

Choose this one: LINUX.X64_193000_client.zip (64-bit) (1,134,912,540 bytes) , not the one named “LINUX.X64_193000_client_home.zip” because it is a preinstalled home that does not contain the CMAN tools.

Access the OCI Console and create a new Compute instance. The default  configuration is OK, just make sure that it is Oracle Linux 7 🙂

Do not forget to add your SSH Public Key to access the VM via SSH!

Access the VM using

ssh opc@{public_ip}

Copy the Oracle Client zip in /tmp using your favorite scp program.

Install CMAN

Follow these steps to install CMAN:

# become root
sudo su - root

# install some prereqs (packages, oracle user, kernel params, etc.):
yum install oracle-database-preinstall-19c.x86_64

# prepare the base directory:
mkdir /u01
chown oracle:oinstall /u01

# become oracle
su - oracle

# prepare the Oracle Home dir
mkdir -p /u01/app/oracle/product/cman1930

# unzip the Client install binaries
mkdir -p $HOME/stage
cd $HOME/stage
unzip /tmp/LINUX.X64_193000_client.zip

# prepare the response file:
cat <<EOF > $HOME/cman.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_clientinstall_response_schema_v19.0.0
ORACLE_HOSTNAME=$(hostname)
UNIX_GROUP_NAME=oinstall
INVENTORY_LOCATION=/u01/app/oraInventory
SELECTED_LANGUAGES=en
ORACLE_HOME=/u01/app/oracle/product/cman1930
ORACLE_BASE=/u01/app/oracle
oracle.install.client.installType=Custom
oracle.install.client.customComponents="oracle.sqlplus:19.0.0.0.0","oracle.network.client:19.0.0.0.0","oracle.network.cman:19.0.0.0.0","oracle.network.listener:19.0.0.0.0"
EOF

# install!
$HOME/stage/client/runInstaller -silent -responseFile $HOME/cman.rsp  ORACLE_HOME_NAME=cman1930

# back as root:
exit

# finish the install
/u01/app/oraInventory/orainstRoot.sh
/u01/app/oracle/product/cman1930/root.sh

 

Basic configuration

# as oracle:

mkdir -p /u01/app/oracle/network/admin
export TNS_ADMIN=/u01/app/oracle/network/admin

cat <<EOF > $TNS_ADMIN/cman-test.ora
cman-test = (configuration=
  (address=(protocol=tcp)(host=$(hostname))(port=1521))
  (parameter_list =
    (log_level=ADMIN)
    (max_connections=1024)
    (idle_timeout=0)
    (registration_invited_nodes = *)
    (inbound_connect_timeout=0)
    (session_timeout=0)
    (outbound_connect_timeout=0)
    (max_gateway_processes=16)
    (min_gateway_processes=2)
    (remote_admin=on)
    (trace_level=off)
    (max_cmctl_sessions=4)
    (event_group=init_and_term,memory_ops)
  )
  (rule_list=
    (rule=
       (src=*)(dst=*)(srv=*)(act=accept)
       (action_list=(aut=off)(moct=0)(mct=0)(mit=0)(conn_stats=on))
  ) )
)
EOF

echo "IFILE=${TNS_ADMIN}/cman-test.ora" >> $TNS_ADMIN/cman.ora

This will create a CMAN configuration named cman-test. Beware that it is very basic and insecure. Please read the CMAN documentation if you want something more secure or sophisticated.

The advantage of having the TNS_ADMIN outside the Oracle Home is that if you need to patch CMAN, you can do it out-of-place without the need to copy the configuration files somewhere else.

The advantage of using IFILE inside cman.ora, is that you can manage easily different CMAN configurations in the same host without editing directly cman.ora, with the risk of messing it up.

Preparing the start/stop script

Create a file /u01/app/oracle/scripts/cman_service.sh with this content:

#!/bin/bash -l

LOCAL_PARSE_OPTIONS="a:c:o:"

Usage () {
        cat <<EOF

        Purpose   : Start/stop a CMAN configuration

        Usage: `basename $0` -a {start|stop|reload|restart|status} -c <config_name> -o <oracle_home>

        Options:
                -a action           One in start|stop|reload|restart|status
                -c config_name      Name of the cman instance (e.g. ais-prod, gen-prod, etc.)
                -o oracle_home      The ORACLE_HOME path that must be used for the operation (e.g. cman1930)
EOF
}


CENTRAL_CONFIG_DIR=/ORA/dbs01/oracle/network/admin

while getopts ":${LOCAL_PARSE_OPTIONS}" opt ; do
        case $opt in
                a)
                        L_Action=$OPTARG
                        ;;
                c)
                        L_Config=$OPTARG
                        ;;
                o)
                        L_OH=$OPTARG
                        ;;
                \?)
                        eerror "Invalid option: -$OPTARG"
                        exit 1
                        ;;
                :)
                        eerror "Option -$OPTARG requires an argument."
                        exit 1
                        ;;
        esac
done

if [ ! $L_Config ] ; then
    Usage
        eerror "Please specify a configuration name with -c. Possible values are: "
        ls -1 $CENTRAL_CONFIG_DIR | sed -e "s/\.ora//" | grep -v cman
        exit 1
fi


## if the install step was OK, we should have a valid OH installed with this name:
export ORACLE_HOME=$L_OH
if [ ! -f $ORACLE_HOME/bin/cmctl ] ; then
        Usage
        echo "Please set a valid ORACLE_HOME name with -o."
        exit 1
fi


export TNS_ADMIN=$CENTRAL_CONFIG_DIR
case $L_Action in
        start)
                $OH/bin/cmctl startup -c $L_Config
                ;;
        stop)
                $OH/bin/cmctl shutdown -c $L_Config
                ;;
        reload)
                $OH/bin/cmctl reload -c $L_Config
                ;;
        restart)
                $OH/bin/cmctl shutdown -c $L_Config
                sleep 1
                $OH/bin/cmctl startup -c $L_Config
                ;;
        status)
                $OH/bin/cmctl show status -c $L_Config
                # do it again for the exit code
                $OH/bin/cmctl show status -c $L_Config | grep "The command completed successfully." >/dev/null
                ;;
        *)
                echo "Invalid action"
                exit 1
                ;;
esac

This is at the same time ORACLE_HOME agnostic and configuration agnostic.

Make it executable:

chmod +x /u01/app/oracle/scripts/cman_service.sh

and try to start CMAN:

$ /u01/app/oracle/scripts/cman_service.sh -o /u01/app/oracle/product/cman1930 -c cman-test -a start
VERSION         = 19.3.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/cman1930
VERSION         = 19.3.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/cman1930

CMCTL for Linux: Version 19.0.0.0.0 - Production on 12-JUL-2019 09:23:50

Copyright (c) 1996, 2019, Oracle.  All rights reserved.

Current instance cman-test is not yet started
Connecting to (DESCRIPTION=(address=(protocol=tcp)(host=ocf-cman-1)(port=1521)))
Starting Oracle Connection Manager instance cman-test. Please wait...
CMAN for Linux: Version 19.0.0.0.0 - Production
Status of the Instance
----------------------
Instance name             cman-test
Version                   CMAN for Linux: Version 19.0.0.0.0 - Production
Start date                12-JUL-2019 09:23:50
Uptime                    0 days 0 hr. 0 min. 9 sec
Num of gateways started   2
Average Load level        0
Log Level                 ADMIN
Trace Level               OFF
Instance Config file      /u01/app/oracle/product/cman1930/network/admin/cman.ora
Instance Log directory    /u01/app/oracle/diag/netcman/ocf-cman-1/cman-test/alert
Instance Trace directory  /u01/app/oracle/diag/netcman/ocf-cman-1/cman-test/trace
The command completed successfully.

Stop should work as well:

$ /u01/app/oracle/scripts/cman_service.sh -o /u01/app/oracle/product/cman1930 -c cman-test -a stop
VERSION         = 19.3.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/cman1930
VERSION         = 19.3.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/cman1930

CMCTL for Linux: Version 19.0.0.0.0 - Production on 12-JUL-2019 09:28:34

Copyright (c) 1996, 2019, Oracle.  All rights reserved.

Current instance cman-test is already started
Connecting to (DESCRIPTION=(address=(protocol=tcp)(host=ocf-cman-1)(port=1521)))
The command completed successfully.

Add the service in systemctl

# as root:

cat <<EOF > /etc/systemd/system/cman-test.service
[Unit]
Description=CMAN Service for cman-test
Documentation=http://www.ludovicocaldara.net/dba/cman-oci-install
After=network-online.target

[Service]
User=oracle
Group=oinstall
LimitNOFILE=10240
MemoryLimit=8G
RestartSec=30s
StartLimitInterval=1800s
StartLimitBurst=20
ExecStart=/u01/app/oracle/scripts/cman_service.sh -c cman-test -a start -o /u01/app/oracle/product/cman1930
ExecReload=/u01/app/oracle/scripts/cman_service.sh -c cman-test -a reload -o /u01/app/oracle/product/cman1930
ExecStop=/u01/app/oracle/scripts/cman_service.sh -c cman-test -a stop -o /u01/app/oracle/product/cman1930
KillMode=control-group
Restart=on-failure
Type=forking

[Install]
WantedBy=multi-user.target
Alias=service-cman-test.service
EOF

/usr/bin/systemctl enable cman-test.service

# start
/usr/bin/systemctl start cman-test
# stop
/usr/bin/systemctl stop cman-test

Open firewall ports

By default, new OL7 images use firewalld. Just open the port 1521 from the public zone:

# as root:
firewall-cmd --permanent --add-port=1521/tcp
firewall-cmd --reload

 

Bonus: have a smart environment!

# as root:
yum install -y git rlwrap

# Connect as oracle
sudo su - oracle

# Clone this repository
git clone https://github.com/ludovicocaldara/COE.git

# Enable the profile scripts
echo ". ~/COE/profile.sh" >> $HOME/.bash_profile

# set the cman1930 home by default:
echo "setoh cman1930" >> $HOME/.bash_profile
echo "export TNS_ADMIN=/u01/app/oracle/network/admin" >> $HOME/.bash_profile

# Load the new profile
. ~/.bash_profile

[root@ocf-cman-1 tmp]# su - oracle
Last login: Fri Jul 12 09:49:09 GMT 2019 on pts/0
VERSION         = 19.3.0.0.0
ORACLE_HOME     = /u01/app/oracle/product/cman1930

# [ oracle@ocf-cman-1:/home/oracle [09:49:54] [19.3.0.0.0 [CLIENT] SID="not set"] 0 ] #
# # ahhh, that;s satisfying

Ludo


Parameter REMOTE_LISTENER pointing to a TNS alias? Beware of how it registers.

$
0
0

On an Oracle Database instance, if I set:

alter system set remote_listener='cluster-scan:1521';

The instance tries to resolve the cluster-scan name to detect if it is a SCAN address.
So, after it solves, it stores all the addresses it gets and registers to them.
I can check which addresses there are with this query:

SQL>  select type, value from v$listener_network where type='REMOTE LISTENER';

TYPE             VALUE
---------------- ---------------------------------------------------------------------------------------------------
REMOTE LISTENER  (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.1)(PORT=1521)))
REMOTE LISTENER  (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.2)(PORT=1521)))
REMOTE LISTENER  (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.3)(PORT=1521)))

In this case, the instance registers to the three addresses discovered, which is OK: all three SCAN listeners will get service updates from the instance.

But if I have this TNS alias:

REMOTE_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(PORT=1521)(HOST=cluster-scan)))

and I set:

alter system set remote_listener='remote_listener';

I get:

SQL>  select type, value from v$listener_network where type='REMOTE LISTENER';

TYPE             VALUE
---------------- ---------------------------------------------------------------------------
REMOTE LISTENER  (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(PORT=1521)(HOST=cluster-scan)))

the result is that the instance registers only at the first IP fot from the DNS, leaving the other SCANs without the service registration and thus random

ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

This is in my opinion quite annoying, as my goal here was to have all the DBs set with:

local_listener=local_listener
remote_listener=remote_listener

in order to facilitate changes of ports, database migrations from different clusters, clones, etc.

So the solution is either to revert to the syntax “cluster-scan:port”, or specifying explicitly all the endpoints in the address list:

REMOTE_LISTENER = (DESCRIPTION= (ADDRESS_LIST=
  (ADDRESS= (PROTOCOL=TCP) (PORT=1521) (HOST=10.0.0.1))
  (ADDRESS= (PROTOCOL=TCP) (PORT=1521) (HOST=10.0.0.2))
  (ADDRESS= (PROTOCOL=TCP) (PORT=1521) (HOST=10.0.0.3))
 ))

I am sure it is “working as designed”, but I wonder if it could be an enhancement to have the address expended fully also in case of TNS alias….
Or… do you know any way to do it from a TNS alias without having the full IP list?

Cheers

Ludo

Checking usage of HugePages by Oracle databases in Linux environments

$
0
0

Yesterday several databases on one server started logging errors in the alert log:

ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2

That means not enough contiguous free memory in the OS. The first thing that I have checked has been of course the memory, and the used huge pages:

# [ oracle@oraserver1:/home/oracle [10:45:46] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ free
              total        used        free      shared  buff/cache   available
Mem:      528076056   398142940     3236764   119855448   126696352     5646964
Swap:      16760828    11615324     5145504

# [ oracle@oraserver1:/home/oracle [10:46:47] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ cat /proc/meminfo | grep Huge
HugePages_Total:   180000
HugePages_Free:    86029
HugePages_Rsvd:    11507
HugePages_Surp:        0
Hugepagesize:       2048 kB

The memory available (last column in the free command) was indeed quite low, but still plenty of space in the huge pages (86k pages free out of 180k).

The usage by Oracle instances:

# [ oracle@oraserver1:/home/oracle [10:45:39] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ sh mem.sh
DB12 : 54081544
DB22 : 37478820
DB32 : 67970828
DB42 : 14846552
DB52 : 16326380
DB62 : 15122048
DB82 : 56900472
DB92 : 14401080
DBA2 : 12622736
DBB2 : 14379916
DBC2 : 46078336
DBD2 : 46137728
DB72 : 37351336
total :  433697776

You can get the code of mem.sh in this post.

Regarding pure shared memory usage, the situation was what I was expecting:

$ ipcs -m | awk 'BEGIN{a=0} {a+=$5} END{print a}'
369394520064

360G of shared memory usage, much more than what was allocated in the huge pages.

I have compared the situation with the other node in the cluster: it had more memory allocated by the databases (because of more load on it), more huge page usage and less 4k pages consumption overall.

$ sh mem.sh
DB12 : 78678000
DB22 : 14220000
DB32 : 14287528
DB42 : 12369352
DB52 : 14868596
DB62 : 14633984
DB82 : 54316104
DB92 : 86148332
DBA2 : 61473288
DBB2 : 68678788
DBC2 : 9831288
DBD2 : 64759352
DB72 : 68114604
total :  562379216

$ free
              total        used        free      shared  buff/cache   available
Mem:      528076056   402288800    17100464     5818032   108686792   114351784
Swap:      16760828       47360    16713468

$ cat /proc/meminfo | grep Huge
AnonHugePages:     10240 kB
HugePages_Total:   176654
HugePages_Free:    15557
HugePages_Rsvd:    15557
HugePages_Surp:        0
Hugepagesize:       2048 kB

So I was wondering if all the DBs were property allocating the SGA in huge pages or not.

This redhat page has been quite useful to create a quick snippet to check the huge page memory allocation per process:

# [ oracle@oraserver1:/home/oracle [10:55:27] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ cat /proc/707/numa_maps | grep -i hug
60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=57 N0=1 kernelpagesize_kB=2048
70000000 default file=/SYSV00000000\040(deleted) huge dirty=1525 mapmax=57 N0=743 N1=782 kernelpagesize_kB=2048
c60000000 interleave:0-1 file=/SYSV0b46df00\040(deleted) huge dirty=1 mapmax=57 N0=1 kernelpagesize_kB=2048


# [ oracle@oraserver1:/home/oracle [10:56:39] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ function pshugepage () {
> HUGEPAGECOUNT=0
> for num in `grep 'huge.*dirty=' /proc/$@/numa_maps | awk '{print $5}' | sed 's/dirty=//'` ; do
> HUGEPAGECOUNT=$((HUGEPAGECOUNT+num))
> done
> echo process $@ using $HUGEPAGECOUNT huge pages
> }

# [ oracle@oraserver1:/home/oracle [10:57:09] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ pshugepage 707
process 707 using 1527 huge pages


# [ oracle@oraserver1:/home/oracle [10:57:11] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ for pid in `ps -eaf | grep [p]mon | awk '{print $2}'` ; do pshugepage $pid ; done
process 707 using 1527 huge pages
process 3685 using 2409 huge pages
process 16092 using 3056 huge pages
process 55718 using 0 huge pages
process 58490 using 0 huge pages
process 70583 using 0 huge pages
process 94479 using 1135 huge pages
process 98216 using 0 huge pages
process 98755 using 0 huge pages
process 100245 using 0 huge pages
process 100265 using 0 huge pages
process 100270 using 0 huge pages
process 101681 using 0 huge pages
process 179079 using 1699 huge pages
process 189585 using 14566 huge pages

It has been easy to spot the databases not using huge pages at all:

# [ oracle@oraserver1:/home/oracle [10:58:26] [19.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ ps -eaf | grep [p]mon
oracle      707      1  0 Sep30 ?        00:23:55 ora_pmon_DB12
oracle     3685      1  0 Nov01 ?        00:09:17 ora_pmon_DB22
oracle    16092      1  0 Oct15 ?        00:04:15 ora_pmon_DB32
oracle    55718      1  0 Aug12 ?        00:08:25 asm_pmon_+ASM2
oracle    58490      1  0 Aug12 ?        00:08:24 apx_pmon_+APX2
oracle    70583      1  0 Aug12 ?        00:57:55 ora_pmon_DB42
oracle    94479      1  0 Oct02 ?        00:32:03 ora_pmon_DB52
oracle    98216      1  0 Aug12 ?        00:58:36 ora_pmon_DB62
oracle    98755      1  0 Aug12 ?        00:59:27 ora_pmon_DB82
oracle   100245      1  0 Aug12 ?        00:56:52 ora_pmon_DB92
oracle   100265      1  0 Aug12 ?        00:51:54 ora_pmon_DBA2
oracle   100270      1  0 Aug12 ?        00:54:57 ora_pmon_DBB2
oracle   101681      1  0 Aug12 ?        00:56:55 ora_pmon_DBC2
oracle   179079      1  0 Sep10 ?        00:35:17 ora_pmon_DBD2
oracle   189585      1  0 Nov01 ?        00:09:34 ora_pmon_DB72

Indeed, after stopping them, the huge page usage has not changed:

# [ oracle@oraserver1:/home/oracle [11:01:52] [11.2.0.4.0 [DBMS EE] SID=DB62] 1 ] #
$ srvctl stop instance -d DB6_SITE1 -i DB62

# [ oracle@oraserver1:/home/oracle [11:02:24] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl stop instance -d DB4_SITE1 -i DB42

# [ oracle@oraserver1:/home/oracle [11:03:29] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl stop instance -d DB8_SITE1 -i DB82

# [ oracle@oraserver1:/home/oracle [11:06:36] [11.2.0.4.0 [DBMS EE] SID=DB62] 130 ] #
$ srvctl stop instance -d DB9_SITE1 -i DB92

# [ oracle@oraserver1:/home/oracle [11:07:16] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl stop instance -d DBA_SITE1 -i DBA2

# [ oracle@oraserver1:/home/oracle [11:07:56] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl stop instance -d DBB_SITE1 -i DBB2

# [ oracle@oraserver1:/home/oracle [11:08:42] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl stop instance -d DBC_SITE1 -i DBC2

# [ oracle@oraserver1:/home/oracle [11:09:16] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ cat /proc/meminfo | grep Huge
HugePages_Total:   180000
HugePages_Free:    86029
HugePages_Rsvd:    11507
HugePages_Surp:        0
Hugepagesize:       2048 kB

But after starting them back I could see the new huge pages reserved/allocated:

# [ oracle@oraserver1:/home/oracle [11:10:35] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DB6_SITE1 -i DB62

# [ oracle@oraserver1:/home/oracle [11:12:14] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DB4_SITE1 -i DB42

# [ oracle@oraserver1:/home/oracle [11:12:54] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DB8_SITE1 -i DB82

# [ oracle@oraserver1:/home/oracle [11:13:41] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DB9_SITE1 -i DB92

# [ oracle@oraserver1:/home/oracle [11:14:43] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DBA_SITE1 -i DBA2

# [ oracle@oraserver1:/home/oracle [11:15:25] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DBB_SITE1 -i DBB2

# [ oracle@oraserver1:/home/oracle [11:15:54] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ srvctl start instance -d DBC_SITE1 -i DBC2

# [ oracle@oraserver1:/home/oracle [11:17:49] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ cat /proc/meminfo | grep Huge
HugePages_Total:   180000
HugePages_Free:    72820
HugePages_Rsvd:    68961
HugePages_Surp:        0
Hugepagesize:       2048 kB

# [ oracle@oraserver1:/home/oracle [11:17:54] [11.2.0.4.0 [DBMS EE] SID=DB62] 0 ] #
$ free
              total        used        free      shared  buff/cache   available
Mem:      528076056   392011828   123587116     5371848    12477112   126250868
Swap:      16760828      587308    16173520

The reason was that the server has been started without huge pages first, and after a few instances started, the huge pages has been set.

HTH

Ludovico

 

Duplicating a DB and setting up Data Guard through CMAN and SSH tunnel

$
0
0

I am fascinated about the new Zero Downtime Migration tool that has been available since November 28th. Whilst I am still in the process of testing it, there is one big requirement that might cause some headache to some customers. It is about network connectivity:

Configuring Connectivity Between the Source and Target Database Servers

The source database server […] can connect to target database instance over target SCAN through the respecitve scan port and vice versa.
The SCAN of the target should be resolvable from the source database server, and the SCAN of the source should resolve from the target server.
Having connectivity from both sides, you can synchronize between the source database and target database from either side. […]

If you are taking cloud migrations seriously, you should have either a VPN site-to-site to the cloud, or a Fast Connect link. At CERN we are quite lucky to have a high bandwidth Fast Connect to OCI Frankfurt.

This requirement might be missing for many customers, so what is a possible solution to setup connectivity for database duplicates and Data Guard setups?

In the picture above you can see a classic situation, that usually has two problems that must be solved:

  • the SCAN addresses are private: not accessible from internet
  • there are multiple SCAN addresses, so tunneling through all of them might be complex

Is it possible to configure CMAN in front of the SCAN listener as a single IP entry and tunnel through SSH to this single IP?

I will show now how to achieve this configuration.

For sake of simplicity, I have put two single instances without SCAN and a CMAN installation on the database servers, but it will work with little modification using SCAN and RAC setups as well. Note that in a Cloud Infrastructure setup, this will require a correct setup of the TDE wallet on both the source and the destination.

Because I put everything on s single host, I have to setup CMAN to listen to another port, but having a separate host for CMAN is a better practice when it has to proxy to SCAN listeners.

Installing and configuring CMAN

The most important part of the whole setup is that the CMAN on the standby site must have a public IP address and open SSH port so that we can tunnel through it.

The on-premises CMAN must have open access to the standby CMAN port 22.

For both primary and standby site you can follow the instructions of my blog post: Install and configure CMAN 19c in the Oracle Cloud, step by step.

In my example, because I install CMAN on the same host of the DB, I configure CMAN to run on port 1522.

CMAN primary config:

cman-onprem = (configuration=
  (address=(protocol=tcp)(host=ludodb01.cern.ch)(port=1522))
  (parameter_list =
    (log_level=ADMIN)
    (max_connections=1024)
    (idle_timeout=0)
    (registration_invited_nodes = *)
    (inbound_connect_timeout=0)
    (session_timeout=0)
    (outbound_connect_timeout=0)
    (max_gateway_processes=16)
    (min_gateway_processes=2)
    (remote_admin=on)
    (trace_level=off)
    (max_cmctl_sessions=4)
    (event_group=init_and_term,memory_ops)
  )
  (rule_list=
    (rule=
       (src=*)(dst=*)(srv=*)(act=accept)
       (action_list=(aut=off)(moct=0)(mct=0)(mit=0)(conn_stats=on))
  ) )
)

CMAN standby config:

cman-cloud = (configuration=
  (address=(protocol=tcp)(host=ludodb02.cern.ch)(port=1522))
  (parameter_list =
    (log_level=ADMIN)
    (max_connections=1024)
    (idle_timeout=0)
    (registration_invited_nodes = *)
    (inbound_connect_timeout=0)
    (session_timeout=0)
    (outbound_connect_timeout=0)
    (max_gateway_processes=16)
    (min_gateway_processes=2)
    (remote_admin=on)
    (trace_level=off)
    (max_cmctl_sessions=4)
    (event_group=init_and_term,memory_ops)
  )
  (rule_list=
    (rule=
       (src=*)(dst=*)(srv=*)(act=accept)
       (action_list=(aut=off)(moct=0)(mct=0)(mit=0)(conn_stats=on))
  ) )
)

This configuration is not secure at all, you might want to secure it further in order to allow only the services needed for setting up Data Guard.

The registration of database services to CMAN through the the remote_listener parameter is optional, as I will register the entries statically in the listener and use a routed connection through CMAN.

Listener configuration

The listener must have a static entry for the database, so that duplicate and switchover work properly.

On primary add to listener.ora:

SID_LIST_LISTENER =
    (SID_LIST =
       (SID_DESC =
         (GLOBAL_DBNAME = db_onprem_DGMGRL.cern.ch)
         (ORACLE_HOME = /u01/app/oracle/product/rdbms1930 )
         (SID_NAME = db)
      )
       (SID_DESC =
         (GLOBAL_DBNAME = db_onprem.cern.ch)
         (ORACLE_HOME = /u01/app/oracle/product/rdbms1930 )
         (SID_NAME = db)
      )
    )

On standby:

SID_LIST_LISTENER =
    (SID_LIST =
       (SID_DESC =
         (GLOBAL_DBNAME = db_cloud_DGMGRL.cern.ch)
         (ORACLE_HOME = /u01/app/oracle/product/rdbms1930 )
         (SID_NAME = db)
      )
       (SID_DESC =
         (GLOBAL_DBNAME = db_cloud.cern.ch)
         (ORACLE_HOME = /u01/app/oracle/product/rdbms1930 )
         (SID_NAME = db)
      )
    )

In a RAC config, all the local listeners must be configured with the correct SID_NAME running on the host. Make sure to reload the listeners 😉

Creating the SSH tunnels

There must be two tunnels open: one that tunnels from on-premises to the cloud and the other that tunnels from the cloud to on-premises.

However, such tunnels can both be created from the on-premises CMAN host that has access to the cloud CMAN host:

# bind local port 1523 to remote port 1522
ssh -NnTf cman-cloud -L 1523:cman-cloud:1522

# bind remote port 1523 to local port 1522
ssh -NnTf cman-cloud -R 1523:cman-onprem:1522

in my case, the hostnames are:

ssh -NnTf ludodb02 -L 1523:ludodb02:1522
ssh -NnTf ludodb02 -R 1523:ludodb01:1522

Important: with CMAN on a host other than the DB server, the CMAN sshd must be configured to have GatewayPorts set to yes:

GatewayPorts yes

After the tunnels are open, any connections to the local CMAN server port 1523 will be forwarded to the remote CMAN port 1522.

Configuring the TNSNAMES to hop through CMAN and SSH tunnel

Both servers must have now one entry for the local database pointing to the actual SCAN (or listener for single instances) and one entry for the remote database pointing to local port 1523 and routing to the remote scan.

On-premises tnsnames.ora:

DB_ONPREM =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ludodb01.cern.ch)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = db_onprem.cern.ch)
    )
  )

DB_CLOUD =
  (DESCRIPTION =
   (SOURCE_ROUTE=yes)
   (ADDRESS_LIST=
     (ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1523))
     (ADDRESS=(PROTOCOL=tcp)(HOST=ludodb02)(PORT=1521)))
    (CONNECT_DATA =
      (SERVICE_NAME = db_cloud.cern.ch)
    )
  )

Cloud tnsnames.ora:

DB_CLOUD =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ludodb02.cern.ch)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = db_cloud.cern.ch)
    )
  )

DB_ONPREM=
 (DESCRIPTION=
   (SOURCE_ROUTE=yes)
   (ADDRESS_LIST=
     (ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1523))
     (ADDRESS=(PROTOCOL=tcp)(HOST=ludodb01)(PORT=1521)))
   (CONNECT_DATA=(SERVICE_NAME=db_onprem.cern.ch)))

After copying the passwordfile and starting nomount the cloud database, it should be possible from both sides to connect as SYSDBA to both DB_CLOUD and DB_ONPREM.

# [ oracle@ludodb02:/u01/app/oracle/network/admin [13:33:41] [19.3.0.0.0 [DBMS EE] SID=db] 0 ] #
#  s_ sys/change_on_install@db_onprem as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Dec 17 13:37:42 2019
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

# [ oracle@ludodb02:/u01/app/oracle/network/admin [13:37:44] [19.3.0.0.0 [DBMS EE] SID=db] 0 ] #
#  s_ sys/change_on_install@db_cloud as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Dec 17 13:37:53 2019
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

# [ oracle@ludodb02:/u01/app/oracle/network/admin [13:37:54] [19.3.0.0.0 [DBMS EE] SID=db] 0 ] #
# tnsping db_onprem

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 17-DEC-2019 13:38:05

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (SOURCE_ROUTE=yes)
(ADDRESS_LIST=
 (ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1523))
 (ADDRESS=(PROTOCOL=tcp)(HOST=ludodb01)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=db_onprem.cern.ch)))
OK (10 msec)

This configuration is ready for both duplicate from active database and for Data Guard.
I still have to figure out if it works with ZDM, but I think it is a big step towards establishing connection between on-premises and the Oracle Cloud when no VPN or Fast Connect are available.

Duplicate from active database

If the setup is correct, this should work:

rman_ target sys/change_on_install@db_onprem auxiliary sys/change_on_install@db_cloud
RMAN> duplicate target database for standby from active database ;

Setting up Data Guard

  • Configure broker config files
  • Add and clear the standby logs
  • Start the broker
  • Create the configuration:
    create configuration db as primary database is db_onprem connect identifier is 'db_onprem';
    add database db_cloud as connect identifier is 'db_cloud';
    edit database db_onprem set property StaticConnectIdentifier='db_onprem';
    edit database db_cloud set property StaticConnectIdentifier='db_cloud';
    enable configuration;
    show configuration;

    The static connect identifier here is better if it uses the TNSNAMES resolution because each database sees each other differently.

Checking the DG config

A validate first:

DGMGRL> show configuration;

Configuration - db

  Protection Mode: MaxPerformance
  Members:
  db_onprem - Primary database
    db_cloud  - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 56 seconds ago)

DGMGRL> validate database db_cloud;

  Database Role:     Physical standby database
  Primary Database:  db_onprem

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Flashback Database Status:
    db_onprem:  Off
    db_cloud :  Off

  Managed by Clusterware:
    db_onprem:  NO
    db_cloud :  NO
    Validating static connect identifier for the primary database db_onprem...
    The static connect identifier allows for a connection to database "db_onprem".

Than a switchover, back and forth:

DGMGRL> switchover to db_cloud;
Performing switchover NOW, please wait...
Operation requires a connection to database "db_cloud"
Connecting ...
Connected to "db_cloud"
Connected as SYSDBA.
New primary database "db_cloud" is opening...
Operation requires start up of instance "db" on database "db_onprem"
Starting instance "db"...
Connected to an idle instance.
ORACLE instance started.
Connected to "db_onprem"
Database mounted.
Connected to "db_onprem"
Switchover succeeded, new primary is "db_cloud"
DGMGRL> show configuration;

Configuration - db

  Protection Mode: MaxPerformance
  Members:
  db_cloud  - Primary database
    db_onprem - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 57 seconds ago)

DGMGRL> switchover to db_onprem;
Performing switchover NOW, please wait...
Operation requires a connection to database "db_onprem"
Connecting ...
Connected to "db_onprem"
Connected as SYSDBA.
New primary database "db_onprem" is opening...
Operation requires start up of instance "db" on database "db_cloud"
Starting instance "db"...
Connected to an idle instance.
ORACLE instance started.
Connected to "db_cloud"
Database mounted.
Connected to "db_cloud"
Switchover succeeded, new primary is "db_onprem"

Conclusion

Yes, it is possible to setup a Data Guard between two sites that have no connections except mono-directional SSH. The SSH tunnels allow SQL*Net communication to a remote endpoint. CMAN allows to proxy through a single endpoint to multiple SCAN addresses.

However, do not forget about the ultimate goal that is to migrate your BUSINESS to the cloud, not just the database. Therefore, having a proper communication to the cloud with proper performance, architecture and security is crucial. Depending on your target Cloud database, Zero Downtime Migration or MV2ADB should be the correct and supported solutions.

ORA-02002 and ORA-00942 while upgrading OWM to 19c

$
0
0

This is a quick post about a problem that we have had while upgrading a DB to 19c.

At 91% of the upgrade, the OWM (Workspace Manager) upgrade was failing with this error error:

*** WARNING: ERRORS FOUND DURING UPGRADE ***

 1. Evaluate the errors found in the upgrade logs
    and determine the proper action.
 2. Rerun the upgrade when the problem is resolved

REASON:
      ERRORS FOUND: During Upgrade
         FILENAME: /u01/app/oracle/home/autoupgrade_cerndb/logs_MYDB/MYDB1/109/dbupgrade/catupgrd20200205101408MYDB_misc_rac520.log AT LINE NUMBER: 46964
------------------------------------------------------
Identifier OWM 20-02-05 10:18:43
SCRIPT    = [/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/admin/owmuany.plb]
ERROR     = [ORA-02002: error while writing to audit trail]
STATEMENT = [select * from wmsys.wm$migration_error_view]
------------------------------------------------------
------------------------------------------------------
Identifier OWM 20-02-05 10:18:43
SCRIPT    = [/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/admin/owmuany.plb]
ERROR     = [ORA-00942: table or view does not exist]
STATEMENT = [as above]
------------------------------------------------------
------------------------------------------------------
Identifier OWM 20-02-05 10:18:43
SCRIPT    = [/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/admin/owmuany.plb]
ERROR     = [ORA-00942: table or view does not exist]
STATEMENT = [as above]
------------------------------------------------------
------------------------------------------------------
Identifier OWM 20-02-05 10:18:43
SCRIPT    = [/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/admin/owmuany.plb]
ERROR     = [ORA-06512: at "WMSYS.LTUTIL", line 4762]
STATEMENT = [as above]
------------------------------------------------------

Indeed, executing the statement was leading consistently to this problem:

sys@MYDB:SQL> select *  from wmsys.wm$migration_error_view;
select *  from wmsys.wm$migration_error_view
                                     *
ERROR at line 1:
ORA-02002: error while writing to audit trail
ORA-00942: table or view does not exist
ORA-00942: table or view does not exist
ORA-06512: at "WMSYS.LTUTIL", line 4762

and we have had this result:

SQL> SELECT distinct(substr(to_char(message),1,60)) OWM_Upgrade_Errors
  2 FROM sys.registry$error
  3  WHERE substr(to_char(message),1,9) != 'ORA-00001'
  4     AND trim(identifier) = 'OWM';

OWM_UPGRADE_ERRORS
-------------------------------------------------------------------
ORA-02002: error while writing to audit trail
ORA-00942: table or view does not exist
ORA-06512: at "WMSYS.LTUTIL", line 4762

So, resuming the autoupgrade job was not a solution.

The view definition is:

create or replace view wmsys.wm$migration_error_view as
select vfield3 error_text
from table(wmsys.ltUtil.wm$getErrors)
/

but the package wmsys.ltUtil is wrapped, so no chance to understand what was happening.

As a quick fix, we have recompiled the binaries with mixed auditing:

# [ oracle@myhost:/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib [11:10:35] [19.6.0.0.0 [DBMS EE] SID=MYDB1] 0 ] #
$ make -f ins_rdbms.mk uniaud_off  ioracle
/usr/bin/ar d /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/libknlopt.a kzaiang.o
/usr/bin/ar cr /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/libknlopt.a /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/kzanang.o
chmod 755 /u01/app/oracle/product/rdbms1960_jvm1960/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/oracle
/u01/app/oracle/product/rdbms1960_jvm1960/bin/orald  -o /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/ -L/u01/app/oracle/product/rdbms1960_jvm1960/lib/ -L/u01/app/oracle/product/rdbms1960_jvm1960/lib/stubs/   -Wl,-E /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/opimai.o /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/ssoraed.o /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv19 -Wl,--no-whole-archive /u01/app/oracle/product/rdbms1960_jvm1960/lib/nautab.o /u01/app/oracle/product/rdbms1960_jvm1960/lib/naeet.o /u01/app/oracle/product/rdbms1960_jvm1960/lib/naect.o /u01/app/oracle/product/rdbms1960_jvm1960/lib/naedhs.o /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/config.o  -ldmext -lserver19 -lodm19 -lofs -lcell19 -lnnet19 -lskgxp19 -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 -lknlopt -loraolap19 -lskjcx19 -lslax19 -lpls19  -lrt -lplp19 -ldmext -lserver19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 `if [ -f /u01/app/oracle/product/rdbms1960_jvm1960/lib/libavserver19.a ] ; then echo "-lavserver19" ; else echo "-lavstub19"; fi` `if [ -f /u01/app/oracle/product/rdbms1960_jvm1960/lib/libavclient19.a ] ; then echo "-lavclient19" ; fi` -lknlopt -lslax19 -lpls19  -lrt -lplp19 -ljavavm19 -lserver19  -lwwg  `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19 -lmm -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lztkg19 `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19   -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 `if /usr/bin/ar tv /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo19 -lserver19"; fi` -L/u01/app/oracle/product/rdbms1960_jvm1960/ctx/lib/ -lctxc19 -lctx19 -lzx19 -lgx19 -lctx19 -lzx19 -lgx19 -lclscest19 -loevm -lclsra19 -ldbcfg19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19 -locr19 -locrb19 -locrutl19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19  -lgeneric19 -lorazip -loraz -llzopro5 -lorabz2 -lorazstd -loralz4 -lipp_z -lipp_bz2 -lippdc -lipps -lippcore  -lippcp -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lsnls19 -lunls19  -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lasmclnt19 -lcommon19 -lcore19  -ledtn19 -laio -lons  -lmql1 -lipc1 -lfthread19    `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/rdbms1960_jvm1960/lib -lm    `cat /u01/app/oracle/product/rdbms1960_jvm1960/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/rdbms1960_jvm1960/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /u01/app/oracle/product/rdbms1960_jvm1960/bin/oracle
mv /u01/app/oracle/product/rdbms1960_jvm1960/rdbms/lib/oracle /u01/app/oracle/product/rdbms1960_jvm1960/bin/oracle
chmod 6751 /u01/app/oracle/product/rdbms1960_jvm1960/bin/oracle
(if [ ! -f /u01/app/oracle/product/rdbms1960_jvm1960/bin/crsd.bin ]; then \
    getcrshome="/u01/app/oracle/product/rdbms1960_jvm1960/srvm/admin/getcrshome" ; \
    if [ -f "$getcrshome" ]; then \
        crshome="`$getcrshome`"; \
        if [ -n "$crshome" ]; then \
            if [ $crshome != /u01/app/oracle/product/rdbms1960_jvm1960 ]; then \
                oracle="/u01/app/oracle/product/rdbms1960_jvm1960/bin/oracle"; \
                $crshome/bin/setasmgidwrap oracle_binary_path=$oracle; \
            fi \
        fi \
    fi \
fi\
);

and put the audit_trail=DB in the upgrade pfile (was NONE in this specific case).

After that, restarted the DB in upgrade mode using the same pfile.

After that, the view was giving no errors anymore and we resumed the autoupgrade job.

SQL> select *  from wmsys.wm$migration_error_view;

no rows selected

This is an old troubleshooting method that I call “Database Administration by guess”: I am not sure about the real cause, but the workaround just worked fine for us.

It would be interesting to know if anyone of you have had the same problem, and what were the auditing parameters in your case…

Ludovico

Understand your Database through graphs

$
0
0

During the last months I have had to deal with highly consolidated databases, where there was no basic rule to achieve something maintainable. Over many years (some schemas are 30 years old) the number of schemas and their dependencies became a problem up to a point where basic operations like patching or upgrading were very complex to achieve.

In order to try to split these big databases and move towards Oracle Multitenant, I have created some graphs that helped me and my colleagues to understand the connections between schemas (grants) and databases (db links).

 

How  have I created the graphs?

I used Gephi , an open source software to generate graphs. Gephi is very powerful, I feel I have used just 1% of its capabilities.

How to create a graph, depends mostly on what you want to achieve and which data you have.

First, some basic terminology: Nodes are the “dots” in the graph, Edges are the lines connecting the dots. Both nodes and edges can have properties (e.g. edges have weight), but you might not need any.

Basic nodes and edges without properties

If you need just to show the dependencies between nodes, a basic edge list with source->target will be enough.

For example, you can have a edge list like this one: gephi_1_edges.csv

Open Gephi, go to New ProjectFile -> Import Spreadsheet, select the file. If you already have a workspace and you want to add the edges to the same workspace, select Append to existing workspace.

This will lead to something very basic:

In the Data Laboratory tab, you might want to copy the value of the ID column to the label column, so that they match:

Now you must care about two things:

First, re-arranging the nodes. With few nodes this is often not required, but when there are many, the default visualization is not satisfying. In the tab Overview , pane Layout there are a few algorithms you can choose. For big graphs I prefer Force Atlas. There are a few parameters to tune, especially  the attraction/repulsion strengths and the gravity. Speed is also crucial if you have many nodes. For this small example I put Repulsion Strength to 2000, Attraction Strength to 1. Clicking on Run starts the algorithm to rearrange the nodes, which with few edges will almost instantaneous (don’t forget to stop it afterwards).

Here is what I get:

Now that the nodes are in place, in the preview pane I can adjust the settings, like showing labels and changing colors. Also, in the Appearance pane I can change the scheme to have for example colors based on ranking.

In this example, I choose to color based on ranking (nodes with more edges are darker).

I also set the Preset Default Straight, Show labels (with smaller size) , proportional size.

Adding nodes properties

Importing edges from CSV gives only a dumb list of edges, without any information about nodes properties. Having properties set might be important in order to change how the graph is displayed.

By importing a node list containing the properties of each node , I can add important information. In this example file, I have columns Id, Label and Sex, that I will use to color the nodes differently: gephi_1_nodes.csv

In the appearance node, I have just selected to partition by sex with a meaningful color.

Using real metadata to understand schemas or dependencies…

I will take, as an example, the dependencies in a database between objects of type VIEW, MATERIALIZED VIEW and TABLE. The database has quite a usage of materialized views and understanding the relation is not always easy.

This is the query that interests me:

SELECT
    owner,
    name,
    type,
    referenced_owner,
    referenced_name,
    referenced_type
FROM
    dba_dependencies
WHERE
    type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND referenced_type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND owner IN (
        SELECT username FROM dba_users
        WHERE oracle_maintained = 'N'
    );

So I need the nodes, for that I need a UNION to get nodes from both sides of the dependency. The best tool to achieve this is SqlCl as it has the native CSV output format:

set sqlformat CSV
set feedback off
spool nodes.csv
SELECT
    owner || '.' || name || ' (' || type || ')' AS "Id",
    owner || '.' || name || ' (' || type || ')' AS "Label",
    owner,
    name,
    type
FROM
    dba_dependencies
WHERE
    type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND referenced_type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND owner IN (
        SELECT  username FROM  dba_users  WHERE oracle_maintained = 'N'
    )
UNION
SELECT
    referenced_owner || '.' || referenced_name || ' (' || referenced_type || ')' AS "Id",
    referenced_owner || '.' || referenced_name || ' (' || referenced_type || ')' AS "Label",
    referenced_owner,
    referenced_name,
    referenced_type
FROM
    dba_dependencies
WHERE
    type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND referenced_type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND owner IN (
        SELECT  username FROM  dba_users  WHERE oracle_maintained = 'N'
    )
;
spool off

The edge list:

spool edges.csv
SELECT
    owner || '.' || name || ' (' || type || ')' AS "Source",
    referenced_owner || '.' || referenced_name || ' (' || referenced_type || ')' AS "Target"
FROM
    dba_dependencies
WHERE
    type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND referenced_type IN ('TABLE', 'MATERIALIZED VIEW', 'VIEW')
    AND owner IN (
        SELECT username FROM dba_users WHERE oracle_maintained = 'N'
    );
spool off

Using the very same procedure as above, it is easy to generate the graph.

I am interested in understanding what is TABLE, what is VIEW and what MATERIALIZED VIEW, so I partition the color by type. I also set the edge color to source so the edge will have the same color of the source node type.

I am also interested in highlighting which tables have more incoming dependencies, so I rank the node size by In-Degree.

 

In the graph:

  • All the red dots are MVIEWS
  • All the blue dots are VIEWS
  • All the black dots are TABLES
  • All the red lines are dependencies between a MVIEW and a  (TABLE|VIEW).
  • All the blue lines are dependencies between a VIEW and a  (TABLE|MVIEW).
  • The bigger the dots, the more incoming dependencies.

With the same approach I can get packages, grants, roles, db_links, client-server dependencies, etc. to better understand the infrastructure.

I hope you like it!

20 years… down the memory lane (and a complimentary news at the end!)

$
0
0

February 22nd, 2000.

I was 20 when I started working as Junior Database Administrator for the service provider SemaGroup, who was born as a branch of the dying Olivetti.

20 years ago, tomorrow! In 20 years, I went through quite a few professional changes. For the records, I would like to write down the main ones.

2000-2003: the years of Sema @ Infostrada

My main customer, for the few next years, was Infostrada, a Telco company who over the years has been merged with Wind Telecomunicazioni and finally acquired by Hutchinson. At that time, I was in a team responsible for the operations, monitoring and availability of all the DB systems of the customer (mostly Digital and HP-UX): Oracle 7.3, 8, 8i, 9i and a few Sybase ASE. We had to respond to all incidents 24/7, including Alert errors, tablespace metrics, etc. It was very demanding, it took me not that much to become confident with the main Oracle technologies. I had the chance to know many colleagues who helped me growing up both technically and personally.

2003-2010: many customers, big projects and new technologies

In 2003, my managers decided to move me back to the main Data center, where Sema Group was providing hosting and managed operations for several big Italian companies.

Different customers mean different technologies. I had to study Microsoft SQL Server (6.5, 7.0 and 2000 at that time), some DB2, and install the first production MySQL (4.0, although I was already playing with it since 3.23). I have also had to become very confident with middle-tier technologies: Oracle Application Server (9 and 10g), Tomcat, Jboss, Apache, but more than this, I have had full responsibility on big projects: Disaster Recovery implementations of big customer environments, data center moving, re-platform, and absolutely heterogeneous customer environments (Windows, Linux, AIX, HP, Compaq, Solaris… SAN, NAS, DAS, Fiber Channels, FCoE, Gigabit Ethernet… Networker, Tivoli, Data Protector… do a cartesian join of all of them and you get the idea.) Among the big customers, I have worked with Costa Crociere, Lavazza, Illy, Fiat, Banca d’Italia, Fila, AirLiquide, Credit Agricole, Carrefour, International Olympic Committee, Heineken, Enel, Banco Popolare S.C., Piaggio, Pagine Gialle, European Space Agency, several public administrations (Regions, Ministries, Cities…)… I am sure I miss many, as they were more than 100.

During these years, I had to deal with incredibly hard on-call rotations (several calls per night) and high customer expectations, who helped giving me confidence and “seniority”.

Over the years, SemaGroup has been acquired/merged a few times. It became Sema, SchlumbergerSema, Schlumberger, AtosOrigin, Engineering.IT, then Engineering.

2010-2012: the team lead experience @ Engineering

In 2010 I became team lead, at the beginning just of a team of 5, then gradually of the whole Engineering Managed Operations DBA team (23 including employees and external collaborators). Sadly, in Italy it is very uncommon to have a technical career: as soon as you are good enough, you have to switch to the management side (either completely or partially). Being team lead, I understood the difference between technical problems and people problems. With the increase of customers and projects, and with the many company acquisitions, we were understaffed, most of us underpaid: working days, nights and week-ends without the chance to recover properly. It took me just a few months to realize that that was the time to say goodbye to the IT market in Italy and seek a better opportunity, abroad.

2012-2018: Senior Consultant @ Trivadis

I still remember perfectly the first call I have got from Trivadis, the very next day after I applied for a position for Oracle Consultant. Four months later I moved to Switzerland: different country, different language and completely different working culture despite the very short distance from my home region (less than 200 KM!).

There I have had many important customers and cool projects: everything was quality-driven rather than quantity-driven. Being part of that company has been a pure joy for me: company culture, colleagues, side-activities, conferences… results!

I have learnt to be outgoing and communicative, and I have kept enriching my technical knowledge (mostly on Oracle and PostgreSQL).

2018-2020: The CERN experience

Once I have got the contract proposal from CERN, it was hard to say no. Awesome conditions (pension, holidays, healthcare, dress code, flexibility…) and a once-in-a-lifetime opportunity to join the geekiest place in the world.

Here I have been working exclusively with Oracle RAC on Linux. There are incredibly large databases (from a few TBs to 1 PB) and interesting technologies, but the vast majority of my time is dealing with legacy database schemas and methodologies. I am trying hard to improve things, but the inertia of such big projects does not allow me to innovate as fast as I would like. So what’s next?

2020- : Back to Trivadis

Last December Trivadis contacted me back and tried to convince me to go back… with success 🙂

Next April 1st I will start back in Trivadis as Principal Consultant. Initially, I will be working on a couple of critical projects: it will be very interesting! Other than that, I am excited to go back to the company (and colleagues) that I have loved so much!

To end this quite long post… I would like to paste here one of the quotes that well represents my feelings since I have moved in Switzerland…

“Why do you go away? So that you can come back. So that you can see the place you came from with new eyes and extra colors. And the people there see you differently too. Coming back to where you started is not the same as never leaving.”

– Terry Pratchett

Parallel Oracle Catalog/Catproc creation with catpcat.sql

$
0
0

With Oracle 19c, Oracle has released a new script, annotated for parallel execution, to create the CATALOG and CATPROC in parallel at instance creation.

I have a customer who is in the process of migrating massively to Multitenant using many CDBs, so I decided to give it a try to check how much time they could save for the CDB creations.

I have run the tests on my laptop, on a VirtualBox VM with 4 vCPUs.

Test 1: catalog.sql + catproc.sql

In this test, I use the classic way (this is also the case when DBCA creates the scripts):

${ORACLE_HOME}/perl/bin/perl ${ORACLE_HOME}/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/CDB1/create -v \
  -b catalog  -U "SYS"/"manager" ${ORACLE_HOME}/rdbms/admin/catalog.sql
${ORACLE_HOME}/perl/bin/perl ${ORACLE_HOME}/rdbms/admin/catcon.pl -n 1 -l /u01/app/oracle/admin/CDB1/create -v \
  -b catproc  -U "SYS"/"manager" ${ORACLE_HOME}/rdbms/admin/catproc.sql

The catalog is created first on CDB$ROOT and PDB$SEED. Then the catproc is created.

Looking at the very first occurrence of BEGIN_RUNNING (start of catalog for CDB$ROOT) and the very last of END_RUNNING in the log (end of catproc in PDB$SEED), I can see that it took ~ 44 minutes to complete:

BEGIN_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/product/db_19_6_0/rdbms/admin/catalog.sql
Container:CDB$ROOT Id:1 20-05-04 11:30:54 Proc:0 ====

END_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/product/db_19_6_0/rdbms/admin/catproc.sql
Container:PDB$SEED Id:2 20-05-04 12:15:00 Proc:0 ====

 

Test 2: catpcat.sql

In this test, I use the new catpcat.sql using catctl.pl, with a parallelism of 4 processes:

${ORACLE_HOME}/perl/bin/perl ${ORACLE_HOME}/rdbms/admin/catctl.pl -n 4 -c 'CDB$ROOT PDB$SEED' \
  -l /u01/app/oracle/admin/CDB2/create -d ${ORACLE_HOME}/rdbms/admin catpcat.sql

This creates catalog and catproc first on CDB$ROOT, than creates them on PDB$SEED. So, same steps but in different orders.

By running vmstat in the background, I noticed during the run that most of the time the creation was running serially, and when there was some parallelism, it was short and compensated by a lot of process synchronizations (waits, sleeps) done by the catctl.pl.

At the end, the process took ~ 45 minutes to complete.

BEGIN_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/product/db_19_6_0/rdbms/admin/catpcatstr.sql
Container:CDB$ROOT Id:1 20-05-04 01:53:22 Proc:0 ====
...
END_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/admin/CDB2/create/catpcatpdb_seed20.0362629753546919.sql
Container:PDB$SEED Id:2 20-05-04 02:38:37 Proc:0 ====

So the answer is no: it is not faster to run catpcat.sql with parallel degree 4 compared to running catalog.sql and catproc.sql serially.

HTH

Ludo


Multitenant Pills – Change DBID for an existing PDB

$
0
0

When you plug the same PDB many times you have to specify “AS COPY” in the syntax:

CREATE PLUGGABLE DATABASE ludo AS CLONE USING '/u01/app/oradata/ludo/ludo.xml';

Otherwise, you will get an error similar to:

ERROR at line 1:
ORA-65122: Pluggable database GUID conflicts with the GUID of an existing container.

There are case, however, where you cannot do it. For example, it the existing PDB should have been the clone, or if you are converting a copy of the same database from Non-CDB to PDB using autoupgrade (with autoupgrade you cannot modify the CREATE PLUGGABLE DATABASE statement).

In this case the solution might be to change the DBID of the existing PDB, via unplug/plug:

ALTER PLUGGABLE DATABASE vico CLOSE;
ALTER PLUGGABLE DATABASE vico UNPLUG INTO '/u01/app/oradata/ludo/ludo.xml';
DROP PLUGGABLE DATABASE vico KEEP DATAFILES;
CREATE PLUGGABLE DATABASE vico AS CLONE  USING '/u01/app/oradata/ludo/ludo.xml' NOCOPY;
ALTER  PLUGGABLE DATABASE vico OPEN;
ALTER  PLUGGABLE DATABASE vico SAVE STATE;

Ludo

Multitenant Pills: Pushing your PDB to the Cloud in one step?

$
0
0

The Oracle Multitenant architecture introduces some nice opportunities, including local and remote cloning (see this example on ORACLE_BASE blog).

However, the only available cloning procedure use the PULL method to get the data from the remote source PDB.

This can be a limitation, especially in environments where the cloning environment does not have direct access to the production, or where the clones must be done in the Cloud with no direct access to the production VLAN on-premises.

So, one common approach is to clone/detach locally, put the PDB files in the Object Store and then attach them in the cloud.

Another approach is to use SSH tunnels. If you follow my blog you can see it is something that I use every now and then to do stuff from on-premises to the cloud.

How to set it up?

Actually, it is super-easy: just prepare a script in the cloud that will do the create pluggable database, then trigger it from on-premises.

This is an example:

SRC_PDB=$1
TGT_PDB=$2
ALIAS=$3

export ORACLE_HOME=/u01/app/oracle/product/19c/dbhome_1
export ORACLE_SID=ORCL

$ORACLE_HOME/bin/sqlplus -s / as sysdba <<EOF
        create database link onprem_temp
          connect to C##ONPREM identified by manager using '$ALIAS';
        create pluggable database $2 from $1@onprem_temp ;
        drop database link onprem_temp;
        alter pluggable database $2 open;
        alter pluggable database $2 save state;
        exit
EOF

It takes as parameters: the name of the source PDB, the name of the target PDB and the SQL*Net descriptor to create the temporary database link from the cloud CDB to the on-premises CDB.

The user C##ONPREM must obviously exist on-premises with the following privileges:

grant create session, sysoper, create pluggable database to C##ONPREM container=all;

The cloud database should use OMF so you don’t have to take care about file name conversions.

At this point, if you have set up correctly the SSH keys to connect to the cloud server, it is just a matter of running the script remotely using the proper SSH tunnel. Once the remote port binding established, the cloud server can contact the on-premises listener port using localhost:remote_bind:

oracle@onpremsrv:~$ ssh opc@cloudsrv -R 1522:onpremsrv:1521 \
 "sudo -u oracle /home/oracle/clone_db.sh PDBSRC PDBDEST localhost:1522/ONPREMCDB"

Database link created.

Pluggable database created.


Database link dropped.

oracle@onpremsrv:~$

Of course the timing depends a lot on the size of the database and your connection to the Cloud.

I have tested this procedure with Oracle Database 19.7 on OCI compute instances and on DBaaS VM instance, it works without any additional work. Of course, it does not work for Autonomous Database 🙂

Ludovico

Multitenant Pills: Partial PDB cloning (and cleanup)

$
0
0

When consolidating to multitenant, there are several consolidation patterns.

  • Big, complex databases usually have special requirements for which it might be a good choice to go to single-tenant (a single PDB in one CDB)
  • Small, relatively easy databases are the best candidate for consolidation to multitenant
  • Schema consolidated databases require special attention, but in general there are several advantages to convert individual schemas (or group of schemas) to individual PDBs

For the latter, there are some techniques to convert a schema in a PDB.

  • export/import (obviously), with eventually Golden Gate to do it online
  • Transportable tablespaces (if the schemas follow strict 1-to-1 tablespace separation
  • partial PDB cloning

We will focus on the last one for this blog post.

Situation

Here we have a PDB with some schemas, each of them has a dedicated tablespace, but accidentally, two of them have also some objects on a common tablespace.

This happens frequently when all the users have quota on the default database tablespace and they do not have necessarily a personal default tablespace.

connect / as sysdba
CREATE PLUGGABLE DATABASE PDB1 ADMIN USER pdbadmin IDENTIFIED BY manager;
ALTER PLUGGABLE DATABASE PDB1OPEN;
ALTER SESSION SET CONTAINER=PDB1;

CREATE TABLESPACE ABC DATAFILE SIZE 50M;
CREATE TABLESPACE DEF DATAFILE SIZE 50M;
CREATE TABLESPACE GHI DATAFILE SIZE 50M;

CREATE USER ABC IDENTIFIED BY ABC DEFAULT TABLESPACE ABC QUOTA UNLIMITED ON ABC;
CREATE USER DEF IDENTIFIED BY DEF DEFAULT TABLESPACE DEF QUOTA UNLIMITED ON DEF;
CREATE USER GHI IDENTIFIED BY GHI DEFAULT TABLESPACE GHI QUOTA UNLIMITED ON GHI;

ALTER USER ABC QUOTA UNLIMITED ON DATA;
ALTER USER DEF QUOTA UNLIMITED ON DATA;
ALTER USER GHI QUOTA UNLIMITED ON DATA;

GRANT CREATE SESSION, ALTER SESSION, CREATE TABLE TO ABC, DEF, GHI;


CONNECT abc/abc@srcsrv:1521/pdb1

CREATE TABLE T1 (A VARCHAR2(50));
INSERT INTO T1 VALUES ('TOTO');
COMMIT;

CREATE TABLE T2 (A VARCHAR2(50)) TABLESPACE DATA;
INSERT INTO T2 VALUES ('TOTO');
COMMIT;


CONNECT def/def@srcsrv:1521/pdb1

CREATE TABLE T1 (A VARCHAR2(50));
INSERT INTO T1 VALUES ('TOTO');
COMMIT;

CREATE TABLE T2 (A VARCHAR2(50)) TABLESPACE DATA;
INSERT INTO T2 VALUES ('TOTO');
COMMIT;


CONNECT ghi/ghi@srcsrv:1521/pdb1

CREATE TABLE T1 (A VARCHAR2(50));
INSERT INTO T1 VALUES ('TOTO');
COMMIT;

This is the typical situation where transportable tablespaces become hard to achieve without some upfront segment movement, as tablespaces are not self-contained.

Thankfully, Oracle Multitenant allows us to clone a PDB from a remote one and specify only a subset of tablespaces.

CREATE PLUGGABLE DATABASE ABC FROM PDB1.ORCL1_PDB1_tempclone 
  USER_TABLESPACES=('ABC','DATA');

Here is a full example script with some checks and fancy parameters:

--------------------------
-- Script: clone_pdb.sql
-- Purpose: (partial) clone a PDB from remote PDB
-- Author: Ludovico Caldara (Trivadis)
--
-- Execute as: DBA in CDB$ROOT
--
-- Requirements: The remote user for the DBlink must have SELECT privileges on a few fixed views:
-- --
-- create user c##remote_user identified by manager container=all;
-- grant create session, sysoper, create pluggable database to c##remote_user container=all;
-- -- granting extra privileges to get file names and total size
-- grant select on sys.v_$pdbs to c##remote_user container=all;
-- grant select on sys.v_$datafile to c##remote_user container=all;
-- grant select on sys.v_$database to c##remote_user container=all;
-- alter user c##remote_user set container_data=all for v_$pdbs container=current;
-- alter user c##remote_user set container_data=all for v_$datafile container=current;
-- alter user c##remote_user set container_data=all for v_$database container=current;
--
--
--
-- Script variables:
define SourceCDB = 'ORCL1'
define SourcePDB = 'PDB1'
define SourceUser = 'C##REMOTE_USER'
define SourcePassword = 'manager'
define SourceAlias = 'srcsrv:1521/PDB1'
define TargetCDB = 'ORCL2'
define TargetPDB = 'ABC'
define UserTBSPClause = 'USER_TABLESPACES=(''ABC'',''DATA'')'
-- in case of non-OMF
define FileNameConvertClause = 'FILE_NAME_CONVERT=(''/&SourceCDB/&SourcePDB/'',''/&TargetCDB/&TargetPDB/'')'

SET ECHO ON SERVEROUTPUT ON

-- Temporary database link name. We try to drop it if it exists but nevermind if it gives error
WHENEVER SQLERROR CONTINUE
DROP DATABASE LINK &SourceCDB._&SourcePDB._tempclone;

-- The other tasks shouls succeed, otherwise exit
WHENEVER SQLERROR EXIT 1
CREATE DATABASE LINK &SourceCDB._&SourcePDB._tempclone connect to &SourceUser identified by &SourcePassword using '&SourceAlias';

-- in this PL/SQL block we make sure that we do not copy if
-- the source is NOARCHIVELOG or not LOCAL UNDO and PDB is not READ ONLY
DECLARE
        l_log_mode sys.v_$database.log_mode%TYPE;
        l_local_undo database_properties.property_value%TYPE;
        l_open_mode sys.v_$pdbs.open_mode%TYPE;

BEGIN
        BEGIN
                SELECT open_mode INTO l_open_mode FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE name='&SourcePDB';
        EXCEPTION
                WHEN no_data_found THEN raise_application_error(-20104,'Pluggable database not found.');
        END;

        -- Check if NOARCHIVELOG and not READ ONLY
        SELECT log_mode INTO l_log_mode FROM sys.v_$database@&SourceCDB._&SourcePDB._tempclone ;
        IF l_log_mode = 'NOARCHIVELOG' AND l_open_mode <> 'READ ONLY' THEN
                raise_application_error(-20102,'The database is in NOARCHIVELOG mode and the PDB is not READ ONLY.');
        END IF;

        -- Check if not LOCAL_UNDO_ENABLED  and not READ ONLY
        SELECT property_value INTO l_local_undo FROM database_properties@&SourceCDB._&SourcePDB._tempclone WHERE  property_name = 'LOCAL_UNDO_ENABLED';
        IF l_local_undo <> 'TRUE' AND l_open_mode <> 'READ ONLY' THEN
                raise_application_error(-20103,'The LOCAL UNDO is not enabled for the database and the PDB is not READ ONLY.');
        END IF;
END;
/

-- let's add some intelligence... Parallel degree depending on PDB size
COLUMN parallel new_value parallel
SELECT CASE
        WHEN ROUND(total_size/POWER(1024,3)) <= 1 THEN 1
        WHEN ROUND(total_size/POWER(1024,3)) >= 5 THEN 4
        ELSE 2 END AS parallel
FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE NAME='&SourcePDB';

-- Real clone PDB
CREATE PLUGGABLE DATABASE &TargetPDB FROM &SourcePDB.@&SourceCDB._&SourcePDB._tempclone &UserTBSPClause PARALLEL &parallel &FileNameConvertClause;

ALTER PLUGGABLE DATABASE &TargetPDB OPEN;
ALTER PLUGGABLE DATABASE &TargetPDB SAVE STATE;
-- drop temporary db link: leaving it there exposes some risks
DROP DATABASE LINK &SourcePDB.@&SourceCDB._&SourcePDB._tempclone

SET ECHO OFF SERVEROUTPUT OFF

This is an example output:

SQL> @/tmp/clone
SQL>
SQL> -- Temporary database link name. We try to drop it if it exists but nevermind if it gives error
SQL> WHENEVER SQLERROR CONTINUE
SQL> DROP DATABASE LINK &SourceCDB._&SourcePDB._tempclone;
old   1: DROP DATABASE LINK &SourceCDB._&SourcePDB._tempclone
new   1: DROP DATABASE LINK ORCL1_PDB1_tempclone

Database link dropped.

SQL>
SQL> -- The other tasks shouls succeed, otherwise exit
SQL> WHENEVER SQLERROR EXIT 1
SQL> CREATE DATABASE LINK &SourceCDB._&SourcePDB._tempclone connect to &SourceUser identified by &SourcePassword using '&SourceAlias';
old   1: CREATE DATABASE LINK &SourceCDB._&SourcePDB._tempclone connect to &SourceUser identified by &SourcePassword using '&SourceAlias'
new   1: CREATE DATABASE LINK ORCL1_PDB1_tempclone connect to C##REMOTE_USER identified by manager using 'srcsrv:1521/PDB1'

Database link created.

SQL>
SQL> -- in this PL/SQL block we make sure that we do not copy if
SQL> -- the source is NOARCHIVELOG or not LOCAL UNDO and PDB is not READ ONLY
SQL> DECLARE
  2          l_log_mode sys.v_$database.log_mode%TYPE;
  3          l_local_undo database_properties.property_value%TYPE;
  4          l_open_mode sys.v_$pdbs.open_mode%TYPE;
  5
  6  BEGIN
  7          BEGIN
  8                  SELECT open_mode INTO l_open_mode FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE name='&SourcePDB';
  9          EXCEPTION
 10                  WHEN no_data_found THEN raise_application_error(-20104,'Pluggable database not found.');
 11          END;
 12
 13          -- Check if NOARCHIVELOG and not READ ONLY
 14          SELECT log_mode INTO l_log_mode FROM sys.v_$database@&SourceCDB._&SourcePDB._tempclone ;
 15          IF l_log_mode = 'NOARCHIVELOG' AND l_open_mode <> 'READ ONLY' THEN
 16                  raise_application_error(-20102,'The database is in NOARCHIVELOG mode and the PDB is not READ ONLY.');
 17          END IF;
 18
 19          -- Check if not LOCAL_UNDO_ENABLED  and not READ ONLY
 20          SELECT property_value INTO l_local_undo FROM database_properties@&SourceCDB._&SourcePDB._tempclone WHERE  property_name = 'LOCAL_UNDO_ENABLED';
 21          IF l_local_undo <> 'TRUE' AND l_open_mode <> 'READ ONLY' THEN
 22                  raise_application_error(-20103,'The LOCAL UNDO is not enabled for the database and the PDB is not READ ONLY.');
 23          END IF;
 24  END;
 25  /
old   8:                SELECT open_mode INTO l_open_mode FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE name='&SourcePDB';
new   8:                SELECT open_mode INTO l_open_mode FROM sys.v_$pdbs@ORCL1_PDB1_tempclone WHERE name='PDB1';
old  14:        SELECT log_mode INTO l_log_mode FROM sys.v_$database@&SourceCDB._&SourcePDB._tempclone ;
new  14:        SELECT log_mode INTO l_log_mode FROM sys.v_$database@ORCL1_PDB1_tempclone ;
old  20:        SELECT property_value INTO l_local_undo FROM database_properties@&SourceCDB._&SourcePDB._tempclone WHERE  property_name = 'LOCAL_UNDO_ENABLED';
new  20:        SELECT property_value INTO l_local_undo FROM database_properties@ORCL1_PDB1_tempclone WHERE  property_name = 'LOCAL_UNDO_ENABLED';

PL/SQL procedure successfully completed.

SQL>
SQL> -- let's add some intelligence... Parallel degree depending on PDB size
SQL> COLUMN parallel new_value parallel
SQL> SELECT CASE
  2          WHEN ROUND(total_size/POWER(1024,3)) <= 1 THEN 1
  3          WHEN ROUND(total_size/POWER(1024,3)) >= 5 THEN 4
  4          ELSE 2 END AS parallel
  5  FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE NAME='&SourcePDB';
old   5: FROM sys.v_$pdbs@&SourceCDB._&SourcePDB._tempclone WHERE NAME='&SourcePDB'
new   5: FROM sys.v_$pdbs@ORCL1_PDB1_tempclone WHERE NAME='PDB1'

  PARALLEL
----------
         2

SQL>
SQL> -- Real clone PDB
SQL> CREATE PLUGGABLE DATABASE &TargetPDB FROM &SourcePDB.@&SourceCDB._&SourcePDB._tempclone &UserTBSPClause PARALLEL &parallel &FileNameConvertClause;
old   1: CREATE PLUGGABLE DATABASE &TargetPDB FROM &SourcePDB.@&SourceCDB._&SourcePDB._tempclone &UserTBSPClause PARALLEL &parallel &FileNameConvertClause
new   1: CREATE PLUGGABLE DATABASE ABC FROM PDB1@ORCL1_PDB1_tempclone USER_TABLESPACES=('ABC','DATA') PARALLEL          2 FILE_NAME_CONVERT=('/ORCL1/PDB1/','/ORCL2/ABC/')

Pluggable database created.

SQL> ALTER PLUGGABLE DATABASE &TargetPDB OPEN;
old   1: ALTER PLUGGABLE DATABASE &TargetPDB OPEN;
new   1: ALTER PLUGGABLE DATABASE ABC OPEN;

Pluggable database altered.

SQL>ALTER PLUGGABLE DATABASE &TargetPDB SAVE STATE; 
old   1: ALTER PLUGGABLE DATABASE &TargetPDB SAVE STATE;
new   1: ALTER PLUGGABLE DATABASE ABC SAVE STATE;

Pluggable database altered.


SQL>
SQL> -- drop temporary db link: leaving it there exposes some risks
SQL> DROP DATABASE LINK &SourcePDB.@&SourceCDB._&SourcePDB._tempclone
  2
SQL> SET ECHO OFF SERVEROUTPUT OFF

If the clone process succeeds, at the end we should have the new ABC pluggable database with ABC and DATA tablespaces only.

SQL> select tablespace_name from dba_tablespaces;

TABLESPACE_NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
DATA
ABC

Yeah!

Any Cleanup needed?

What happened to the users? Actually, they are all still there:

SQL> select username from dba_users where oracle_maintained='N';

USERNAME
--------------------------------------------------------------------------------
PDBADMIN
C##REMOTE_CLONE
ABC
DEF
GHI

5 rows selected.

And the segments in the two skipped tablespaces are not there:

SQL> select owner, segment_name, tablespace_name from dba_segments 
2> where owner in ('ABC','DEF','GHI');

OWNER                SEGMENT_NAME         TABLESPACE_NAME
-------------------- -------------------- ------------------------------
ABC                  T2                   DATA
DEF                  T2                   DATA
ABC                  T1                   ABC

So the table definitions are also gone?

SQL> select owner, table_name, tablespace_name from dba_tables
  2    where owner in ('ABC','DEF','GHI');

OWNER                TABLE_NAME           TABLESPACE_NAME
-------------------- -------------------- ------------------------------
ABC                  T2                   DATA
ABC                  T1                   ABC
DEF                  T2                   DATA
DEF                  T1                   DEF
GHI                  T1                   GHI

Not at all! The tables are still there and reference to tablespaces that do not exist anymore. Possible?

Actually, the tablespaces definition are still there if we look at v$tablespace:

SQL> select name from v$tablespace;

NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
DATA
ABC
DEF
GHI

8 rows selected.

If we give a look at the DBA_TABLESPACES view definition, there are a few interesting filters:

create or replace view DBA_TABLESPACES
...
where ts.online$ != 3
and bitand(flags,2048) != 2048
and bitand(ts.flags, 16777216) != 16777216
...
/

What is their meaning?

online$     /* status (see KTT.H): */
             /* 1 = ONLINE, 2 = OFFLINE, 3 = INVALID */

  flags      /* various flags: see ktt3.h */
             /* 0x800 = this actually describes a group */ -- 2048
             /* 0x1000000 = bad transport of ts in pdb */ -- 16777216

So the first WHERE clause skips all the INVALID TABLESPACES (when you drop a tablespace it is still stored in ts$ with this state), the second skips the definition of TEMPORARY TABLESPACE GROUPS, the third one is actually our candidate.

Indeed, this is what we get from ts$ for these tablespaces:

SQL> select name, decode(online$,1,'ONLINE',2,'OFFLINE',3,'INVALID',4,'READ ONLY','UNDEFINED') as STATUS, bitand(flags,2048), bitand(flags,16777216) from ts$;

NAME                           STATUS    BITAND(FLAGS,2048) BITAND(FLAGS,16777216)
------------------------------ --------- ------------------ ----------------------
SYSTEM                         ONLINE                     0                      0
SYSAUX                         ONLINE                     0                      0
UNDOTBS1                       ONLINE                     0                      0
TEMP                           ONLINE                     0                      0
DATA                           ONLINE                     0                      0
ABC                            ONLINE                     0                      0
DEF                            OFFLINE                    0               16777216
GHI                            OFFLINE                    0               16777216

8 rows selected.

So the two tablespaces are filtered out because of this new multitenant flag.

If we try to drop the tablespaces, it succeeds:

SQL> drop tablespace def including contents and datafiles;

Tablespace dropped.

SQL> drop tablespace ghi including contents and datafiles;

Tablespace dropped.

But the user GHI, who has no objects anymore, is still there.

SQL> select username from dba_users
  2  where username  in ('ABC','DEF','GHI') ;

USERNAME
-------------------------
GHI
DEF
ABC

So we need to drop it explicitly.

SQL> drop user ghi cascade;

User dropped.

SQL>

Automate the cleanup

This is an example PL/SQL that is aimed to automate the cleanup.

Actually:

  • Users that had segments in one of the excluded tablespaces but do not have any segments left are just LOCKED (for security reasons, you can guess why).
  • Tablespaces that meet the “excluded PDB” criteria, are just dropped

set serveroutput on
DECLARE

        -- get the users that had tables in the non-cloned TBSP that do not have any other segments...
        CURSOR c_users IS
                SELECT DISTINCT owner as username FROM dba_tables t
                WHERE t.tablespace_name IN (
                        SELECT name FROM sys.ts$ WHERE online$=2 AND bitand(flags, 16777216)=16777216
                ) AND NOT EXISTS (
                        SELECT 1 FROM dba_segments s WHERE s.owner=t.owner
                );

        -- get the list of non-cloned TBSP
        CURSOR c_tbsps IS
                SELECT name FROM sys.ts$ WHERE online$=2 AND bitand(flags, 16777216)=16777216;

        r_users c_users%ROWTYPE;
        r_tbsps c_tbsps%ROWTYPE;

        e_drop_def_tbsp EXCEPTION;
        PRAGMA EXCEPTION_INIT (e_drop_def_tbsp, -12919);


BEGIN
        -- check if currently in a PDB
        IF sys_context('userenv','con_name') = 'CDB$ROOT' THEN
                raise_application_error(-20101,'The current container is CDB$ROOT. It must be a PDB.');
        END IF;

        -- loop all candidate users and lock them out
        OPEN c_users;
        LOOP
                        FETCH  c_users  INTO r_users;
                        EXIT WHEN c_users%NOTFOUND;
                        dbms_output.put_line ('ALTER USER '||r_users.username||' ACCOUNT LOCK');
                        execute immediate 'ALTER USER '||r_users.username||' ACCOUNT LOCK';
                        dbms_output.put_line ('User '||r_users.username||' locked.');
        END LOOP;

        -- loop all candidate tbsps and drop them
        OPEN c_tbsps;
        LOOP
                FETCH  c_tbsps  INTO r_tbsps;
                EXIT WHEN c_tbsps%NOTFOUND;
                dbms_output.put_line ('DROP TABLESPACE '||r_tbsps.name||' INCLUDING CONTENTS AND DATAFILES');
                BEGIN
                        execute immediate 'DROP TABLESPACE '||r_tbsps.name||' INCLUDING CONTENTS AND DATAFILES';
                        dbms_output.put_line ('Tablespace '||r_tbsps.name||' dropped.');
                EXCEPTION
                        WHEN e_drop_def_tbsp THEN
                                dbms_output.put_line ('Cannot drop default tablespace '||r_tbsps.name||'. Please change the default tablespace and drop it afterwards.');
                END;
        END LOOP;
END;
/

This is the output for the clone procedure we have just seen:

SQL> @cleanup_partial_pdb.sql
ALTER USER GHI ACCOUNT LOCK
User GHI locked.
DROP TABLESPACE DEF INCLUDING CONTENTS AND DATAFILES
Tablespace DEF dropped.
DROP TABLESPACE GHI INCLUDING CONTENTS AND DATAFILES
Tablespace GHI dropped.

PL/SQL procedure successfully completed.

The PL/SQL block can be quite slow depending on the segments and tablespaces, so it might be a good idea to have a custom script instead of this automated one.

What about user DEF?

The automated procedure has not locked the account DEF. Why?

Actually, the user DEF still has some segments in the DATA tablespace. Hence, the procedure cannot be sure what was the original intention: copy the user ABC ? The clone procedure allows only to specify the tablespaces, so this is the only possible result.

Promo: If you need to migrate to Multitenant and you need consulting/training, just contact me, I can help you 🙂

 

Awk to format the files .ora (listener.ora, tnsnames.ora, etc)

$
0
0

Tired of formatting the tnsnames.ora to make it more readable, I have taken the nice awk examples from Jeremy Schneider: https://ardentperf.com/2008/11/28/parsing-listenerora-with-awk-and-sed/ and created a function to format all files .ora (lisp-like config files).

Example, before:

$ cat tnsnames.ora
LUDOCDB1_SITE2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP )(HOST = newbox02 )(PORT = 1522 ))
    (CONNECT_DATA =
      (SERVICE_NAME = LUDOCDB1_SITE2_DGMGRL )
    )
  )
LUDOCDB1_SITE1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP )(HOST = newbox01 )(PORT = 1522 )) (CONNECT_DATA = (SERVICE_NAME = LUDOCDB1_SITE1_DGMGRL )
    )
  )
LUDOCDB1 = (DESCRIPTION = (CONNECT_TIMEOUT = 5 )(TRANSPORT_CONNECT_TIMEOUT = 3 ) (ADDRESS_LIST = (LOAD_BALANCE = OFF ) (ADDRESS = (PROTOCOL = TCP )(HOST = newbox01 )(PORT = 1522 )) (ADDRESS = (PROTOCOL = TCP )(HOST = newbox02 )(PORT = 1522 ))) (CONNECT_DATA = (SERVICE_NAME = LUDOCDB1_RW )))

# read only:
LUDOCDB1_RO =
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 5 )(TRANSPORT_CONNECT_TIMEOUT = 3 )
    (ADDRESS_LIST = (LOAD_BALANCE = OFF )
      (ADDRESS = (PROTOCOL = TCP )(HOST = newbox02 )(PORT = 1522 ))
      (ADDRESS = (PROTOCOL = TCP )(HOST = newbox01 )(PORT = 1522 ))
    ) (CONNECT_DATA =
      (SERVICE_NAME = LUDOCDB1_RO )
    )
  )

and after:

# cat tnsnames.ora | tidy_dotora
LUDOCDB1_SITE2=
   (DESCRIPTION=
     (ADDRESS=
       (PROTOCOL=TCP)
       (HOST=newbox02)
       (PORT=1522)
     )
     (CONNECT_DATA=
       (SERVICE_NAME=LUDOCDB1_SITE2_DGMGRL)
     )
   )

LUDOCDB1_SITE1=
   (DESCRIPTION=
     (ADDRESS=
       (PROTOCOL=TCP)
       (HOST=newbox01)
       (PORT=1522)
     )
     (CONNECT_DATA=
       (SERVICE_NAME=LUDOCDB1_SITE1_DGMGRL)
     )
   )

LUDOCDB1=
   (DESCRIPTION=
     (CONNECT_TIMEOUT=5)
     (TRANSPORT_CONNECT_TIMEOUT=3)
     (ADDRESS_LIST=
       (LOAD_BALANCE=OFF)
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=newbox01)
         (PORT=1522)
       )
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=newbox02)
         (PORT=1522)
       )
     )
     (CONNECT_DATA=
       (SERVICE_NAME=LUDOCDB1_RW)
     )
   )
# read only:

LUDOCDB1_RO=
   (DESCRIPTION=
     (CONNECT_TIMEOUT=5)
     (TRANSPORT_CONNECT_TIMEOUT=3)
     (ADDRESS_LIST=
       (LOAD_BALANCE=OFF)
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=newbox02)
         (PORT=1522)
       )
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=newbox01)
         (PORT=1522)
       )
     )
     (CONNECT_DATA=
       (SERVICE_NAME=LUDOCDB1_RO)
     )
   )

The AWK script:

tidy_dotora () {
 
# Heavily based on Jeremy Scheider's work:
# https://ardentperf.com/2008/11/28/parsing-listenerora-with-awk-and-sed/
#
# you can source this function in bash and use it like:
# cat $TNS_ADMIN/listener.ora | tidy_dotora 
 
awk '
 
# function for padding
function pad (string, len, char) {
	ret = string;
	for ( padi = length(string); padi<len ; padi++) {
			ret = sprintf("%s%s",ret,char);
	}
	return ret;
}
	
BEGIN {
	level=1;
	first=1;
	lastcomment=0;
}
	
{
#MAIN
 
	# just skip any comments and print as is
	if ($0 ~ "^[[:space:]]*#") {
		if (lastcomment==0) {
			printf("\n");
		}
		print;
		lastcomment=1;
		next;
	}
	lastcomment=0;
	
	# this puts every occurrence of =, ( and ) in different tokens
	gsub(/=/,"`=");
	gsub(/\(/,"`(");
	gsub(/\)/,"`)");
	split($0,tokens,"`");
 
	i=1; while(i in tokens) {
 
		# trim token and continue if empty
		gsub(/ /, "",tokens[i]);
		if(!tokens[i]) {i++; continue;}
 
		# got ( "open bracket": new level begins
		# increase the level, newline and pad
		if(tokens[i]~"^[(]") {
			level++;
			printf ("\n");
			printf (pad("", 2*level-1, " "));
		}
	
		# got ) "close bracket" : level ends
		# decrease the level but newline only if another one was closed immediately before
		if(tokens[i]~"^[)]") {
			level--;
			if (wentdown==1) {
				printf("\n");
				printf (pad("", 2*level+1, " "));
			}
			wentdown=1;
		} else {
			wentdown=0;
		}
	
		# if level==1 and is alphanumeric, it is a "TOP" entry (LISTENER, SID_LIST_LISTENER or property)
		# add extra line (and eventually track it for other usage)
		if(level==1 && i==1 && tokens[i]~"[A-Za-z]") {
			TOP=tokens[i];
			if (first==1) {
				first=0;
			} else {
				printf "\n\n";
			}
		}
 
		printf (tokens[i]);
		i++;
	}
}
END {
	# new line at the end of file
	printf("\n");
}' 
}

I have included the function in the COE github repo. More functions to come (hopefully).

Ludo

How to get the Data Guard broker configuration from a SQL query?

$
0
0

Everybody knows that you can get the Data Guard configuration in dgmgrl with the command:

show configuration;

but few know that this information is actually available in x$drc:

SQL> desc x$drc
 Name            Null?    Type
 --------------- -------- ---------------
 ADDR                     RAW(8)
 INDX                     NUMBER
 INST_ID                  NUMBER
 CON_ID                   NUMBER
 OBJECT_ID                NUMBER
 ATTRIBUTE                VARCHAR2(30)
 VALUE                    VARCHAR2(4000)
 PARENT_ID                VARCHAR2(15)
 STATUS                   VARCHAR2(30)
 MESSAGE                  VARCHAR2(256)
 ERRNUM                   NUMBER
 VALUE_RAW                RAW(4095)
 ERRTIME                  NUMBER

The table design is not very friendly, because it is a mix of ATTRIBUTE/VALUE pairs (hence many rows per object) and specific columns.

So, in order to get something usable to show the databases and their status, the only solution is to make use of PIVOT().

SELECT
    piv.*,
    obj.status
FROM
    (
        SELECT
            object_id,
            attribute,
            value
        FROM
            x$drc
        WHERE
            object_id IN (
                SELECT
                    object_id
                FROM
                    x$drc
                WHERE
                    attribute = 'DATABASE'
            )
    ) drc PIVOT (
        MAX ( value )
        FOR attribute
        IN (
    'DATABASE'                     DATABASE                     ,
    'intended_state'               intended_state               ,
    'connect_string'               connect_string               ,
    'RAC'                          RAC                          ,
    'enabled'                      enabled                      ,
    'role'                         role                         ,
    'receive_from'                 receive_from                 ,
    'ship_to'                      ship_to                      ,
    'dgb_connect'                  dgb_connect                  ,
    -- 'static_connect_identifier'    static_connect_identifier    ,
    'FSFOTargetValidity'           FSFOTargetValidity           
	)
    )
    piv
    JOIN x$drc obj ON ( obj.object_id = piv.object_id
                        AND obj.attribute = 'DATABASE' );

To get it in a friendly format, I recommend using SQLcl and settting

set sqlformat ansiconsole

This is what I get on a simple two databases configuration (primary/standby):

OBJECT_ID DATABASE         INTENDED_STATE      CONNECT_STRING                        RAC   ENABLED   ROLE       RECEIVE_FROM   SHIP_TO          DGB_CONNECT                           FSFOTARGETVALIDITY   STATUS
   16842752 ludocdb2_site1   READ-WRITE-XPTON    newbox01:1521/ludoCDB2_SITE1_DGMGRL   NO    YES       PRIMARY    -N/A-          ludocdb2_site2   newbox01:1521/ludoCDB2_SITE1_DGMGRL   2                    SUCCESS
   33619968 ludocdb2_site2   PHYSICAL-APPLY-ON   newbox02:1521/ludoCDB2_SITE2_DGMGRL   NO    YES       PHYSICAL   -UNKNOWN-      -N/A-            newbox02:1521/ludoCDB2_SITE2_DGMGRL   1                    SUCCESS

Although it is much easier to get this information from DGMGRL, get it programmatically is more sure/flexible using the SQL interface, as you know what you want to get, no matter how the dgmgrl syntax changes.

Looking forward to have the REST APIs in a future version of Data Guard 🙂

Ludovico

Viewing all 119 articles
Browse latest View live