As multitenant Oracle Databases becomes the standard for database implementations, you need to ensure that you maintain your recovery window even as your pluggable moves around your environment.

Above is the recommended practice we have all been hearing about to make upgrades of your Oracle Database easier. Unplug from your current CDB (CDBPROD122) plug into a new CDB (CDB19C) that has the new release.  What you need to think about however, is how am I  going to ensure that I can recover my pluggable database to any point in time, all the way this migration without a huge amount of downtime?

RMAN> backup incremental level 0 pluggable database PDBDWPROD plus archivelog delete input;


Starting backup at 15-APR-22
current log archived
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=426 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
channel ORA_SBT_TAPE_1: starting compressed archived log backup set
channel ORA_SBT_TAPE_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=185 RECID=186 STAMP=1102096068
input archived log thread=1 sequence=186 RECID=187 STAMP=1102096071
input archived log thread=1 sequence=187 RECID=188 STAMP=1102096147
input archived log thread=1 sequence=188 RECID=189 STAMP=1102096166
input archived log thread=1 sequence=189 RECID=190 STAMP=1102096288
channel ORA_SBT_TAPE_1: starting piece 1 at 15-APR-22
channel ORA_SBT_TAPE_1: finished piece 1 at 15-APR-22
piece handle=6l0r19t1_213_1_1 tag=TAG20220415T175129 comment=API Version 2.0,MMS Version 23.0.0.1
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
channel ORA_SBT_TAPE_1: deleting archived log(s)
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_185_k5mcy4w8_.arc RECID=186 STAMP=1102096068
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_186_k5mcy7xv_.arc RECID=187 STAMP=1102096071
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_187_k5md0m8l_.arc RECID=188 STAMP=1102096147
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_188_k5md16rt_.arc RECID=189 STAMP=1102096166
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_189_k5md50qy_.arc RECID=190 STAMP=1102096288
Finished backup at 15-APR-22

Starting backup at 15-APR-22
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting compressed incremental level 0 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00066 name=/u01/app/oracle/oradata/OLDCDB/PDBDWPROD/sysaux01.dbf
input datafile file number=00065 name=/u01/app/oracle/oradata/OLDCDB/PDBDWPROD/system01.dbf
input datafile file number=00068 name=/u01/app/oracle/oradata/OLDCDB/PDBDWPROD/PDBDWPROD.dbf
input datafile file number=00067 name=/u01/app/oracle/oradata/OLDCDB/PDBDWPROD/undotbs01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 15-APR-22
channel ORA_SBT_TAPE_1: finished piece 1 at 15-APR-22
piece handle=6m0r19t5_214_1_1 tag=TAG20220415T175132 comment=API Version 2.0,MMS Version 23.0.0.1
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15
Finished backup at 15-APR-22

Starting backup at 15-APR-22
current log archived
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting compressed archived log backup set
channel ORA_SBT_TAPE_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=190 RECID=191 STAMP=1102096309
channel ORA_SBT_TAPE_1: starting piece 1 at 15-APR-22
channel ORA_SBT_TAPE_1: finished piece 1 at 15-APR-22
piece handle=6n0r19tm_215_1_1 tag=TAG20220415T175150 comment=API Version 2.0,MMS Version 23.0.0.1
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
channel ORA_SBT_TAPE_1: deleting archived log(s)
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_190_k5md5osn_.arc RECID=191 STAMP=1102096309
Finished backup at 15-APR-22

Starting Control File and SPFILE Autobackup at 15-APR-22
piece handle=c-1180802953-20220415-07 comment=API Version 2.0,MMS Version 23.0.0.1
Finished Control File and SPFILE Autobackup at 15-APR-22

RMAN>

RMAN> backup archivelog all delete input;

Starting backup at 15-APR-22
current log archived
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=442 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
channel ORA_SBT_TAPE_1: starting compressed archived log backup set
channel ORA_SBT_TAPE_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=191 RECID=192 STAMP=1102096412
input archived log thread=1 sequence=192 RECID=193 STAMP=1102096418
input archived log thread=1 sequence=193 RECID=194 STAMP=1102096424
input archived log thread=1 sequence=194 RECID=195 STAMP=1102096502
channel ORA_SBT_TAPE_1: starting piece 1 at 15-APR-22
channel ORA_SBT_TAPE_1: finished piece 1 at 15-APR-22
piece handle=6p0r1a3n_217_1_1 tag=TAG20220415T175503 comment=API Version 2.0,MMS Version 23.0.0.1
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:07
channel ORA_SBT_TAPE_1: deleting archived log(s)
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_191_k5md8w57_.arc RECID=192 STAMP=1102096412
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_192_k5md926n_.arc RECID=193 STAMP=1102096418
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_193_k5md9893_.arc RECID=194 STAMP=1102096424
archived log file name=/u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_194_k5mdcpko_.arc RECID=195 STAMP=1102096502
Finished backup at 15-APR-22

Starting Control File and SPFILE Autobackup at 15-APR-22
piece handle=c-1180802953-20220415-08 comment=API Version 2.0,MMS Version 23.0.0.1
Finished Control File and SPFILE Autobackup at 15-APR-22

SQL>  alter pluggable database PDBDWPROD  close immediate;

Pluggable database altered.

SQL> ALTER PLUGGABLE DATABASE PDBDWPROD  UNPLUG INTO '/tmp/PDBDWPROD.xml';

Pluggable database altered.

SQL>

SQL>  create pluggable database PDBDWPROD using  '/tmp/PDBDWPROD.xml' nocopy tempfile reuse KEYSTORE IDENTIFIED BY "change-on-install" ;

Pluggable database created.

SQL>  alter pluggable database PDBDWPROD open;

Pluggable database altered.

SQL>  alter session set container=PDBDWPROD;

Session altered.

SQL> create table bgrenn.postmove as select * from dba_objects ;

Table created.

############################ perform a couple of log switches

SQL>  alter session set container=CDB$ROOT;
Session altered.

SQL> alter system archive log current;
System altered.

SQL> alter system archive log current;
System altered.

SQL>  alter session set container=PDBDWPROD;
Session altered.

############################ create a restore point

SQL> create restore point PDBDWPROD_restore;
Restore point created.

############################  create a second table

SQL> create table bgrenn.postrestorepoint as select * from dba_objects ;
Table created.

############################ perform a couple of log switches

SQL> alter session set container=CDB$ROOT;
Session altered.

SQL> alter system archive log current;
System altered.

SQL> alter system archive log current;
System altered.

SQL> alter system archive log current;
System altered.

rman> SET PREPLUGIN CONTAINER=PDBDWPROD;

rman> list preplugin backup of pluggable database PDBDWPROD;

RMAN>  list preplugin backup of pluggable database PDBDWPROD;

starting full resync of recovery catalog
full resync complete

List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
209     Incr 0  284.50M    SBT_TAPE    00:00:07     15-APR-22
        BP Key: 209   Status: AVAILABLE  Compressed: YES  Tag: TAG20220415T175132
        Handle: 6m0r19t5_214_1_1   Media: objectstorage.us-ashburn-1.oraclecloud.com/n/xxx/oldcdb
  List of Datafiles in backup set 209
  Container ID: 5, PDB Name: PDBDWPROD
  File LV Type Ckp SCN    Ckp Time  Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- --------- ----------- ------ ----
  59   0  Incr 6346380    15-APR-22              NO    /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/system01.dbf
  60   0  Incr 6346380    15-APR-22              NO    /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/sysaux01.dbf
  61   0  Incr 6346380    15-APR-22              NO    /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/undotbs01.dbf
  62   0  Incr 6346380    15-APR-22              NO    /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/PDBDWPROD.dbf

list preplugin backup of archivelog all;

List of Backup Sets
===================


BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
208     2.25M      SBT_TAPE    00:00:01     15-APR-22
        BP Key: 208   Status: AVAILABLE  Compressed: YES  Tag: TAG20220415T175129
        Handle: 6l0r19t1_213_1_1   Media: objectstorage.us-ashburn-1.oraclecloud.com/n/xxx/oldcdb

  List of Archived Logs in backup set 208
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    185     6345022    15-APR-22 6345387    15-APR-22
  1    186     6345387    15-APR-22 6345399    15-APR-22
  1    187     6345399    15-APR-22 6345803    15-APR-22
  1    188     6345803    15-APR-22 6345912    15-APR-22
  1    189     6345912    15-APR-22 6346322    15-APR-22

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
210     256.00K    SBT_TAPE    00:00:00     15-APR-22
        BP Key: 210   Status: AVAILABLE  Compressed: YES  Tag: TAG20220415T175150
        Handle: 6n0r19tm_215_1_1   Media: objectstorage.us-ashburn-1.oraclecloud.com/n/xxx/oldcdb

  List of Archived Logs in backup set 210
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    190     6346322    15-APR-22 6346391    15-APR-22

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
212     512.00K    SBT_TAPE    00:00:02     15-APR-22
        BP Key: 212   Status: AVAILABLE  Compressed: YES  Tag: TAG20220415T175503
        Handle: 6p0r1a3n_217_1_1   Media: objectstorage.us-ashburn-1.oraclecloud.com/n/id20skavsofo/oldcdb

  List of Archived Logs in backup set 212
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    191     6346391    15-APR-22 6346585    15-APR-22
  1    192     6346585    15-APR-22 6346593    15-APR-22
  1    193     6346593    15-APR-22 6346601    15-APR-22
  1    194     6346601    15-APR-22 6346663    15-APR-22

RMAN> alter pluggable database PDBDWPROD close;

Statement processed
starting full resync of recovery catalog
full resync complete

RMAN> restore pluggable database PDBDWPROD   from preplugin;

Starting restore at 15-APR-22
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1

channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00059 to /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/system01.dbf
channel ORA_SBT_TAPE_1: restoring datafile 00060 to /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/sysaux01.dbf
channel ORA_SBT_TAPE_1: restoring datafile 00061 to /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/undotbs01.dbf
channel ORA_SBT_TAPE_1: restoring datafile 00062 to /u01/app/oracle/oradata/OLDCDB/PDBDWPROD/PDBDWPROD.dbf
channel ORA_SBT_TAPE_1: reading from backup piece 6m0r19t5_214_1_1
channel ORA_SBT_TAPE_1: piece handle=6m0r19t5_214_1_1 tag=TAG20220415T175132
channel ORA_SBT_TAPE_1: restored backup piece 1
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:25
Finished restore at 15-APR-22

 recover pluggable database PDBDWPROD   from preplugin;
RMAN>

Starting recover at 15-APR-22
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1

starting media recovery

channel ORA_SBT_TAPE_1: starting archived log restore to default destination
channel ORA_SBT_TAPE_1: restoring archived log
archived log thread=1 sequence=190
channel ORA_SBT_TAPE_1: reading from backup piece 6n0r19tm_215_1_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 04/15/2022 18:23:58
ORA-19870: error while restoring backup piece 6n0r19tm_215_1_1
ORA-19827: Restoring preplugin files to a recovery area is not supported.

RMAN>

RMAN> restore archivelog sequence 190;

Starting restore at 15-APR-22
starting full resync of recovery catalog
full resync complete
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=449 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=71 device type=DISK

channel ORA_SBT_TAPE_1: starting archived log restore to default destination
channel ORA_SBT_TAPE_1: restoring archived log
archived log thread=1 sequence=190
channel ORA_SBT_TAPE_1: reading from backup piece 6n0r19tm_215_1_1
channel ORA_SBT_TAPE_1: piece handle=6n0r19tm_215_1_1 tag=TAG20220415T175150
channel ORA_SBT_TAPE_1: restored backup piece 1
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:01
Finished restore at 15-APR-22

RMAN> list archivelog sequence 190;

List of Archived Log Copies for database with db_unique_name OLDCDB
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
8134    1    190     A 15-APR-22
        Name: /u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_190_k5mgd1h2_.arc

RMAN> restore archivelog sequence 190;

Starting restore at 15-APR-22
starting full resync of recovery catalog
full resync complete
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=449 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=71 device type=DISK

channel ORA_SBT_TAPE_1: starting archived log restore to default destination
channel ORA_SBT_TAPE_1: restoring archived log
archived log thread=1 sequence=190
channel ORA_SBT_TAPE_1: reading from backup piece 6n0r19tm_215_1_1
channel ORA_SBT_TAPE_1: piece handle=6n0r19tm_215_1_1 tag=TAG20220415T175150
channel ORA_SBT_TAPE_1: restored backup piece 1
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:01
Finished restore at 15-APR-22

RMAN> list archivelog sequence 190;

List of Archived Log Copies for database with db_unique_name OLDCDB
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - ---------
8134    1    190     A 15-APR-22
        Name: /u01/app/oracle/fast_recovery_area/OLDCDB/archivelog/2022_04_15/o1_mf_1_190_k5mgd1h2_.arc

RMAN>  recover pluggable database PDBDWPROD   from preplugin;

Starting recover at 15-APR-22
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 191 is already on disk as file /tmp/o1_mf_1_191_k5mgw2fr_.arc
archived log for thread 1 with sequence 192 is already on disk as file /tmp/o1_mf_1_192_k5mgw83q_.arc
archived log for thread 1 with sequence 193 is already on disk as file /tmp/o1_mf_1_193_k5mgwlf8_.arc
archived log for thread 1 with sequence 194 is already on disk as file /tmp/o1_mf_1_194_k5mgx0t1_.arc
unable to find archived log
archived log thread=1 sequence=195
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 04/15/2022 18:40:31
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 195 and starting SCN of 6346663

SQL> select sequence#,status from v$log;

 SEQUENCE# STATUS
---------- ----------------
       195 CURRENT
       193 INACTIVE
       194 INACTIVE

RMAN> recover pluggable database PDBDWPROD   from preplugin;

Starting recover at 15-APR-22
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 195 is already on disk as file /tmp/o1_mf_1_195_k5mhf79h_.arc
media recovery complete, elapsed time: 00:00:01
Finished recover at 15-APR-22

RMAN>

RMAN> recover pluggable database PDBDWPROD  until restore point PDBDWPROD_restore;

Starting recover at 15-APR-22
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4 device type=DISK


starting media recovery

archived log for thread 1 with sequence 101 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_101_k5mf0rr5_.arc
archived log for thread 1 with sequence 102 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_102_k5mf0s8b_.arc
archived log for thread 1 with sequence 103 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_103_k5mf1wof_.arc
archived log for thread 1 with sequence 104 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_104_k5mf1zqm_.arc
archived log for thread 1 with sequence 105 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_105_k5mf22sk_.arc
archived log for thread 1 with sequence 106 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_106_k5mf91fn_.arc
archived log for thread 1 with sequence 107 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_107_k5mf94g2_.arc
archived log for thread 1 with sequence 108 is already on disk as file /u01/app/oracle/fast_recovery_area/NEWCDB/archivelog/2022_04_15/o1_mf_1_108_k5mf9wk2_.arc
media recovery complete, elapsed time: 00:00:01
Finished recover at 15-APR-22

RMAN> alter pluggable database PDBDWPROD open resetlogs;

Statement processed
starting full resync of recovery catalog
full resync complete

SQL>  alter session set container=PDBDWPROD;

Session altered.

SQL> select table_name from dba_tables where owner='BGRENN';

TABLE_NAME
--------------------------------------------------------------------------------
POSTMOVE

SQL> select count(1) from bgrenn.postmove;

  COUNT(1)
----------
     73610

 Recovery Continuity with Multitenant is something you need to understand as you migrate databases from one CDB to another

Above is from a presentation that I recently gave to my internal Oracle Team. This was such a big hit (and very eye opening) that I wanted to make sure I share the information on my Blog

The first thing to point out is that before we (Oracle) moved to the multitenant architecture, life was simple. Below is my slide showing how databases were moving around as they upgraded.  Regardless of whether it was an out-of-place upgrade, or migrating it to a different host, the DB name stays, and the backups stay contiguous.

But, like many things in life, new ideas came along that changed the way we do things.  Multitenant is one of those things.  Don't get me wrong, multitenant is a great feature giving DBAs a lot more flexibility.  Below are a couple of pictures that show all the wonderful things that multitenant can do.

Above are the 2 slides from my presentation.  These slides are often used to show the benefits of multitenant.  I did point out on the last slide encryption keys that are used to secure the database with TDE.

The use of Encryption keys is an important point to think about.  With Multitenant (if you think about how it works), the CDB has a different encryption key from the PDB.  If I create an encrypted backup of my CDB, it is encrypted with the CDB key. The backup (and actual datafiles) for my PDB are encrypted with the PDB key.

Below is the next slide I used.  All the information on multitenant talks about how easy it is to unplug/plug (which it is), but ensuring you maintain your recovery window is the hard part.

Database backup and recovery in a multitenant environment

• Backups that were taken through previous CDBs (for example an archival backup) needs to be restored through the CDB is was backed up through.
• Backups that were taken in original CDB can only be restored back to the original CDB.
• Pre-plugin backups provide a gateway between plugging in and when the first backup is taken
• Backups to the new CDB will restored back to the new CDB.

 Object Retention Rules is one of the new features that was released in ZFSSA version 8.8.36.  Before I talk about Object Retention Rules on buckets on ZFSSA, I am going to go how to leverage the new access control polies that go along with managing objects, buckets, and retention. 

User Architecture

If you want to read my post on configuring authentication you can find it here.

What I want to go through in this post is how you can configure a set of user roles on ZFSSA with different permissions based public/private keys.

This will help you isolate and secure backups that were sent from multiple sources, and allow you to define both a security administrator (to apply retention policies), and an auditor to view the existence of backups without having the ability to delete or update backups.

In the "User Architecture" diagram at the beginning of this post you see that I have defined 5 user roles  that will be used to manage the object store security for the backups.

Users:

• SECADMIN - This user role is the security administrator for all 3 object store backups, and all three buckets.  This user role is responsible for creating, deleting and assigning retention rules to the buckets.
• AUDITOR - This user reviews the backups and has a read only view of all 3 backups. The auditor cannot delete or update any objects, but they can view the existence of the backup pieces.
• GLUSER  - This user controls the backups for GLDB only
• APUSER  - This user controls the backups for APDB only
• DWUSER - This user controls the backups for DWDB only

NOTE: Because the Object Store API controls the access to objects in the bucket, all access to objects in a bucket is through the bucket owner. I can have multiple buckets on the same share, managed by different different users, but access WITHIN the bucket is only granted to the bucket owner.

Based on the above note, I am going to create 3 users to manage the buckets for the 3 database backups.

The 2 additional user roles, SECADMIN and AUDITOR are going to control their access through the use of RSA keys.

 Because I am not going to use pre-authenticated URLs for my backups (which requires login), all 3 users are going to be created as "no-login" users.    Below is an example of creating the APUSER.

I created all 3 users as no-login users

Project/Share for Object Storage

Now I am going to create a project and share to store the backup pieces for all 3 databases.  The project is going to be "dbbackups" and the share is going to be "dbbackups".  I am going to set the default user for the share to "oracle" and I am also going to grant the other 3 users "Full Control" of the share. I will later limit the permissions for these users.

Share User Access

User certificates:

Authentication to the object store is through the use of RSA public/private certificates.

For each user/role I created a certificate that will be used for authentication. 

The following table shows the users/roles and the fingerprint that identifies them.

Authentication:

Within the OCI service on the ZFSSA I combine the user and key (fingerprint) to provide the role.

First I will add the SECADMIN role.  Notice that I am adding this users access to all 3 database backup "users".  This will allow the SECADMIN role to manage bucket creation/deletion and retention to the individual buckets.  The role SECADMIN is accessed through the key.

I will start by adding the key owned by this role (SECADMIN) to the 3 users APUSER, GLUSER and DWUSER.

Now that I have the SECADMIN role assigned to the 3 users, I want set the proper capabilities for this role.  I click on the pencil to edit the key configuration, and I can see the permissions assigned to this user/key combination.  I want to allow the SECADMIN the ability create buckets, delete buckets and  to control the retention within the 3 users buckets.  This role will need the ability to read the bucket.  Notice that this role does not have ability to read any of the objects within the bucket

Now I am going to move on to the AUDITOR role.  This role will be configured using the AUDITOR key assigned to all 3 users.  Within each user the AUDITOR will be granted the ability to read the bucket and the objects but not make any changes.

I now have both the SECADMIN role and the AUDITOR role defined for all 3 users. Below is what is configured within the OCI service. Notice that that there are 2 keys set for each user, and the there are 2 unique keys (one for SECADMIN and one for AUDITOR).

Finally I am going to add the 3 users that own the buckets and grant them access to create objects, but not control the retention or be able to add/remove buckets.

Once completed with adding users/keys I have my 2 roles defined and assigned to each user, and I have an individual key for each user/backup.

When completed, the chart below shows the permissions for each user/role.

OCI cli configuration :

I added entries to the ~/.oci/config file for each of the users/roles configured for the service.

Below is an example entry for the SECADMIN role with the APDB bucket.

[SECADMIN_APDB]
user=ocid1.user.oc1..apuser
fingerprint=0a:35:21:1b:5c:eb:09:8c:e9:44:42:f2:7c:b5:bc:f6
key_file=~/keys/secadmin.ppk
tenancy=ocid1.tenancy.oc1..nobody
region=us-phoenix-1
endpoint=http://150.136.215.19
os.object.bucket-name=apdb
namespace-name=dbbackups
compartment-id=dbbackups

Below is a table of the entries that I added to the config file.

Creating buckets:

Now I am going to create my 3 buckets using the SECADMIN role. Below is an example of adding the bucket for APDB

[oracle@oracle-19c-test-tde keys]$ oci os bucket create  --namespace-name dbbackups  --endpoint http://150.136.215.19  --config-file ~/.oci/config --profile SECADMIN_APDB    --name apdb  --compartment-id dbbackups
{
"data": {
"approximate-count": null,
"approximate-size": null,
"auto-tiering": null,
"compartment-id": "dbbackups",
"created-by": "apuser",
"defined-tags": null,
"etag": "2f0b55dbbb925ebbaabbc37e3ce342fa",
"freeform-tags": null,
"id": "2f0b55dbbb925ebbaabbc37e3ce342fa",
"is-read-only": null,
"kms-key-id": null,
"metadata": null,
"name": "apdb",
"namespace": "dbbackups",
"object-events-enabled": null,
"object-lifecycle-policy-etag": null,
"public-access-type": "NoPublicAccess",
"replication-enabled": null,
"storage-tier": "Standard",
"time-created": "2022-05-17T17:55:49+00:00",
"versioning": "Disabled"
  },
"etag": "2f0b55dbbb925ebbaabbc37e3ce342fa"
}

I then did the same thing for the GLDB bucket using SECADMIN_GLDB, and the DWDB bucket using SECADMIN_DWDB.

Once the buckets were created, I attempted to create buckets with both the AUDITOR role, and the DB role.  You can see below that both of these configurations did not have the correct privileges.

[oracle@oracle-19c-test-tde keys]$ oci os bucket create  --namespace-name dbbackups  --endpoint http://150.136.215.19  --config-file ~/.oci/config --profile AUDITOR_APDB    --name apdb  --compartment-id dbbackups
ServiceError:
{
"code": "BucketNotFound",
"message": "Either the bucket does not exist in the namespace or you are not authorized to access it",
"opc-request-id": "tx3a37f1dee0cc4778a1201-006283e2a1",
"status": 404
}
[oracle@oracle-19c-test-tde keys]$ oci os bucket create  --namespace-name dbbackups  --endpoint http://150.136.215.19  --config-file ~/.oci/config --profile APDB    --name apdb  --compartment-id dbbackups
ServiceError:
{
"code": "BucketNotFound",
"message": "Either the bucket does not exist in the namespace or you are not authorized to access it",
"opc-request-id": "tx46435ae6b8234982b3fbd-006283e2a9",
"status": 404
}

[oracle@oracle-19c-test-tde keys]$ oci os bucket list --namespace-name dbbackups  --endpoint http://150.136.215.19  --config-file ~/.oci/config --profile SECADMIN_APDB    --compartment-id dbbackups

{
"data": [
    {
"compartment-id": "dbbackups",
"created-by": "apuser",
"defined-tags": null,
"etag": "2f0b55dbbb925ebbaabbc37e3ce342fa",
"freeform-tags": null,
"name": "apdb",
"namespace": "dbbackups",
"time-created": "2022-05-17T17:55:49+00:00"
    },
    {
"compartment-id": "dbbackups",
"created-by": "dwuser",
"defined-tags": null,
"etag": "866ded83e5ea2a29c66dca0d01036f0e",
"freeform-tags": null,
"name": "dwdb",
"namespace": "dbbackups",
"time-created": "2022-05-17T17:58:32+00:00"
    },
    {
"compartment-id": "dbbackups",
"created-by": "gluser",
"defined-tags": null,
"etag": "2169cf94f86009f66ca8770c1c58febb",
"freeform-tags": null,
"name": "gldb",
"namespace": "dbbackups",
"time-created": "2022-05-17T17:58:17+00:00"
    }
  ]
}

[oracle@oracle-19c-test-tde keys]$ oci os retention-rule create --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile SECADMIN_APDB --bucket-name apdb --time-amount 30  --time-unit days --display-name APDB-30-day-Bound-backups
{
"data": {
"display-name": "APDB-30-day-Bound-backups",
"duration": {
"time-amount": 30,
"time-unit": "DAYS"
    },
"etag": "2c9ab8ff9c4743392d308365d9f72e05",
"id": "2c9ab8ff9c4743392d308365d9f72e05",
"time-created": "2022-05-17T18:49:24+00:00",
"time-modified": "2022-05-17T18:49:24+00:00",
"time-rule-locked": null
  }
}

[oracle@oracle-19c-test-tde keys]$ oci os retention-rule create --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_GLDB --bucket-name gldb --time-amount 30  --time-unit days --display-name APDB-30-day-Bound-backups
ServiceError:
{
"code": "BucketNotFound",
"message": "Either the bucket does not exist in the namespace or you are not authorized to access it",
"opc-request-id": "tx52e8849aa6444c639d59b-006283ee99",
"status": 404
}

[oracle@oracle-19c-test-tde keys]$ oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_APDB --bucket-name apdb
oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_GLDB --bucket-name gldb
oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_DWDB --bucket-name dwdb


{
"data": {
"items": [
      {
"display-name": "APDB-30-day-Bound-backups",
"duration": {
"time-amount": 30,
"time-unit": "DAYS"
        },
"etag": "2c9ab8ff9c4743392d308365d9f72e05",
"id": "2c9ab8ff9c4743392d308365d9f72e05",
"time-created": "2022-05-17T18:49:24+00:00",
"time-modified": "2022-05-17T18:49:24+00:00",
"time-rule-locked": null
      }
    ]
  }
}
[oracle@oracle-19c-test-tde keys]$ oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_GLDB --bucket-name gldb
{
"data": {
"items": [
      {
"display-name": "GLDB-30-day-Bound-backups",
"duration": {
"time-amount": 30,
"time-unit": "DAYS"
        },
"etag": "ee0d6114310a9971f5a464b428916e48",
"id": "ee0d6114310a9971f5a464b428916e48",
"time-created": "2022-05-17T18:56:45+00:00",
"time-modified": "2022-05-17T18:56:45+00:00",
"time-rule-locked": null
      }
    ]
  }
}
[oracle@oracle-19c-test-tde keys]$ oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_DWDB --bucket-name dwdb
{
"data": {
"items": [
      {
"display-name": "DWDB-30-day-Bound-backups",
"duration": {
"time-amount": 30,
"time-unit": "DAYS"
        },
"etag": "96cc109a7308d5f849541be72d87757a",
"id": "96cc109a7308d5f849541be72d87757a",
"time-created": "2022-05-17T18:57:42+00:00",
"time-modified": "2022-05-17T18:57:42+00:00",
"time-rule-locked": null
      }
    ]
  }
}

Audit Backups :

[oracle@oracle-19c-test-tde keys]$ oci os retention-rule list --endpoint http://150.136.215.19 --namespace-name dbbackups --config-file ~/.oci/config --profile AUDITOR_APDB --bucket-name apdb
{
"data": {
"items": [
      {
"display-name": "APDB-30-day-Bound-backups",
"duration": {
"time-amount": 30,
"time-unit": "DAYS"
        },
"etag": "2c9ab8ff9c4743392d308365d9f72e05",
"id": "2c9ab8ff9c4743392d308365d9f72e05",
"time-created": "2022-05-17T18:49:24+00:00",
"time-modified": "2022-05-17T18:49:24+00:00",
"time-rule-locked": null
      }
    ]
  }
}

[oracle@oracle-19c-test-tde ~]$ python2  /home/oracle/ociconfig/lib/odbsrmt.py --mode report --ocitype bmc  --host http://150.136.215.19 --dir /home/oracle/keys/reports --base apdbreport --pvtkeyfile  /home/oracle/keys/auditor.ppk --pubfingerprint a8:31:78:c2:b4:4f:44:93:bd:4f:f1:72:1c:37:c8:86 --tocid ocid1.tenancy.oc1..nobody --uocid ocid1.user.oc1..apuser --container apdb  --dbid 2867715978
odbsrmt.py: ALL outputs will be written to [/home/oracle/keys/reports/apdbreport12193.lst]
odbsrmt.py: Processing container apdb...
cloud_slave_processors: Thread Thread_0 starting to download metadata XML files...
cloud_slave_processors: Thread Thread_0 successfully done
odbsrmt.py: ALL outputs have been written to [/home/oracle/keys/reports/apdbreport12193.lst]

FileName
Container                Dbname         Dbid        FileSize          LastModified                BackupType                  Incremental  Compressed   Encrypted
870toeq3_263_1_1
apdb                     ORCLCDB        2867715978  1285029888        2022-05-17 19:09:45         Datafile                    true         false        true   
890toetk_265_1_1
apdb                     ORCLCDB        2867715978  2217476096        2022-05-17 19:12:17         ArchivedLog                 false        false        true   
8a0tof0j_266_1_1
apdb                     ORCLCDB        2867715978  2790260736        2022-05-17 19:14:15         Datafile                    true         false        true   
8b0tof4g_267_1_1
apdb                     ORCLCDB        2867715978  2124677120        2022-05-17 19:15:52         Datafile                    true         false        true   
8c0tof7f_268_1_1
apdb                     ORCLCDB        2867715978  536346624         2022-05-17 19:16:21         Datafile                    true         false        true   
8d0tof89_269_1_1
apdb                     ORCLCDB        2867715978  262144            2022-05-17 19:16:25         ArchivedLog                 false        false        true   
c-2867715978-20220517-00
apdb                     ORCLCDB        2867715978  18874368          2022-05-17 19:09:47         ControlFile SPFILE          false        false        true   
c-2867715978-20220517-01
apdb                     ORCLCDB        2867715978  18874368          2022-05-17 19:16:26         ControlFile SPFILE          false        false        true   
Total Storage: 8.37 GB

Conclusion :

 The latest release of ZFSSA Software OS8.8.45 includes file retention lock joining object retention lock and snapshot retention lock.

3 types of retention lock

Legal Hold

You might need to preserve certain business data in response to potential or on-going lawsuits. A legal hold does not have a defined retention period and remains in effect until removed.  Once the legal hold is removed, all protected data is immediately eligible for deletion.

Data Governance

Data Governance locks data sets (snapshot, object or file) for a period of time protecting the data from deletion.  You might need to protect certain data sets as a part of internal business process requirements or protect data sets from a cyber attack. While retaining the data for a defined length of time is necessary, that time period could change.

Regulatory Compliance

Your industry might require you to retain a certain class of data for a defined length of time. Your data retention regulations might also require that you lock the retention settings. Regulatory compliance only allows you to increase the retention time.

3 implementations of retention lock

Object storage

Object storage retention is managed through the OCI client tool and Object retention is enforced through the API. Current retention settings are applied to all objects when they are accessed.  Adding a rule immediately takes affect for all objects.  

Administration of retention rules can be managed through the use of RSA certificates.  It is recommended to create a separation of duties between a security administrator, and the object owner.

Retention on object storage is implemented in the following way based on the retention lock type.

Legal hold

Legal holds are implemented by placing an indefinite retention rule on a bucket.  Creating this rule ensures that all objects within the bucket can not be deleted, and cannot be changed. Only new objects can be stored.

Data Governance

Data Governance is implemented by placing a time bound retention rule on a bucket.  The rule sets a lock on all objects for a set length of time.  The rule can be later deleted. For cyber protection it is recommended to implement this with a separation of duties.

Regulatory Compliance

Regulatory Compliance is implemented by placing a locked time bound retention rule on a bucket with a grace period.  When a locked time bound retention rule is created it immediately takes effect, but there is a grace period of at least 14 days which allows you to test the rule. Once the grace period expires the rule cannot be deleted even by an administrator.

Snapshots

Snapshot locking is managed their the BUI, or CLI.  Individual snapshots can be locked, and scheduled snapshots can be created and automatically locked.  Permission for controlling snapshot locking can be assigned to ZFSSA users allowing to create a separation of duties. Shared or projects cannot be removed if they contained locked snapshots.

Retention on snapshots is implemented in the following way based on the retention lock type.

Legal hold

Legal holds on snapshots is handled by creating a snapshot, and locking the snapshot. The snapshot cannot be removed until the lock on the snapshot is removed.  There is no mechanism to schedule unlimited snapshots. but it is possible to create a large number of daily snapshots that are retained as locked snapshots for 1000s+ of days.

Data Governance

Data governance of snapshots is handled through the use of scheduled locked snapshots.  A schedule is created with both a retention, and "keep at most" setting. This allows you to manage snapshots for a locked number of snapshots, while automatically cleaning up snapshots that are past the retention number.  The snapshots can be unlocked and removed, and the schedule can be removed by an administrator with the correct privileges.

Regulatory Compliance

Regulatory compliance of snapshots is handled through the use of locked snapshot schedules.  Similar to data governance, a snapshot schedule is created, but when regulatory compliance is set, the schedule cannot be decreased or removed as long as data exists within the snapshot.  

File Retention

File retention is set at the share or project level and controls updating and deletion of all data contained on the share/project.  A default file retention is set and all new files will inherit the default setting in effect when the file is created. It is also possible to manually set the retention on a file overriding the default setting inherited by the file.

Legal Hold

Legal hold cannot be easily implemented with file retention.  File retention of individual files is set at file creation, and can only be changed manually.

Data Governance

Data governance is implemented by creating a NEW project and share with a file retention policy of privileged.  Privileged mode allows you to create a default retention setting for all new files, and change that setting (longer or shorter) going forward.  

Regulatory Compliance

Regulatory compliance  is implemented by creating a NEW project and share with a file retention policy of mandatory (no override).  Mandatory mode does not allow you to decrease the default file retention.  The project/share cannot be removed when locked files exist, and the storage pool cannot be removed when locked files exist within the pool. This mode also requires an NTP server be utilized, and root is locked out of any remote access.

The best way to explore these new features is by using the ZFSSA image in OCI to test different scenarios.

 dddd

 test55

 Migrating an Oracle database from on-premise to OCI is especially challenging when the database is quite large.  In this blog post I will walk through the steps to migrate to OCI leveraging an on-disk local backup copied to object storage.

Step #1 - Upload backup pieces to object storage.

Step #1A - Take a full backup to a separate location on disk 

RMAN> show all;

RMAN configuration parameters for database with db_unique_name ACMEDBP are:


CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   '/acmedb/ocimigrate/backup_%d_%U';
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;

List of Backups
===============
Key     TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
4125    B  A  A DISK        21-JUN-22       1       1       YES        TAG20220621T141019
4151    B  A  A DISK        21-JUN-22       1       1       YES        TAG20220621T141201
4167    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4168    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4169    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4170    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4171    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4172    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4173    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4174    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4175    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4176    B  0  A DISK        21-JUN-22       1       1       YES        TAG20220621T141202
4208    B  A  A DISK        21-JUN-22       1       1       YES        TAG20220621T141309
4220    B  F  A DISK        21-JUN-22       1       1       YES        TAG20220621T141310

 oci os bucket create --namespace id2avsofo --name acmedb_migrate --compartment-id ocid1.compartment.oc1..aaaaaaaanqbquh2bwju4igabu5g7clir2b4ykd3tyq
{
"data": {
"approximate-count": null,
"approximate-size": null,
"auto-tiering": null,
"compartment-id": "ocid1.compartment.oc1..aaaaaaaanqbquh2bwju4igabu5g7clir2b4ykd3tyq",
"created-by": "ocid1.user.oc1..aaaaaaaanqbquh2bwju4igabu5g7clir2b4ykd3tyq",
"defined-tags": {
"Oracle-Tags": {
"CreatedBy": "oracleidentitycloudservice/john.smith@oracle.com",
"CreatedOn": "2022-06-21T14:36:19.680Z"
      }
    },
"etag": "e0f028ac-d80d-4e09-8e60-876d90f57893",
"freeform-tags": {},
"id": "ocid1.bucket.oc1.iad.aaaaaaaanqbquh2bwju4igabu5g7clir2b4ykd3tyq",
"is-read-only": false,
"kms-key-id": null,
"metadata": {},
"name": "acmedb_migrate",
"namespace": "id2avsofo",
"object-events-enabled": false,
"object-lifecycle-policy-etag": null,
"public-access-type": "NoPublicAccess",
"replication-enabled": false,
"storage-tier": "Standard",
"time-created": "2022-06-21T14:36:19.763000+00:00",
"versioning": "Disabled"
  },
"etag": "e0f028ac-d80d-4e09-8e60-876d90f57893"
}

 oci os object bulk-upload --namespace-name id20skavsofo    --bucket-name acmedb_migrate --src-dir /acmedb/ocimigrate/ --parallel-upload-count 10

Uploaded backup_RADB_3u10k6hj_126_1_1  [####################################]  100%
Uploaded backup_RADB_4710k6jl_135_1_1  [####################################]  100%
Uploaded backup_RADB_4610k6jh_134_1_1  [####################################]  100%
Uploaded backup_RADB_3n10k6b0_119_1_1  [####################################]  100%
Uploaded backup_RADB_3m10k6b0_118_1_1  [####################################]  100%
Uploaded backup_RADB_3r10k6ec_123_1_1  [####################################]  100%
Uploaded backup_RADB_4510k6jh_133_1_1  [####################################]  100%
Uploaded backup_RADB_4010k6hj_128_1_1  [####################################]  100%
Uploaded backup_RADB_3v10k6hj_127_1_1  [####################################]  100%
Uploaded backup_RADB_4110k6hk_129_1_1  [####################################]  100%
Uploaded backup_RADB_4210k6id_130_1_1  [####################################]  100%
Uploaded backup_RADB_4310k6ie_131_1_1  [####################################]  100%
Uploaded backup_RADB_3l10k6b0_117_1_1  [####################################]  100%
Uploaded backup_RADB_4410k6ie_132_1_1  [####################################]  100%
Uploaded backup_RADB_3k10k6b0_116_1_1  [####################################]  100%
Uploaded backup_RADB_3t10k6hj_125_1_1  [####################################]  100%

{
"skipped-objects": [],
"upload-failures": {},
"uploaded-objects": {
"backup_RADB_3k10k6b0_116_1_1": {
"etag": "ab4a1017-3ba7-46e2-a2ee-3f4cd9a82ad3",
"last-modified": "Tue, 21 Jun 2022 14:57:42 GMT",
"opc-multipart-md5": "W0hYIzfAWUVzACWNudcQDg==-3"
    },
"backup_RADB_3l10k6b0_117_1_1": {
"etag": "a620076e-975f-4d8c-87e8-394c4cf966cd",
"last-modified": "Tue, 21 Jun 2022 14:57:41 GMT",
"opc-multipart-md5": "zapGBx8Imcdk91JM2+gORQ==-3"
    },
"backup_RADB_3m10k6b0_118_1_1": {
"etag": "a96c35c0-4c0b-4646-ae38-723f92c8496e",
"last-modified": "Tue, 21 Jun 2022 14:57:32 GMT",
"opc-content-md5": "vNAsU3vLcjzp6OwEeLXGgA=="
    },
"backup_RADB_3n10k6b0_119_1_1": {
"etag": "8f565894-5097-4ebb-9569-fdd31cc0c22d",
"last-modified": "Tue, 21 Jun 2022 14:57:31 GMT",
"opc-content-md5": "aSUSQWv5b+EfoLy9L9UBYQ=="
    },
"backup_RADB_3r10k6ec_123_1_1": {
"etag": "120dead4-c8ae-44de-9d27-39e1c28a2c48",
"last-modified": "Tue, 21 Jun 2022 14:57:33 GMT",
"opc-content-md5": "4wHBrgZXuIMlYWriBbs1ng=="
    },
"backup_RADB_3s10k6hh_124_1_1": {
"etag": "07d74b7f-68d6-4a77-9c4d-42f78c51c692",
"last-modified": "Tue, 21 Jun 2022 14:57:28 GMT",
"opc-content-md5": "uzRd51bAKvFjhbbsfL1YAg=="
    },
"backup_RADB_3t10k6hj_125_1_1": {
"etag": "e5d3225b-a687-47e1-ad31-f4270ce31ddd",
"last-modified": "Tue, 21 Jun 2022 14:57:42 GMT",
"opc-multipart-md5": "aZIirf98ZNqwBAlIeWzuhQ==-3"
    },
"backup_RADB_3u10k6hj_126_1_1": {
"etag": "5f5cc5ad-4aa3-4c3a-8848-16b3442a1e2c",
"last-modified": "Tue, 21 Jun 2022 14:57:28 GMT",
"opc-content-md5": "dT6EYLv1yzf6LZCn1/Dsvw=="
    },
"backup_RADB_3v10k6hj_127_1_1": {
"etag": "297daece-be72-475f-b40d-982fb7115cd3",
"last-modified": "Tue, 21 Jun 2022 14:57:36 GMT",
"opc-content-md5": "Zt3h5YfHU6F771ahltYhDQ=="
    },
"backup_RADB_4010k6hj_128_1_1": {
"etag": "9d723f2a-962e-4d03-9283-fc8a68f53af8",
"last-modified": "Tue, 21 Jun 2022 14:57:35 GMT",
"opc-content-md5": "KuNzVyUQrrSsA/kgioq9oA=="
    },
"backup_RADB_4110k6hk_129_1_1": {
"etag": "16f7f02a-e5ae-48a2-a7d2-b6d1dedc82ad",
"last-modified": "Tue, 21 Jun 2022 14:57:36 GMT",
"opc-content-md5": "24SzzZwg7iu7PV8TBpMXEg=="
    },
"backup_RADB_4210k6id_130_1_1": {
"etag": "0584e14f-53dc-4251-8bad-907f357a283e",
"last-modified": "Tue, 21 Jun 2022 14:57:37 GMT",
"opc-content-md5": "sjPsmoeFsMhZISAmaVN0vQ=="
    },
"backup_RADB_4310k6ie_131_1_1": {
"etag": "176aea41-dd31-4404-99f4-ffd59c521fd3",
"last-modified": "Tue, 21 Jun 2022 14:57:40 GMT",
"opc-content-md5": "2ksAQ2UuU/75YyRKujlLXg=="
    },
"backup_RADB_4410k6ie_132_1_1": {
"etag": "766c7585-3837-490b-8563-f3be3d24c98e",
"last-modified": "Tue, 21 Jun 2022 14:57:41 GMT",
"opc-content-md5": "sh4CFUC/vnxjmMZ5mfgT3Q=="
    },
"backup_RADB_4510k6jh_133_1_1": {
"etag": "2de62d73-e44c-4f25-a41d-d45c556054dd",
"last-modified": "Tue, 21 Jun 2022 14:57:34 GMT",
"opc-content-md5": "4tVrHqwYG57STn9W6c2Mqw=="
    },
"backup_RADB_4610k6jh_134_1_1": {
"etag": "4667419d-9555-4edb-bd6d-749a1ee7660b",
"last-modified": "Tue, 21 Jun 2022 14:57:29 GMT",
"opc-content-md5": "/MVdDn/vA2IXUcCmtdgKnw=="
    },
"backup_RADB_4710k6jl_135_1_1": {
"etag": "d467810a-d62e-42b3-bf7b-019913707312",
"last-modified": "Tue, 21 Jun 2022 14:57:29 GMT",
"opc-content-md5": "hq8PEQ3PUwyTMWyUBfW4ew=="
    }
  }
}

export ORACLE_SID=dummy
 rman target /
RMAN> startup nomount;

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/19c/dbhome_1/dbs/initdummy.ora'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area    1073737792 bytes

Fixed Size                     8904768 bytes
Variable Size                276824064 bytes
Database Buffers             780140544 bytes
Redo Buffers                   7868416 bytes

RMAN> run {
          allocate channel t1 device type sbt parms='SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
       send channel t1 'export backuppiece backup_RADB_3r10k6ec_123_1_1';
        }
2> 3> 4>
allocated channel: t1
channel t1: SID=19 device type=SBT_TAPE
channel t1: Oracle Database Backup Service Library VER=23.0.0.1

sent command to channel: t1
released channel: t1

RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';


  run {
           catalog device type 'sbt_tape' backuppiece 'backup_RADB_3r10k6ec_123_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_3s10k6hh_124_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_3t10k6hj_125_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_3u10k6hj_126_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_3v10k6hj_127_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_4010k6hj_128_1_1';
          catalog device type 'sbt_tape' backuppiece ' backup_RADB_4110k6hk_129_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_4210k6id_130_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_4310k6ie_131_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_4410k6ie_132_1_1';
          catalog device type 'sbt_tape' backuppiece 'backup_RADB_4510k6jh_133_1_1';
        }

old RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

RMAN>
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13>
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=406 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: SID=22 device type=SBT_TAPE
channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: SID=407 device type=SBT_TAPE
...
...
...
channel ORA_SBT_TAPE_4: SID=23 device type=SBT_TAPE
channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=23.0.0.1
channel ORA_SBT_TAPE_1: cataloged backup piece
backup piece handle=backup_RADB_4510k6jh_133_1_1 RECID=212 STAMP=1107964867

RMAN>

RMAN> list backup summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
4220    B  F  A DISK        21-JUN-22       1       1       YES        TAG20220621T141310
4258    B  A  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141019
4270    B  A  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141201
4282    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4292    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4303    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4315    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4446    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4468    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4490    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4514    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202
4539    B  0  A SBT_TAPE    21-JUN-22       1       1       YES        TAG20220621T141202

RMAN>

retentionPeriods

• "Minimum file retention period" - This is the minimum amount of time in the future that you can set a file retention to be. If you set the file retention date manually the retention time must be at least this far if not longer in the future. If you set the "Default file retention period", it must be at least the "Minimum file retention period" if not longer.  The default value for this setting is "0 seconds".

• "Maximum file retention period"- This is the maximum amount of time in the future that you can set a file retention to be. If you set the file retention date manually the retention time must at most this far if not shorter in the future. If you set the "Default file retention period", it must be at most  the "maximum file retention period" if not shorter. The default value for this setting is "5 years".

• "Default file retention period"- This is the default amount of time in the future that you can set a file retention to be.  This value has to fall within the minimum and maximum file retention period.  Unless this value is set to a value greater than "0 seconds" no files are locked by default.

NOTE : The most common method used to lock files is to set the "Default file retention period" to a value greater than '0 seconds". When this is set (and file retention is turned on) any files created will be locked for this period of time.

retentionAuto

When an administrator is granted the "retentionAuto" authorization they are given the authority to set the Automatic file retention grace period.

This value controls how long after the last access time the ZFS waits to lock the file.  The default setting is "0 seconds".  Until this value is set to a value greater than "0 seconds" no files are automatically locked (using the Default file retention period).  The only method to lock files when this value is left as "0", the default, is to manually lock files.

NOTE: A very important item to understand is that the ZFS locks the file once it has not been updated for this period of time. If you have a process that holds a file open without writing to it, for example an RMAN channel, it may lock the file before it is closed.

Be sure to set the grace period to be longer than the amount of time a process may pause writing to a file.  DO NOT set it too short.  If you wish to lock a file immediately after you have finished writing to it (because you have a long grace period) you can remove the "w" bit from the files using chmod. This will bypass the grace period.

If the share is configured with mandatory retention, the automatic grace period cannot be increased, it can only be lowered.

retentionMandatory

When an administrator is granted the "retentionMandatory" authorization they are given the authority to create a share with a "mandatory (no override)" file retention.  This authorization is not necessary to create a "privileged override" file system.

Be aware that in order to create a file system with "mandatory" file retention the ZFS must be configured with the following settings. The "file retention" service must be running, and the file system needs to be a mirrored configuration

• Remote root user login via the BUI/REST needs to be turned off in the HTTPS service
• Remote root login via SSH needs to be turned off in the SSH service
• NTP sync needs to be configured in the NTP service
• NTP service needs to be on-line.

NOTE : You must ensure that the ZFS administrator is granted these authorizations before attempting to configure file retention. If the administration user is not granted the proper authorization you will permission errors like below.

"You are not authorized to perform this action. If you wish to proceed, contact an administrator to obtain the proper credentials.

Grace period

Default retention

Minimum/Maximum File retention

• If the retention date is set manually on a file, the retention period must fall within the minimum and maximum retention period.

Display current Lock Expirations.

In order to display the lock expiration on Linux the first thing you need to do is to change the share/project setting to "Update access time on read" off . Through the CLI this is a "set atime=false".

Once this settings is made, the client will then display the lock time as the "atime". In my example at the top of the block, you can see by executing "ls -lu" the file lock time is displayed.

NOTE: you can also use the find command to search for files using the "atime" This will allow to find all the locked files.

Manually setting a retention date

Setting File retention lock

1. Touch the file and set the access date. This can be done with

 2. Remove the write bit with chmod guo-2

3.  execute a cmod to make the file read only.

Below is an example where I am taking a file that does not contain retention, and setting the date to January 1, 2025.

First I am going to create a file and touch it setting the atime to a future data.

$echo 'xxxx'> myfile4.txt

$ls -al myfile4.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ touch -a -t "2501011200" myfile3.txt
$ ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jan  1  2025 myfile3.txt
$rm myfile3.txt
$ls -lu myfile3.txt
ls: cannot access myfile3.txt: No such file or directory

You can see that I set the "atime" and it display a future date, but I was still able to delete the file.

Now I am going to move to  remove the write bit before deleting.

$echo 'xxxx'> myfile4.txt

$ls -al myfile4.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ touch -a -t "2501011200" myfile3.txt
$ ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jan  1  2025 myfile3.txt
$chmod ugo-w  myfile3.txt
$rm myfile3.txt
ls: cannot access myfile3.txt: No such file or directory

Still, I am able to delete the file.. Finally I am going to do all three 

$echo 'xxxx'> myfile4.txt

$ls -al myfile4.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jul 15 20:40 myfile3.txt

$ touch -a -t "2501011200" myfile3.txt
$ ls -lu myfile3.txt
-rw-r--r--. 1 nobody oinstall 5 Jan  1  2025 myfile3.txt
$chmod ugo-w  myfile3.txt
$chmod a=r  myfile3.txt
#$rm myfile3.txt
rm: remove write-protected regular file ‘myfile3.txt’? y
rm: cannot remove ‘myfile3.txt’: Operation not permitted

echo 'xxxx'> myfile3.txt
touch -a -t "2501011200" myfile3.txt
chmod ugo-w  myfile3.txt
chmod a=r  myfile3.txt

ZFSSA replication can be used to create locked offsite backups. In this post I will show you how to take advantage of the new "Locked Snapshot" feature of ZFSSA and the ZFS Image in OCI to create an offsite backup strategy to OCI.

Step 1 - Configure remote replication from source ZFSSA to ZFS appliance in OCI. 

Step 2 -  Ensure the source project/share has "Enable retention policy for Scheduled Snapshots" turned on

• Retention Policy MUST be enabled for the share (or inherited from the project).
• The source snapshot MUST be locked when it is created

Step 4 -  Add replication to downstream ZFS in OCI

Step 6 -  Configure the replication schedule

Step 7 -  Validate the replication

 OCI Object Storage provides both lifecycle rules and retention lock.  How to take advantage of both these features isn't always as easy as it looks.

 In this post I will go through an example customer request and how to implement a backup strategy to accomplish the requirements.

This image above gives you an idea of what they are looking to accomplish.

Requirements

• RMAN retention is to keep a 14 day point in time recovery window
• All long term backups beyond 14 days are cataloged as KEEP backups
• All buckets are protected with a retention rule to prevent backups from being deleted before they become obsolete
• Backups are moved to lower tier storage when appropriate to save costs.

Backup strategy

Challenges

Solutions

Backup examples

Mon-Sat 5:30 backup job

dg=$(date +%Y%m%d)
rman <<EOD
run {
ALLOCATE CHANNEL daily1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
ALLOCATE CHANNEL daily2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
backup incremental level 1 database tag="full_backup_${dg}" plus archivelog tag="full_backup_${dg}";
   }
exit
EOD

1) Clean up archive logs first

dg=$(date +%Y%m%d:%H)
rman <<EOD
run {
ALLOCATE CHANNEL daily1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
ALLOCATE CHANNEL daily2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
backup archivelog tag="full_backup_${dg}";
   }
exit
EOD

2a) If this 1st of the month then execute this script to send the full backup to the 13_MONTH bucket

dg=$(date +%Y%m%d)
rman <<EOD
run {
ALLOCATE CHANNEL monthly1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/13_MONTH.ora)';
ALLOCATE CHANNEL monthly2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/13_MONTH.ora)';
backup incremental level 1 database tag="full_backup_${dg}" plus archivelog tag="full_backup_${dg}";
   }
exit
EOD

2b) If this is NOT the 1st of the month execute this script and send the full backup to the 6_WEEK bucket

dg=$(date +%Y%m%d)
rman <<EOD
run {
ALLOCATE CHANNEL weekly1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/6_WEEK.ora)';
ALLOCATE CHANNEL weekly2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/6_WEEK.ora)';
backup incremental level 1 database tag="full_backup_${dg}" plus archivelog tag="full_backup_${dg}";
   }
exit
EOD

dg=$(date --date "-14 days" +%Y%m%d)
rman <<EOD
CHANGE BACKUPSET TAG="full_backup_${dg}" keep until time 'sysdate + 390';
EOD

dg=$(date --date "-14 days" +%Y%m%d)
rman <<EOD
CHANGE BACKUPSET TAG="full_backup_${dg}" keep until time 'sysdate + 28';
EOD

dg=$(date +%Y%m%d)
rman <<EOD
run {
ALLOCATE CHANNEL yearly1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/7_YEAR.ora)';
ALLOCATE CHANNEL yearly2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/7_YEAR.ora)';
backup incremental level 1 database tag="payrun_backup_${dg}" plus archivelog tag="full_backup_${dg}";
   }
exit
EOD

run {
ALLOCATE CHANNEL daily1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
ALLOCATE CHANNEL daily2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/14_DAY.ora)';
ALLOCATE CHANNEL weekly1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/6_WEEK.ora)';
ALLOCATE CHANNEL weekly2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/6_WEEK.ora)';
ALLOCATE CHANNEL monthly1 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/13_MONTH.ora)';
ALLOCATE CHANNEL monthly2 DEVICE TYPE     'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/cloudbackup/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/13_MONTH.ora)';
restore database validate;
    }

channel daily1: starting validation of datafile backup set
channel daily1: reading from backup piece h613o4a4_550_1_1
channel daily1: ORA-19870: error while restoring backup piece h613o4a4_550_1_1
ORA-19507: failed to retrieve sequential file, handle="h613o4a4_550_1_1", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   KBHS-07502: File not found
KBHS-01404: See trace file /u01/app/oracle/diag/rdbms/acmedbp/acmedbp/trace/sbtio_4819_140461854265664.log for det
failover to previous backup

channel daily1: starting validation of datafile backup set
channel daily1: reading from backup piece gq13o3rm_538_1_1
channel daily1: piece handle=gq13o3rm_538_1_1 tag=FULL_073122
channel daily1: restored backup piece 1
channel daily1: validation complete, elapsed time: 00:00:08

 Why do I need both? 

 How to leverage both retention types 

 How do I implement this ? 

 FULLBACKUP - Snapshot retention share 

 DAILYBACKUPS - File Retention share 

DAILY BACKUP SCRIPT 

run {
  ALLOCATE CHANNEL Z1 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';
  ALLOCATE CHANNEL Z2 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';
  ALLOCATE CHANNEL Z3 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';
  ALLOCATE CHANNEL Z4 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';
  ALLOCATE CHANNEL Z5 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';
  ALLOCATE CHANNEL Z6 TYPE DISK  format '/fullbackup/radb/DATA_%N_%f.dbf';

  backup
    section size 32G
    incremental level 1
    for recover of copy with tag 'DEMODBTEST' database FORMAT='/dailybackups/radb/FRA_%d_%T_%U.bkp';
  recover copy of database with tag 'DEMODBTEST' ;
  RELEASE CHANNEL Z1;
  RELEASE CHANNEL Z2;
  RELEASE CHANNEL Z3;
  RELEASE CHANNEL Z4;
  RELEASE CHANNEL Z5;
  RELEASE CHANNEL Z6;
}

 ARCHIVELOG BACKUP SCRIPT 

run {
  ALLOCATE CHANNEL Z1 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';
  ALLOCATE CHANNEL Z2 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';
  ALLOCATE CHANNEL Z3 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';
  ALLOCATE CHANNEL Z4 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';
  ALLOCATE CHANNEL Z5 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';
  ALLOCATE CHANNEL Z6 TYPE DISK  format '/dailybackups/radb/ARCH_%U.bkup';


  backup
    section size 32G
    filesperset 32
    archivelog all;
  RELEASE CHANNEL Z1;
  RELEASE CHANNEL Z2;
  RELEASE CHANNEL Z3;
  RELEASE CHANNEL Z4;
  RELEASE CHANNEL Z5;
  RELEASE CHANNEL Z6;
}

 RESULT: 

 In this post  I will go through how to estimate how much space you need to store backups on the Recovery Appliance to meet your Compliance Window.

This is critical to understand, since compliance protected backups cannot be removed from the RA, and if all space is utilized to meet Compliance Windows, new backups will be refused.

First a bit about Compliance window.

COMPLIANCE WINDOW

Compliance Window is set at the Policy level.  All databases within that policy will inherit the Compliance Window going forward.  Below is some more detail you need to know on Compliance Window.

• The Compliance Window cannot be greater than the Recovery Window Goal
• You cannot set the Policy to "Auto Tune" reserve space when setting a Compliance Window. You must manage the reserve space as you did in the past.
• The Compliance Window can be adjusted up or down once set, but this will not affect any previous backups. Backups previously created observe the Compliance Window in effect when the backup was created.
• The RA does not have to be in Compliance Mode (disabled direct root access) in order to set the a Compliance Window.

Space management for Compliance Window

Reserved Space

Recovery Window Goal

Compliance Window

ESTIMATING COMPLIANCE SPACE

Select dbms_ra.estimate_space ('TIMSP' , numtodsinterval(45,'day')) from dual
       *
ERROR at line 1:
ORA-14551: cannot perform a DML operation inside a query
ORA-06512: at "RASYS.DBMS_RA_MISC", line 5092
ORA-06512: at "RASYS.DBMS_RA", line 1204
ORA-06512: at line 1

 This post is intended to be a Quick Start Guide for those who are new to ZDLRA (RA for short).  I spend part of time working with customers who are new the RA and often the same topics/questions come up.  I wanted to put together a "Quick Start" guide that they can use to learn more about these common topics.

The steps I would follow for anyone new to the RA are.

Documentation

The documentation can be found here. Within the documentation there are several sections that will help you manage the RA.

Get Started 

Install and configure the Recovery Appliance

Learn about the Recovery Appliance.

Administer the Recovery Appliance

Administer the Recovery Appliance

Deployment

Administration

I work with backup and recovery of the Oracle Database, and sometimes this means looking at the Incremental Merge backup strategy.  I know this isn't the best backup/recovery strategy, and below are few posts giving you more detail on the topic.

• Chris Craft's blog post - "Using incremental merge"
• Tim Chien's blog post -"Why CDB(snapshots) are NOT backups"

They have some great points, and I typically don't recommend using incremental merge backups.  The incremental merge backup strategy is almost always paired with snapshots to increase the recovery window.

Below is an image of how these are typically paired with snapshots.

NOTE: When you perform "switch to copy" the database is now accessing datafiles using the backup copy.  This is not supported on Exadata for any storage other than Oracle ZFS.

Given all of that, I got the question "I am using the incremental merge strategy on Oracle an ZFS appliance using snapshots. If I perform a switch-to-copy recovery of one or more datafiles, how do I avoid forcing a new full backup on the next incremental merge backup?".

I thought this was a great question, and I created a test database, and started googlin'.  Below are some of the posts I looked at.

• Oracle 12c and RMAN switch datafile to copy, is it really so easy?
• Oracle docs for "alter database move datafile"

NON-OMF image copy backups

DBCS (now named Oracle Base Database service, but I will call it DBCS throughout this post) in OCI  can help make a DBA's life easier.  When I was testing the new Autonomous Recovery Service for Oracle Database in OCI, I created a LOT of different DBCS systems to test backup and recovery.  Along the way I learned a lot about the workings of DBCS, and I came to appreciate how it makes sense, even if you are a BYOL (bring your own license) customer.

I'm more of a an "old school" DBA, preferring command line, and scripting processes myself.  I am typically not a fan of automation.  When using DBCS I was surprised by all the things it would do for me that I would have to do manually.

Install oracle software and create a database

Having installed oracle software hundreds of times, and having created test databases, I didn't think I would care much about automation that did this for me.

Central Software image management

What I found in OCI, is that you can create your own software images that can be used to ensure each new database environment is consistent.  OCI gives you ability to create your own set of release images (which can include patches).  This ensures each time I create a new DBCS environment, and choose my custom image, it's running the same version in all environments. No more installing base release, then patches, and then then any possible one-off patches.  This makes the installation of the database software much, much easier, and ensures consistency.

Easy Database creation

Flex

Balanced Performance configuration

Standard

 One of the key features of the ZDLRA is the ability to capture changes from the database "real-time" just like a standby database does. In this blog post I am going to demonstrate what is happening during this process so that you can get a better understanding of how it works.

If you look at the GIF above, I will explain what is happening, and show what happens with a demo of the process.

The ZDLRA uses the same process as a standby database.  In fact if you look at the flow of the real-time redo you will notice the redo blocks are sent to BOTH the local redo log files, AND to the staging area on the ZDLRA.  The staging area on the ZDLRA acts just like a standby redo does on a standby database.

As the ZDLRA receives the REDO blocks from the protected database they are validated to ensure that they are valid Oracle Redo block information.  This ensures that a man-in-the-middle attack does not change any of the backup information.  The validation process also assures that if the database is attacked by ransomware (changing blocks), the redo received is not tainted.

The next thing that happens during the process is the logic when a LOG SWITCH occurs.  As we all know, when a log switch occurs on a database instance, the contents of the redo log are written to an archive log.  With real-time redo, this causes the contents of the redo staging area on the ZDLRA (picture a standby redo log) to become a backup set of an archive log.  The RMAN catalog on the ZDLRA is then updated with the internal location of the backup set.

Log switch operation

ZDLRA is configured as a redo destination

List backup of recent archive logs from RMAN catalog

Perform a log switch

List backup of archive logs from RMAN catalog

Restore and Recovery using partial log information

List the active redo log and current SCN

Create a table 

Abort the database

As you probably know, if I shut down the database gracefully, the DB will automatically clean out the redo logs and archive it's contents. Because I want to demonstrate what happens with crash I am going to shut the database down with an ABORT to ensure the log switch doesn't occur.  Then start the database mount so I can look at the current redo log information

Verify that the log switch did not occur

Next I am going to look at the REDO Log information and verify that my table creation (SCN 32908369) is still in the active redo log and did not get archived during the shutdown.

Restore the database

Next I am going to restore the database from backup.

Recover the database

Recover using archive logs on disk

Recover using partial redo log

Open the database and display table contents.

Conclusion : I am hoping this post gives you a better idea of how Real-time redo works on the ZDLRA, and how it handles recovering transactions after a database crash

 Validation of Oracle backups on ZDLRA is often one the most overlooked features of the product. With the rise of ransomware, the question "how to I ensure that I have validated Oracle backups" is critical.

I know there are a lot of vendors out there that provide a great solution for most generic backups. But, as you probably know, Oracle Database backups are different from other system backups and they provide unique challenges which include

• The backup of a large database consists of 100s, if not 1000s of backup pieces. All of which are necessary to successfully restore the database.
• Oracle Database backups won't contain "ransomware signatures" or any easy way of determining if the backup pieces are tainted.
• Oracle Database backups are in a proprietary format that can only be validated by performing a "restore validate" which reads, and validates  the contents of Oracle database backup pieces.

How ZDLRA provides superior validation

Backups land on flash during ingest

Backup pieces are validated

 Backup pieces are stored and virtual full created

Weekly validation for both block content, and restore continuity

Validation during replication

ZDLRA in a Cyber vault

 Autotuned_reserved_space is a new policy setting that got released with 21.1 and you should be using it. When I talk to customers about how to manage databases on a ZDLRA, the biggest confusion comes in when I talk about reserved space.  Reserved space needs to be understood, and properly managed. This new feature in 21.1 allows the ZDLRA to handle the reserved space for you, and I explain how to use it in this blog post.  First let's go through space usage, and reserved space in general.

Space usage on the ZDLRA. 

Recovery Window goal (which drives the space utilization)

The recovery window goal is set at the policy level, and this value (in days) is the number of days that you want to keep as a recovery window for all databases that are a member of this policy.  This will drive the space utilization.

Total space

The ZDLRA comes with all the space pre-allocated.  When you are looking at OEM, or in the SAR report you will see the total space listed. You want to make sure that you have enough space for your database backups and any incoming new backups.

Used Space

Recovery Window Space

Reserved space

Autotuned_reserved_space 

AUTOTUNED_RESERVED_SPACE

MAX_RESERVED_SPACE

AUTOTUNE_SPACE_LIMIT

SUMMARY: