Wednesday, October 12, 2016

Recover Standby Database from Missing Archive Logs

Problem

Archive logs are transfer to standby server but did not apply. When I check alert log following error appear.
Primary Alert Log
ORA-00308: cannot open archived log '/u01/ebs/archive/1_74_920278814.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Standby Alert Log
FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 71-74
Error because Some of the archive logs are delete before apply it on standby database

Solution

1. Login to Primary and Standby db and find the current scn no.
On Primary
[oracle@ebst ebs]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Sep 29 15:08:50 2016

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options  
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-----------
 5.9653E+12

SQL> SELECT to_char(CURRENT_SCN) FROM V$DATABASE;

TO_CHAR(CURRENT_SCN)
----------------------------------------
5965326366951
On Standby
[oracle@demo ebs]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Sep 29 14:31:40 2016

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options  
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-----------
 5.9653E+12

SQL> SELECT to_char(CURRENT_SCN) FROM V$DATABASE;

TO_CHAR(CURRENT_SCN)
----------------------------------------
5965326339755

2. Stop the redo apply and shutdown the Standby database 

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown immediate
ORA-01109: database not open
Database dismounted ORACLE instance shut down.
3. Connect to Primary DB and get the incremental backup from current SCN of standby database where standby is lagging behind
[oracle@ebst 11.1.0]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Tue Oct 4 10:48:56 2016

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: EBS (DBID=2053341531)

RMAN>  backup incremental from scn 5965326339755 database format '/u01/ebs/stb_%U%T';

Starting backup at 04-OCT-16

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=386 device type=DISK
backup will be obsolete on date 11-OCT-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00014 name=/u01/ebs/db/apps_st/data/a_txn_data04.dbf
input datafile file number=00393 name=/u01/ebs/db/apps_st/data/a_txn_ind01.dbf
input datafile file number=00402 name=/u01/ebs/db/apps_st/data/a_txn_data03.dbf
input datafile file number=00406 name=/u01/ebs/db/apps_st/data/a_txn_ind05.dbf
input datafile file number=00353 name=/u01/ebs/db/apps_st/data/system08.dbf
input datafile file number=00405 name=/u01/ebs/db/apps_st/data/a_txn_ind04.dbf
input datafile file number=00379 name=/u01/ebs/db/apps_st/data/undo01.dbf
input datafile file number=00401 name=/u01/ebs/db/apps_st/data/a_txn_data02.dbf
input datafile file number=00013 name=/u01/ebs/db/apps_st/data/system12.dbf
input datafile file number=00403 name=/u01/ebs/db/apps_st/data/a_txn_ind02.dbf
input datafile file number=00352 name=/u01/ebs/db/apps_st/data/system09.dbf
input datafile file number=00394 name=/u01/ebs/db/apps_st/data/a_ref01.dbf
input datafile file number=00400 name=/u01/ebs/db/apps_st/data/a_media01.dbf
input datafile file number=00392 name=/u01/ebs/db/apps_st/data/a_txn_data01.dbf
input datafile file number=00407 name=/u01/ebs/db/apps_st/data/a_ref02.dbf
input datafile file number=00396 name=/u01/ebs/db/apps_st/data/a_summ01.dbf
input datafile file number=00015 name=/u01/ebs/db/apps_st/data/a_txn_ind06.dbf
input datafile file number=00404 name=/u01/ebs/db/apps_st/data/a_txn_ind03.dbf
input datafile file number=00395 name=/u01/ebs/db/apps_st/data/a_int01.dbf
input datafile file number=00001 name=/u01/ebs/db/apps_st/data/system01.dbf
input datafile file number=00002 name=/u01/ebs/db/apps_st/data/system02.dbf
input datafile file number=00003 name=/u01/ebs/db/apps_st/data/system03.dbf
input datafile file number=00004 name=/u01/ebs/db/apps_st/data/system04.dbf
input datafile file number=00005 name=/u01/ebs/db/apps_st/data/system05.dbf
input datafile file number=00398 name=/u01/ebs/db/apps_st/data/a_archive01.dbf
input datafile file number=00295 name=/u01/ebs/db/apps_st/data/system06.dbf
input datafile file number=00351 name=/u01/ebs/db/apps_st/data/system07.dbf
input datafile file number=00354 name=/u01/ebs/db/apps_st/data/system11.dbf
input datafile file number=00288 name=/u01/ebs/db/apps_st/data/system10.dbf
input datafile file number=00011 name=/u01/ebs/db/apps_st/data/sysaux01.dbf
input datafile file number=00008 name=/u01/ebs/db/apps_st/data/a_queue02.dbf
input datafile file number=00012 name=/u01/ebs/db/apps_st/data/apps_ts_tools01.dbf
input datafile file number=00399 name=/u01/ebs/db/apps_st/data/a_queue01.dbf
input datafile file number=00016 name=/u01/ebs/db/apps_st/data/a_ref03.dbf
input datafile file number=00018 name=/u01/ebs/db/apps_st/data/sysaux02.dbf
input datafile file number=00314 name=/u01/ebs/db/apps_st/data/portal01.dbf
input datafile file number=00397 name=/u01/ebs/db/apps_st/data/a_nolog01.dbf
input datafile file number=00006 name=/u01/ebs/db/apps_st/data/ctxd01.dbf
input datafile file number=00010 name=/u01/ebs/db/apps_st/data/olap.dbf
input datafile file number=00017 name=/u01/ebs/db/apps_st/data/a_int02.dbf
input datafile file number=00009 name=/u01/ebs/db/apps_st/data/odm.dbf
input datafile file number=00007 name=/u01/ebs/db/apps_st/data/owad01.dbf
channel ORA_DISK_1: starting piece 1 at 04-OCT-16
channel ORA_DISK_1: finished piece 1 at 04-OCT-16
piece handle=/u01/ebs/stb_2irhgppk_1_120161004 tag=TAG20161004T104923 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:06:25

using channel ORA_DISK_1
backup will be obsolete on date 11-OCT-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 04-OCT-16
channel ORA_DISK_1: finished piece 1 at 04-OCT-16
piece handle=/u01/ebs/stb_2jrhgq5m_1_120161004 tag=TAG20161004T104923 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 04-OCT-16
4. Create standby controlfile and  copy all rman backup files and standby controlfile to standby server
SQL> alter database create standby controlfile as '/u01/ebs/stby.ctl';

Database altered.
5. restore controlfile
RMAN> startup nomount;

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2166536 bytes
Variable Size                436207864 bytes
Database Buffers             616562688 bytes
Redo Buffers                  14000128 bytes

RMAN> restore controlfile from '/u01/ebs/stby.ctl';

Starting restore at 04-OCT-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=383 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=/u01/ebs/db/apps_st/data/cntrl01.dbf
output file name=/u01/ebs/db/apps_st/data/cntrl02.dbf
output file name=/u01/ebs/db/apps_st/data/cntrl03.dbf
Finished restore at 04-OCT-16
6. Mount the DR database and Catalog backups
RMAN> startup nomount;

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2166536 bytes
Variable Size                436207864 bytes
Database Buffers             616562688 bytes
Redo Buffers                  14000128 bytes

RMAN> alter database mount;

using target database control file instead of recovery catalog
database mounted

RMAN> catalog backuppiece '/u01/ebs/stb_2irhgk_1_120161004';

cataloged backup piece
backup piece handle=/u01/ebs/stb_2irhgk_1_120161004 RECID=7 STAMP=924353015

RMAN> catalog backuppiece '/u01/ebs/stb_2jrhgm_1_120161004';

cataloged backup piece
backup piece handle=/u01/ebs/stb_2jrhgm_1_120161004 RECID=8 STAMP=924353024
7. Start the recovery
If there any datafile changes in primary, then first restore particular datafile and do the recovery
RMAN> recover database noredo;

Starting recover at 04-OCT-16
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=349 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /u01/ebs/db/apps_st/data/system01.dbf
destination for restore of datafile 00002: /u01/ebs/db/apps_st/data/system02.dbf
destination for restore of datafile 00003: /u01/ebs/db/apps_st/data/system03.dbf
destination for restore of datafile 00004: /u01/ebs/db/apps_st/data/system04.dbf
destination for restore of datafile 00005: /u01/ebs/db/apps_st/data/system05.dbf
destination for restore of datafile 00006: /u01/ebs/db/apps_st/data/ctxd01.dbf
destination for restore of datafile 00007: /u01/ebs/db/apps_st/data/owad01.dbf
destination for restore of datafile 00008: /u01/ebs/db/apps_st/data/a_queue02.dbf
destination for restore of datafile 00009: /u01/ebs/db/apps_st/data/odm.dbf
destination for restore of datafile 00010: /u01/ebs/db/apps_st/data/olap.dbf
destination for restore of datafile 00011: /u01/ebs/db/apps_st/data/sysaux01.dbf
destination for restore of datafile 00012: /u01/ebs/db/apps_st/data/apps_ts_tools01.dbf
destination for restore of datafile 00013: /u01/ebs/db/apps_st/data/system12.dbf
destination for restore of datafile 00014: /u01/ebs/db/apps_st/data/a_txn_data04.dbf
destination for restore of datafile 00015: /u01/ebs/db/apps_st/data/a_txn_ind06.dbf
destination for restore of datafile 00016: /u01/ebs/db/apps_st/data/a_ref03.dbf
destination for restore of datafile 00017: /u01/ebs/db/apps_st/data/a_int02.dbf
destination for restore of datafile 00018: /u01/ebs/db/apps_st/data/sysaux02.dbf
destination for restore of datafile 00288: /u01/ebs/db/apps_st/data/system10.dbf
destination for restore of datafile 00295: /u01/ebs/db/apps_st/data/system06.dbf
destination for restore of datafile 00314: /u01/ebs/db/apps_st/data/portal01.dbf
destination for restore of datafile 00351: /u01/ebs/db/apps_st/data/system07.dbf
destination for restore of datafile 00352: /u01/ebs/db/apps_st/data/system09.dbf
destination for restore of datafile 00353: /u01/ebs/db/apps_st/data/system08.dbf
destination for restore of datafile 00354: /u01/ebs/db/apps_st/data/system11.dbf
destination for restore of datafile 00379: /u01/ebs/db/apps_st/data/undo01.dbf
destination for restore of datafile 00392: /u01/ebs/db/apps_st/data/a_txn_data01.dbf
destination for restore of datafile 00393: /u01/ebs/db/apps_st/data/a_txn_ind01.dbf
destination for restore of datafile 00394: /u01/ebs/db/apps_st/data/a_ref01.dbf
destination for restore of datafile 00395: /u01/ebs/db/apps_st/data/a_int01.dbf
destination for restore of datafile 00396: /u01/ebs/db/apps_st/data/a_summ01.dbf
destination for restore of datafile 00397: /u01/ebs/db/apps_st/data/a_nolog01.dbf
destination for restore of datafile 00398: /u01/ebs/db/apps_st/data/a_archive01.dbf
destination for restore of datafile 00399: /u01/ebs/db/apps_st/data/a_queue01.dbf
destination for restore of datafile 00400: /u01/ebs/db/apps_st/data/a_media01.dbf
destination for restore of datafile 00401: /u01/ebs/db/apps_st/data/a_txn_data02.dbf
destination for restore of datafile 00402: /u01/ebs/db/apps_st/data/a_txn_data03.dbf
destination for restore of datafile 00403: /u01/ebs/db/apps_st/data/a_txn_ind02.dbf
destination for restore of datafile 00404: /u01/ebs/db/apps_st/data/a_txn_ind03.dbf
destination for restore of datafile 00405: /u01/ebs/db/apps_st/data/a_txn_ind04.dbf
destination for restore of datafile 00406: /u01/ebs/db/apps_st/data/a_txn_ind05.dbf
destination for restore of datafile 00407: /u01/ebs/db/apps_st/data/a_ref02.dbf
channel ORA_DISK_1: reading from backup piece /u01/ebs/stb_2irhgppk_1_120161004
channel ORA_DISK_1: piece handle=/u01/ebs/stb_2irhgppk_1_120161004 tag=TAG20161004T104923
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
Finished recover at 04-OCT-16
7. Start the redo apply
SQL> alter database recover managed standby database disconnect from session;

Database altered.

1 comment:

  1. Cajun vs Titanium Drill Bits - Titanium-Arts
    Cajun titanium pots and pans vs Titanium Drill Bits. Cajun vs. Titanium toaks titanium 750ml pot Drill titanium solvent trap monocore Bits. Cajun vs. Titanium titanium wheels Drill Bits. Cajun vs. Titanium Drill Bits. Cajun titanium mens ring vs.

    ReplyDelete