Recover Standby Database from Missing Archive Logs

Problem

Archive logs are transfer to standby server but did not apply. When I check alert log following error appear.

Primary Alert Log
ORA-00308: cannot open archived log '/u01/ebs/archive/1_74_920278814.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory

Standby Alert Log
FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 71-74

Error because Some of the archive logs are delete before apply it on standby database

Solution

1. Login to Primary and Standby db and find the current scn no.

On Primary
[oracle@ebst ebs]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Sep 29 15:08:50 2016

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options  
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-----------
 5.9653E+12

SQL> SELECT to_char(CURRENT_SCN) FROM V$DATABASE;

TO_CHAR(CURRENT_SCN)
----------------------------------------
5965326366951

On Standby
[oracle@demo ebs]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Sep 29 14:31:40 2016

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options  
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-----------
 5.9653E+12

SQL> SELECT to_char(CURRENT_SCN) FROM V$DATABASE;

TO_CHAR(CURRENT_SCN)
----------------------------------------
5965326339755

2. Stop the redo apply and shutdown the Standby database 

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown immediate
ORA-01109: database not open
Database dismounted ORACLE instance shut down.

3. Connect to Primary DB and get the incremental backup from current SCN of standby database where standby is lagging behind

[oracle@ebst 11.1.0]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Tue Oct 4 10:48:56 2016

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: EBS (DBID=2053341531)

RMAN>  backup incremental from scn 5965326339755 database format '/u01/ebs/stb_%U%T';

Starting backup at 04-OCT-16

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=386 device type=DISK
backup will be obsolete on date 11-OCT-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00014 name=/u01/ebs/db/apps_st/data/a_txn_data04.dbf
input datafile file number=00393 name=/u01/ebs/db/apps_st/data/a_txn_ind01.dbf
input datafile file number=00402 name=/u01/ebs/db/apps_st/data/a_txn_data03.dbf
input datafile file number=00406 name=/u01/ebs/db/apps_st/data/a_txn_ind05.dbf
input datafile file number=00353 name=/u01/ebs/db/apps_st/data/system08.dbf
input datafile file number=00405 name=/u01/ebs/db/apps_st/data/a_txn_ind04.dbf
input datafile file number=00379 name=/u01/ebs/db/apps_st/data/undo01.dbf
input datafile file number=00401 name=/u01/ebs/db/apps_st/data/a_txn_data02.dbf
input datafile file number=00013 name=/u01/ebs/db/apps_st/data/system12.dbf
input datafile file number=00403 name=/u01/ebs/db/apps_st/data/a_txn_ind02.dbf
input datafile file number=00352 name=/u01/ebs/db/apps_st/data/system09.dbf
input datafile file number=00394 name=/u01/ebs/db/apps_st/data/a_ref01.dbf
input datafile file number=00400 name=/u01/ebs/db/apps_st/data/a_media01.dbf
input datafile file number=00392 name=/u01/ebs/db/apps_st/data/a_txn_data01.dbf
input datafile file number=00407 name=/u01/ebs/db/apps_st/data/a_ref02.dbf
input datafile file number=00396 name=/u01/ebs/db/apps_st/data/a_summ01.dbf
input datafile file number=00015 name=/u01/ebs/db/apps_st/data/a_txn_ind06.dbf
input datafile file number=00404 name=/u01/ebs/db/apps_st/data/a_txn_ind03.dbf
input datafile file number=00395 name=/u01/ebs/db/apps_st/data/a_int01.dbf
input datafile file number=00001 name=/u01/ebs/db/apps_st/data/system01.dbf
input datafile file number=00002 name=/u01/ebs/db/apps_st/data/system02.dbf
input datafile file number=00003 name=/u01/ebs/db/apps_st/data/system03.dbf
input datafile file number=00004 name=/u01/ebs/db/apps_st/data/system04.dbf
input datafile file number=00005 name=/u01/ebs/db/apps_st/data/system05.dbf
input datafile file number=00398 name=/u01/ebs/db/apps_st/data/a_archive01.dbf
input datafile file number=00295 name=/u01/ebs/db/apps_st/data/system06.dbf
input datafile file number=00351 name=/u01/ebs/db/apps_st/data/system07.dbf
input datafile file number=00354 name=/u01/ebs/db/apps_st/data/system11.dbf
input datafile file number=00288 name=/u01/ebs/db/apps_st/data/system10.dbf
input datafile file number=00011 name=/u01/ebs/db/apps_st/data/sysaux01.dbf
input datafile file number=00008 name=/u01/ebs/db/apps_st/data/a_queue02.dbf
input datafile file number=00012 name=/u01/ebs/db/apps_st/data/apps_ts_tools01.dbf
input datafile file number=00399 name=/u01/ebs/db/apps_st/data/a_queue01.dbf
input datafile file number=00016 name=/u01/ebs/db/apps_st/data/a_ref03.dbf
input datafile file number=00018 name=/u01/ebs/db/apps_st/data/sysaux02.dbf
input datafile file number=00314 name=/u01/ebs/db/apps_st/data/portal01.dbf
input datafile file number=00397 name=/u01/ebs/db/apps_st/data/a_nolog01.dbf
input datafile file number=00006 name=/u01/ebs/db/apps_st/data/ctxd01.dbf
input datafile file number=00010 name=/u01/ebs/db/apps_st/data/olap.dbf
input datafile file number=00017 name=/u01/ebs/db/apps_st/data/a_int02.dbf
input datafile file number=00009 name=/u01/ebs/db/apps_st/data/odm.dbf
input datafile file number=00007 name=/u01/ebs/db/apps_st/data/owad01.dbf
channel ORA_DISK_1: starting piece 1 at 04-OCT-16
channel ORA_DISK_1: finished piece 1 at 04-OCT-16
piece handle=/u01/ebs/stb_2irhgppk_1_120161004 tag=TAG20161004T104923 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:06:25

using channel ORA_DISK_1
backup will be obsolete on date 11-OCT-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 04-OCT-16
channel ORA_DISK_1: finished piece 1 at 04-OCT-16
piece handle=/u01/ebs/stb_2jrhgq5m_1_120161004 tag=TAG20161004T104923 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 04-OCT-16

4. Create standby controlfile and  copy all rman backup files and standby controlfile to standby server

SQL> alter database create standby controlfile as '/u01/ebs/stby.ctl';

Database altered.

5. restore controlfile

RMAN> startup nomount;

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2166536 bytes
Variable Size                436207864 bytes
Database Buffers             616562688 bytes
Redo Buffers                  14000128 bytes

RMAN> restore controlfile from '/u01/ebs/stby.ctl';

Starting restore at 04-OCT-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=383 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=/u01/ebs/db/apps_st/data/cntrl01.dbf
output file name=/u01/ebs/db/apps_st/data/cntrl02.dbf
output file name=/u01/ebs/db/apps_st/data/cntrl03.dbf
Finished restore at 04-OCT-16

6. Mount the DR database and Catalog backups

RMAN> startup nomount;

Oracle instance started

Total System Global Area    1068937216 bytes

Fixed Size                     2166536 bytes
Variable Size                436207864 bytes
Database Buffers             616562688 bytes
Redo Buffers                  14000128 bytes

RMAN> alter database mount;

using target database control file instead of recovery catalog
database mounted

RMAN> catalog backuppiece '/u01/ebs/stb_2irhgk_1_120161004';

cataloged backup piece
backup piece handle=/u01/ebs/stb_2irhgk_1_120161004 RECID=7 STAMP=924353015

RMAN> catalog backuppiece '/u01/ebs/stb_2jrhgm_1_120161004';

cataloged backup piece
backup piece handle=/u01/ebs/stb_2jrhgm_1_120161004 RECID=8 STAMP=924353024

7. Start the recovery
If there any datafile changes in primary, then first restore particular datafile and do the recovery

RMAN> recover database noredo;

Starting recover at 04-OCT-16
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=349 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /u01/ebs/db/apps_st/data/system01.dbf
destination for restore of datafile 00002: /u01/ebs/db/apps_st/data/system02.dbf
destination for restore of datafile 00003: /u01/ebs/db/apps_st/data/system03.dbf
destination for restore of datafile 00004: /u01/ebs/db/apps_st/data/system04.dbf
destination for restore of datafile 00005: /u01/ebs/db/apps_st/data/system05.dbf
destination for restore of datafile 00006: /u01/ebs/db/apps_st/data/ctxd01.dbf
destination for restore of datafile 00007: /u01/ebs/db/apps_st/data/owad01.dbf
destination for restore of datafile 00008: /u01/ebs/db/apps_st/data/a_queue02.dbf
destination for restore of datafile 00009: /u01/ebs/db/apps_st/data/odm.dbf
destination for restore of datafile 00010: /u01/ebs/db/apps_st/data/olap.dbf
destination for restore of datafile 00011: /u01/ebs/db/apps_st/data/sysaux01.dbf
destination for restore of datafile 00012: /u01/ebs/db/apps_st/data/apps_ts_tools01.dbf
destination for restore of datafile 00013: /u01/ebs/db/apps_st/data/system12.dbf
destination for restore of datafile 00014: /u01/ebs/db/apps_st/data/a_txn_data04.dbf
destination for restore of datafile 00015: /u01/ebs/db/apps_st/data/a_txn_ind06.dbf
destination for restore of datafile 00016: /u01/ebs/db/apps_st/data/a_ref03.dbf
destination for restore of datafile 00017: /u01/ebs/db/apps_st/data/a_int02.dbf
destination for restore of datafile 00018: /u01/ebs/db/apps_st/data/sysaux02.dbf
destination for restore of datafile 00288: /u01/ebs/db/apps_st/data/system10.dbf
destination for restore of datafile 00295: /u01/ebs/db/apps_st/data/system06.dbf
destination for restore of datafile 00314: /u01/ebs/db/apps_st/data/portal01.dbf
destination for restore of datafile 00351: /u01/ebs/db/apps_st/data/system07.dbf
destination for restore of datafile 00352: /u01/ebs/db/apps_st/data/system09.dbf
destination for restore of datafile 00353: /u01/ebs/db/apps_st/data/system08.dbf
destination for restore of datafile 00354: /u01/ebs/db/apps_st/data/system11.dbf
destination for restore of datafile 00379: /u01/ebs/db/apps_st/data/undo01.dbf
destination for restore of datafile 00392: /u01/ebs/db/apps_st/data/a_txn_data01.dbf
destination for restore of datafile 00393: /u01/ebs/db/apps_st/data/a_txn_ind01.dbf
destination for restore of datafile 00394: /u01/ebs/db/apps_st/data/a_ref01.dbf
destination for restore of datafile 00395: /u01/ebs/db/apps_st/data/a_int01.dbf
destination for restore of datafile 00396: /u01/ebs/db/apps_st/data/a_summ01.dbf
destination for restore of datafile 00397: /u01/ebs/db/apps_st/data/a_nolog01.dbf
destination for restore of datafile 00398: /u01/ebs/db/apps_st/data/a_archive01.dbf
destination for restore of datafile 00399: /u01/ebs/db/apps_st/data/a_queue01.dbf
destination for restore of datafile 00400: /u01/ebs/db/apps_st/data/a_media01.dbf
destination for restore of datafile 00401: /u01/ebs/db/apps_st/data/a_txn_data02.dbf
destination for restore of datafile 00402: /u01/ebs/db/apps_st/data/a_txn_data03.dbf
destination for restore of datafile 00403: /u01/ebs/db/apps_st/data/a_txn_ind02.dbf
destination for restore of datafile 00404: /u01/ebs/db/apps_st/data/a_txn_ind03.dbf
destination for restore of datafile 00405: /u01/ebs/db/apps_st/data/a_txn_ind04.dbf
destination for restore of datafile 00406: /u01/ebs/db/apps_st/data/a_txn_ind05.dbf
destination for restore of datafile 00407: /u01/ebs/db/apps_st/data/a_ref02.dbf
channel ORA_DISK_1: reading from backup piece /u01/ebs/stb_2irhgppk_1_120161004
channel ORA_DISK_1: piece handle=/u01/ebs/stb_2irhgppk_1_120161004 tag=TAG20161004T104923
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
Finished recover at 04-OCT-16

7. Start the redo apply

SQL> alter database recover managed standby database disconnect from session;

Database altered.

APP-PO-14142: get_po_parameters-10: ORA-01422 When Open Purchasing Form

Problem

Following Error Occur when open a purchasing form

Solution

1. Connect apps user and run below query.

select object_name,policy_name, package,function,enable
from sys.all_policies
where object_name = upper('PO_DOCUMENT_TYPES_B')
union
select object_name,policy_name, package,function,enable
from sys.all_policies
where object_name = upper('PO_DOCUMENT_TYPES_TL')
union
select object_name,policy_name, package,function,enable
from sys.all_policies
where object_name = upper('PO_SYSTEM_PARAMETERS')
union
select object_name,policy_name, package,function,enable
from sys.all_policies
where object_name = upper('FINANCIALS_SYSTEM_PARAMETERS');

If query does not return above four policies, then add those

2. Run below script via apps user

SQL>revoke EXEMPT ACCESS POLICY from APPS;

3. Verify the issue by running following query.

select count(*) from po_document_types;

If above query return 0 rows, then issue will be resolved

Reference : 
Choosing Approval Assignments Form From Menu - Nothing Happens - Form Does Not Open (Doc ID 1063223.1)

R12 Error : ORA-1691: unable to extend lobsegment

Problem

Following error occurred while attached and save the attachment on oracle forms in R12.
ORA-1691: unable to extend lobsegment APPLSYS.SYS_LOB0000033088C00004$$ by 2362205 in tablespace APPLSYSD

Solution

Check if there enough free space in tablespaces. if so please follow the steps

1. Verify NEXT_EXTENT & MAX_EXTENTS value of the segment highlighted in error

SQL> select BYTES,EXTENTS,NEXT_EXTENT,MAX_EXTENTS from dba_segments where segment_name='SYS_LOB0000033088C00004$$';

     BYTES    EXTENTS NEXT_EXTENT MAX_EXTENTS
---------- ---------- ----------- -----------
3.8703E+10         33  1.9351E+10         505

NEXT_EXTENT found to be very huge (19 GB) which is the root cause of this issue

2.Reduce segment NEXT_EXTENT size from 19 GB to 1 MB

SQL> select owner, table_name, column_name, SEGMENT_NAME  from dba_lobs where segment_name='SYS_LOB0000033088C00004$$';

OWNER          TABLE_NAME       COLUMN_NAME       SEGMENT_NAME
-----------------------------------------------------------------------------
APPLSYS        FND_LOBS          FILE_DATA      SYS_LOB0000033088C00004$$

SQL> alter table applsys.FND_LOBS modify lob ("FILE_DATA") (STORAGE (NEXT 1048576));

Table altered.

SQL> select BYTES,EXTENTS,NEXT_EXTENT,MAX_EXTENTS from dba_segments where segment_name='SYS_LOB0000033088C00004$$';

     BYTES    EXTENTS NEXT_EXTENT MAX_EXTENTS
---------- ---------- ----------- -----------
3.8703E+10         33     1048576         505

NEXT_EXTENT is greater than 1 GB & Contiguous free space of the datafile which segment is located may not accommodate next few extents which will lead to allocation of next extent in new datafile. Please verify if other datafiles in the same tablespace has sufficient contiguous space available . If not then add a datafile with size equal to or greater than NEXT_EXTENT else reduce NEXT_EXTENT size

Reference : 
Ora-1691: Unable To Extend Lobsegment Applsys.Sys_lob0000033489c00004$$ By 92170 (Doc ID 378377.1)

How To Re-attach Oracle Homes To The Central Inventory in R12.2

Problem 

Accidentally deleted oraInventory directory. 

Solution

In 12.2, application tier has 4 homes and a DB home. have to re‐attach all oracle homes. each oracle_home_name must be unique. 

For Application Tier (need to run on each file system) 

/u01/VIS/fs1/FMW_Home/oracle_common/oui/bin/runInstaller 
‐silent ‐attachHome ‐invPtrLoc /etc/oraInst.loc 
ORACLE_HOME="/u01/VIS/fs1/FMW_Home/oracle_common" 
ORACLE_HOME_NAME= "OH481075858" CLUSTER_NODES="{}" 

/u01/VIS/fs1/FMW_Home/webtier/oui/bin/runInstaller 
‐silent ‐attachHome ‐invPtrLoc /etc/oraInst.loc 
ORACLE_HOME="/u01/VIS/fs1/FMW_Home/webtier"
ORACLE_HOME_NAME="OH657355352" CLUSTER_NODES="{}" 

/u01/VIS/fs1/EBSapps/10.1.2/oui/bin/runInstaller 
‐silent ‐attachHome ‐invPtrLoc /etc/oraInst.loc 
ORACLE_HOME="/u01/VIS/fs1/EBSapps/10.1.2"
ORACLE_HOME_NAME="VIS_TOOLS__u01_VIS_fs1_EBSapps_10_1_2" 

/u01/VIS/fs1/FMW_Home/Oracle_EBS‐app1/oui/bin/runInstaller 
‐silent ‐attachHome ‐invPtrLoc /etc/oraInst.loc 
ORACLE_HOME="/u01/VIS/fs1/FMW_Home/Oracle_EBS‐app1" 
ORACLE_HOME_NAME="u01_VIS_fs1_Oracle_EBS_app" 

For DB home 

/u01/VIS/db/tech_st/11.2.0/oui/bin/runInstaller 
‐silent ‐attachHome ‐invPtrLoc /etc/oraInst.loc
ORACLE_HOME="/u01/VIS/db/tech_st/11.2.0"
ORACLE_HOME_NAME="OraDb11g_home1" 

After successfully attach above oracle homes you can find the entries from oraInventory/ContentsXML/inventory.xml file

fs1

<HOME NAME="OH481075858" LOC="/u01/VIS/fs1/FMW_Home/oracle_common" TYPE="O" IDX="5">

<HOME NAME="OH657355352" LOC="/u01/VIS/fs1/FMW_Home/webtier" TYPE="O" IDX="6">

<HOME NAME="VIS_TOOLS__u01_VIS_fs1_EBSapps_10_1_2" LOC="/u01/VIS/fs1/EBSapps/10.1.2" TYPE="O" IDX="7"/>

<HOME NAME="u01_VIS_fs1_Oracle_EBS_app" LOC="/u01/VIS/fs1/FMW_Home/Oracle_EBS‐app1" TYPE="O" IDX="8"/>

fs2

<HOME NAME="OH781075858" LOC="/u01/VIS/fs2/FMW_Home/oracle_common" TYPE="O" IDX="5">

<HOME NAME="OH957355352" LOC="/u01/VIS/fs2/FMW_Home/webtier" TYPE="O" IDX="6">

<HOME NAME="VIS_TOOLS__u01_VIS_fs2_EBSapps_10_1_2" LOC="/u01/VIS/fs2/EBSapps/10.1.2" TYPE="O" IDX="7"/>

<HOME NAME="u01_VIS_fs2_Oracle_EBS_app" LOC="/u01/VIS/fs2/FMW_Home/Oracle_EBS‐app1" TYPE="O" IDX="8"/> 

Database

<HOME NAME="OraDb11g_home1" LOC="/u01/VIS/db/tech_st/11.2.0"  TYPE="O" IDX="1"/>

R12 Error : APP-FND-00500: AFPPRN received a return code of failure from routine FDUPRN

Problem

Concurrent program completed with following warning

APP-FND-00500: AFPPRN received a return code of failure from routine FDUPRN. Cannot write to the communication channel

Cause: AFPPRN received a return code of failure from the OSD routine FDUPRN. Cannot write to the communication channel.

Solution

1. Log into Applications with the System Administrator responsibility.

2. Navigate to Concurrent -> program -> define

3. Query the concurrent program. 

4. In the Output options are listed, clear the check box before the word “Print” and Leave the “Save” option selected to avoid the warning when running the program because it was unable to delete the output file which was never created.

4. Save the Changes.

Re-create lost Context file in R12

We can retrieve both application and Database tier context file by running adclonectx.pl script. Execute the following command on the applications tier.

perl <COMMON_TOP>/clone/bin/adclonectx.pl retrieve

The above command can be used only when the <INST_TOP> is still intact. In case the <INST_TOP> has also been lost accidentally, the context file may be retrieved as follows

perl <ORACLE_HOME>/appsutil/clone/bin/adclonectx.pl retrieve

On being prompted for the context file to be retrieved, select the option of retrieving the context file which you need (APPL_TOP or Database) to retrieve it to the default location specified by the script.


 The adbldxml.pl utility can be used to generate the context file on the database tier only. It is commonly used when migrating AutoConfig to a database tier not created using Rapid Install or when upgrading a database tier from Applications 11i to EBusiness Suite Release 12. This utility is not supported on the applications tier.

perl <ORACLE_HOME>/appsutil/bin adbldxml.pl appsuser=apps

Steps to change Apps password for EBS 12.2

Follow all the below steps to change apps password in 12.2 instance. Passwords for all the 3 schemas APPS, APPLSYS and APPS_NE will be changed together while changing APPLSYS password

1. Shut down the application tier services

2. Change the APPLSYS password

FNDCPASS apps/<apps_pw> 0 Y SYSTEM/<system_pw> SYSTEM APPLSYS <new_password>

3.Run Autoconfig on Application Tier with newly changed password

4. Start the Adminserver

cd $ADMIN_SCRIPTS_HOME
./adadminsrvctl.sh start

5. Change the "apps" password in WLS Datasource as follows

• Log in to WLS Administration Console.

•  Click Lock & Edit in Change Center

•  In the Domain Structure tree, expand Services, then select Data Sources

•  On the "Summary of JDBC Data Sources" page, select EBSDataSource

•  On the "Settings for EBSDataSource" page, select the Connection Pool tab.

• Enter the new password in the "Password" and "Confirm Password" field
•  Click Save
                                                                                                               
•  Click Activate Changes in Change Center

Reference
Steps to change the APPS, APPLSYS, and APPS_NE password using FNDCPASS or AFPASSWD for EBS 12.2 (Doc ID 1674462.1)