TROUBLESHOOTING - Oracle ASM disk not found/visible/discovered issues

Posted by PDSERVICE on Apr 09, 2020 In

APPLIES TO:

Oracle Database Cloud Schema Service - Version N/A and later

Oracle Database Exadata Cloud Machine - Version N/A and later

Oracle Database Exadata Express Cloud Service - Version N/A and later

Oracle Cloud Infrastructure - Database Service - Version N/A and later

Oracle Database Cloud Exadata Service - Version N/A and later

Information in this document applies to any platform.

PURPOSE

This note will assist in troubleshooting disk not found / visible / discovered issues with ASM

Typically this means that the disk in question is not found in the v$asm_disk view

Common errors indicating that a disk is missing/not found are :

ORA-15063: ASM discovered an insufficient number of disks for diskgroup s%

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "%" is missing

TROUBLESHOOTING STEPS

If an existing diskgroup is suddenly missing a disk ...

Determine what disk is missing

If on non-RAC ... or ALL RAC nodes exhibit these errors

a) Locate and open the ASM alert log (this may require multiple logs for RAC)

b) Locate the last successfull mount of the diskgroup (this will show a list of disks)

c) Locate each successful ALTER DISKGROUP ... ADD DISK since that time

d) Combine the lists of disks found in b) and c) above

e) Compare this list against those shown in V$ASM_DISK and determine the missing disk(s)

If on RAC and at least one node can mount the diskgroup

Compare the V$ASM_DISK entries on the node that can mount to those that cannot this will show which disk(s) is missing

The following is a set of steps that will assist in resolving disk not found issues

1) ASM_DISKSTRING is not set correctly

Examine the ASM_DISKSTRING setting in the parameter file or via SHOW PARAMETER

If ASM_DISKSTRING is NOT SET ... then the following default is used

Default ASM_DISKSTRING per OS

Operating System Default Search String

=======================================

Solaris (32/64 bit) /dev/rdsk/*

Windows NT/XP \\.\orcldisk*

Linux (32/64 bit) /dev/raw/*

LINUX (ASMLIB) ORCL:*

LINUX (ASMLIB) /dev/oracleasm/disks/* ( as a workaround )

HPUX /dev/rdsk/*

HP-UX(Tru 64) /dev/rdisk/*

AIX /dev/rhdisk*

IF ASM_DISKSTRING is SET ... then verify that the setting includes the disks that are needed to be seen by ASM

2) Operating system drive ownership

Make sure that the disk is owned by the OS user who installed the ASM Oracle Home ... and that the disk is mounted correctly (with the correct owner)

3) Operating system drive permissions

Make sure that the permissions are set correctly at the disk level ... 660 is normal ... but if there are problems use 777 as a test

4) RAC is being used

If RAC is in use ... then ALL disks need to be visible on all nodes where ASM is / will be running ... before an attempt is made to add the disks to a diskgroup

5) Use OS utilities to determine which disk cannot be found

TRUSSing or STRACEing the RBAL process while selecting * from v$asm_disk can often show errors in the path of the command

EXAMPLE:

=========

SESSION #1

strace -f -o /tmp/rbal.trc -p <OS pid of RBAL process>

<OR>

truss -ef -o /tmp/rbal.out -p <OS pid for RBAL process>

SESSION #2

select * from v$asm_disk

SESSION #3

<CTRL-C>

Examine the rbal.out for errors: For example,

1147090: 1871929: chdir("dev/") = 0

1147090: 1871929: statx("rhdisk8, ", 0x0FFFFFFFFFFFAA80, 176, 010) Err#2 ENOENT

<< This says that rhdisk8 cannot be found >>

NOTE ... If a crash occured during an Add or Drop of a disk ... then the disk(s) in question may still be part of the diskgroup ... so all of the steps above need to take into consideration this (these) disks

PORT SPECIFIC ISSUES

1) HEWLETT PACKARD (HP)

If this is problem is occuring on HP (RISC or Itanium) ... and all of the above are not helping ... and the customer is using HP Logical Volume Manager (LVM) then

Note.433770.1 Cannot Discover Disks in ASM After Upgrade on 10.2.0.3 on HP-UX Itanium

Note.434500.1 When Starting A Database With The 10.2.0.3 Executables, Error "ORA-15059 invalid device type for ASM disk"

2) IBM AIX

Note.353761.1 Assigning a Physical Volume ID (PVID) To An Existing ASM Disk Corrupts the ASM Disk Header

NOTE:1174604.1 - ASM Is Not Detecting EMC PowerPath Raw Devices Or Regular Raw Devices On AIX

3) SOLARIS

Note.368840.1 ASM does not discover disk on Solaris platform

4) LINUX

Note.457369.1 RASM is Unable to Detect ASMLIB Disks/Devices.

You are here

TROUBLESHOOTING - Oracle ASM disk not found/visible/discovered issues

TROUBLESHOOTING - Oracle ASM disk not found/visible/discovered issues