Oracle ASM Corruption: How To Fix The ASM Disk HEADER_STATUS From FORMER To MEMBER.

Posted by PDSERVICE on Apr 09, 2020 In

ASM Corruption: Case #1: How To Fix The ASM Disk HEADER_STATUS From FORMER or PROVISIONED To MEMBER.

Author: Esteban D. Bernal

This document explains, in detail, the steps required (with an example) to fix the ASM disk header from FORMER to MEMBER under the next scenarios:

A) Diskgroup was dropped by accident using the “SQL> drop <DG name> diskgroup;” statement.

B) Or under strange situations as described in the next bug:

Bug.13331814 ASM DISKS TURNED INTO FORMER WHILE DISKGROUP IS MOUNTED.

Example:

1) You created a diskgroup with the next disk member:

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:16:57 2014

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

SQL> set linesize 100

SQL> column path format a50

SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';

NAME PATH HEADER_STATU

ORCL:SAN_A_DISK_1 PROVISIONED

SQL> create diskgroup TESTDG external redundancy disk 'ORCL:SAN_A_DISK_1';

Diskgroup created.

SQL> select group_number, name, state from v$asm_diskgroup where name like 'TESTDG';

GROUP_NUMBER NAME STATE

1 TESTDG MOUNTED

SQL> set linesize 100

SQL> column path format a50

SQL> select name, path , header_status from v$asm_disk where group_number = 1;

NAME PATH HEADER_STATU

SAN_A_DISK_1 ORCL:SAN_A_DISK_1 MEMBER

2) Then later, the diskgroup is dropped by accident and cannot be mounted again:

SQL> drop diskgroup TESTDG;

Diskgroup dropped.

SQL> alter diskgroup TESTDG mount; alter diskgroup TESTDG mount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "TESTDG" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TESTDG"

3) The original disk member is now shown as a FORMER disk:

SQL> set linesize 100

SQL> column path format a50

SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';

4) Kfed also reports the FORMER status:

[grid@asmlnx1 ~]$ kfed read /dev/oracleasm/disks/SAN_A_DISK_1 | head -25

kfbh.block.obj: 2147483648 ; 0x008: disk=0

kfbh.check: 2862642817 ; 0x00c: 0xaaa07681

kfdhdb.driver.provstr:ORCLDISKSAN_A_DISK_1 ; 0x000: length=20

kfdhdb.driver.reserved[0]: 1598964051 ; 0x008: 0x5f4e4153

kfdhdb.driver.reserved[1]: 1229217601 ; 0x00c: 0x49445f41

kfdhdb.driver.reserved[2]: 828328787 ; 0x010: 0x315f4b53 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000

kfdhdb.dskname: SAN_A_DISK_1 ; 0x028: length=12

kfdhdb.grpname: TESTDG ; 0x048: length=6 kfdhdb.fgname: SAN_A_DISK_1 ; 0x068: length=12

5) The ASM disk header needs to be patched/updated with the HEADER_STATUS = “MEMBER” in order to mount the diskgroup again as follow:

5.1) Backup the first 10 MB of the associated disks using the dd command:

[grid@asmlnx1 ~]$ mkdir /tmp/asm

[grid@asmlnx1~]$

of=/tmp/SAN_A_DISK_1.dump bs=1048576 count=50 50+0 records in

50+0 records out

52428800 bytes (52 MB) copied, 2.0715 seconds, 25.3 MB/s 5.2) Generate the template patch from the FORMER disk:

/tmp/asm/patch_SAN_A_DISK_1.txt

[grid@asmlnx1~]$ ls -l /tmp/asm/patch_SAN_A_DISK_1.txt

-rw-r--r-- 1 grid oinstall 6594 Apr 10 16:48 /tmp/asm/patch_SAN_A_DISK_1.txt

5.3) Edit the file (/tmp/asm/patch_SAN_A_DISK_1.txt) and change the code # from 4 to 3 and the status from FORMER to MEMBER in the next line (make sure you save the changes):

Original line:

New line:

Note: #4 stands for “FORMER” & #3 stands for “MEMBER”

5.4) Verify the new values in the patch file:

[grid@asmlnx1~]$ grep kfdhdb.hdrsts /tmp/asm/patch_SAN_A_DISK_1.txt

5.5) Now you can patch the FORMER disk using the “/tmp/asm/patch_SAN_A_DISK_1.txt” patch file:

text=/tmp/asm/patch_SAN_A_DISK_1.txt

5.6) Verify the new values in the real disk (ASM disk header was fixed with the HEADER_STATUS= “MEMBER”):

[grid@asmlnx1~]$ kfed read /dev/oracleasm/disks/SAN_A_DISK_1 | grep kfdhdb.hdrsts

6) Now you can mount the diskgroup:

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:31:25 2014

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

SQL> alter diskgroup TESTDG mount;

Diskgroup altered.

SQL> set linesize 100

SQL> column path format a50

SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';

SQL> alter diskgroup TESTDG mount;

Diskgroup altered.

Annex: Syntax to create the ASMLIB disk:

[root@asmlnx1 ~]# /etc/init.d/oracleasm querydisk -p SAN_A_DISK_1 Disk "SAN_A_DISK_1" is a valid ASM disk

/dev/asmdisk1_4k_udev_p1: LABEL="SAN_A_DISK_1" TYPE="oracleasm"

[root@asmlnx1 ~]# /etc/init.d/oracleasm createdisk SAN_A_DISK_1

/dev/asmdisk1_4k_udev_p1

Marking disk "SAN_A_DISK_1" as an ASM disk: [ OK ]

You are here

Oracle ASM Corruption: How To Fix The ASM Disk HEADER_STATUS From FORMER To MEMBER.

Oracle ASM Corruption: How To Fix The ASM Disk HEADER_STATUS From FORMER To MEMBER.

ASM Corruption: Case #1: How To Fix The ASM Disk HEADER_STATUS From FORMER or PROVISIONED To MEMBER.