Oracle 19c ASM Storage: Block Corruption

 Oracle 19c ASM Storage: Block Corruption

Special errors, step-by-step diagnosis and fix, with live examples

Environment used in all examples: Oracle 19c EE, database PRODDB, ASM instance +ASM1, diskgroups +DATA and +FRA, host dbprod01, OS Oracle Linux with multipath. Sample outputs are illustrative of what each error looks like; replace file numbers, block numbers and disk names with your own values.

Rule for every case: identify the layer first (database, ASM, OS/storage), then fix at that layer. Never repair at the database layer if the storage layer is still failing.

1. Quick triage table

Error	Layer	Meaning / go to
ORA-01578 + ORA-01110	Database	Physical block corruption in a datafile. Case A
ORA-15130 / ORA-15075 / ORA-27072 in ASM log; disk OFFLINE	ASM / storage	Disk dropped or offlined after I/O error. Case B
ORA-15032 + ORA-15040 + ORA-15042	ASM	Diskgroup will not mount, disk missing. Case C
ORA-15196, ORA-15036	ASM	Invalid ASM block or disk header. Case D
ORA-26040	Database	NOLOGGING block, logically corrupt. Case E
ORA-00600 [kcbz_check_objd_typ]	Database	Block/object mismatch, in-memory or lost write. Case F
ORA-15080 / ORA-15081 / ORA-27072	OS / storage	I/O failed or could not be submitted. Case G

2. Common first steps (always run)

Step 1: Read both alert logs

# as oracle (database)

adrci> set home diag/rdbms/proddb/PRODDB

adrci> show alert -tail 200

 

# as grid (ASM)

adrci> set home diag/asm/+asm/+ASM1

adrci> show alert -tail 200

 

grep -E "ORA-01578|ORA-01110|ORA-26040|ORA-00600|ORA-15[0-9]{3}|ORA-27[0-9]{3}" alert_*.log | tail -50

 

Step 2: ASM layer health (as grid, sqlplus / as sysasm)

SELECT name,state,type,total_mb,free_mb,offline_disks FROM v$asm_diskgroup;

 

SELECT group_number gn,disk_number dn,name,path,mode_status,state,header_status,

       read_errs,write_errs FROM v$asm_disk ORDER BY 1,2;

 

SELECT * FROM v$asm_operation;

 

Healthy output: state MOUNTED, offline_disks 0, mode_status ONLINE, header_status MEMBER, read_errs and write_errs 0, no rows in v$asm_operation. Anything else points to Case B, C, D or G.

Step 3: Database layer scope

SELECT file#,block#,blocks,corruption_type FROM v$database_block_corruption;

 

rman target /

RMAN> VALIDATE CHECK LOGICAL DATABASE;

 

Step 4: OS and storage layer

dmesg -T | grep -iE "i/o error|scsi|multipath|blk_update|reset"

tail -200 /var/log/messages

multipath -ll

ls -l /dev/mapper/ /dev/oracleasm/disks/    # owner must be grid:asmadmin, mode 660

 

Case A: ORA-01578 / ORA-01110 physical block corruption

Live example: error as seen

ORA-01578: ORACLE data block corrupted (file # 7, block # 128945)

ORA-01110: data file 7: '+DATA/PRODDB/DATAFILE/users.259.1098123456'

 

Step A1: Confirm and list all corrupt blocks

SQL> SELECT file#,block#,blocks,corruption_type FROM v$database_block_corruption;

 

FILE#  BLOCK#   BLOCKS CORRUPTION_TYPE

----- -------- ------- ---------------

    7   128945       1 CORRUPT

 

Step A2: Find the affected object

SELECT owner,segment_name,segment_type,tablespace_name

FROM dba_extents

WHERE file_id=7 AND 128945 BETWEEN block_id AND block_id+blocks-1;

 

OWNER   SEGMENT_NAME   SEGMENT_TYPE   TABLESPACE_NAME

------- -------------- -------------- ---------------

APPS    XX_GL_LINES    TABLE          USERS

 

No rows returned means the block is free space or the file header; RMAN repair still applies.

Step A3: Rule out storage cause first

Run Step 2 and Step 4 above. If read_errs is non-zero or dmesg shows I/O errors, fix Case B or G first, otherwise the block will be corrupted again.

Step A4: Repair with RMAN block media recovery (best method)

RMAN> RECOVER DATAFILE 7 BLOCK 128945;

 

-- or repair everything listed in v$database_block_corruption:

RMAN> RECOVER CORRUPTION LIST;

 

Starting recover at 16-OCT-26

channel ORA_DISK_1: restoring block(s) from datafile backup set

starting media recovery

media recovery complete, elapsed time: 00:00:04

Finished recover at 16-OCT-26

 

Needs a valid backup of the datafile plus archivelogs (or flashback logs). Database stays open, only that block is unavailable during the repair.

Step A5: Verify

RMAN> VALIDATE DATAFILE 7 CHECK LOGICAL;

SQL> SELECT * FROM v$database_block_corruption;     -- expect no rows

 

Fallback when no backup exists

•      Index block: ALTER INDEX owner.idx REBUILD ONLINE; (data is rebuilt from the table, no loss).

•      Table block: salvage rows with DBMS_REPAIR, then CTAS to a new table. Rows in the bad block are lost.

EXEC DBMS_REPAIR.ADMIN_TABLES('REPAIR_TABLE',DBMS_REPAIR.REPAIR_TABLE,DBMS_REPAIR.CREATE_ACTION,'USERS');

 

DECLARE n BINARY_INTEGER; BEGIN

  DBMS_REPAIR.CHECK_OBJECT('APPS','XX_GL_LINES',corrupt_count=>n);

  DBMS_OUTPUT.PUT_LINE('corrupt='||n);

END; /

 

EXEC DBMS_REPAIR.SKIP_CORRUPT_BLOCKS('APPS','XX_GL_LINES',flags=>DBMS_REPAIR.SKIP_FLAG);

 

CREATE TABLE apps.xx_gl_lines_new AS SELECT * FROM apps.xx_gl_lines;

-- compare counts, rename, recreate indexes, constraints, grants, then drop the old table

 

Case B: ASM disk offlined after I/O error

Live example: error as seen (ASM alert log)

WARNING: Write Failed. group:1 disk:3 AU:5120 offset:0 size:4096

ORA-27072: File I/O error

Linux-x86_64 Error: 5: Input/output error

NOTE: process _user12345_+asm1 initiating offline of disk 3.3915941234 (DATA_0003) with mask 0x7e in group 1

WARNING: Disk 3 (DATA_0003) in group 1 will be dropped in: (12960) secs on ASM inst 1

 

The 12960 seconds is disk_repair_time (3.6 hours). If the disk comes back before the timer ends, ASM only resyncs changed extents. After it expires the disk is force-dropped and a full rebalance is needed.

Step B1: Check state and redundancy

SELECT name,type,offline_disks FROM v$asm_diskgroup;

SELECT name,path,mode_status,mount_status,header_status,repair_timer FROM v$asm_disk WHERE group_number=1;

 

NAME        MODE_STATUS  MOUNT_STATUS  HEADER_STATUS  REPAIR_TIMER

----------- ------------ ------------- -------------- ------------

DATA_0003   OFFLINE      MISSING       UNKNOWN        11840

 

•      EXTERNAL redundancy: the diskgroup is dismounted or at risk. Treat as an outage and go to Case C and the restore path.

•      NORMAL or HIGH redundancy: ASM serves reads from the mirror and also auto-repairs corrupt blocks from the good copy. The database keeps running.

Step B2: Fix the OS path (before touching ASM)

multipath -ll | grep -A6 DATA_0003

multipathd -k"show paths"

multipath -r                     # reload maps after storage team fixes the LUN/HBA

ls -l /dev/mapper/DATA_0003      # must be grid:asmadmin 660

 

Step B3: Bring the disk online

SQL> ALTER DISKGROUP DATA ONLINE DISK DATA_0003;

-- or all offline disks:

SQL> ALTER DISKGROUP DATA ONLINE ALL;

 

SQL> SELECT group_number,operation,state,power,est_minutes FROM v$asm_operation;

GROUP_NUMBER OPERATION STATE POWER EST_MINUTES

------------ --------- ----- ----- -----------

           1 REBAL     RUN       4           6

 

Step B4: If the disk was already force-dropped

-- repair timer expired; disk header_status is now FORMER or CANDIDATE

SQL> ALTER DISKGROUP DATA ADD DISK '/dev/mapper/DATA_0003' NAME DATA_0003 REBALANCE POWER 4;

 

Step B5: Verify

SQL> SELECT name,state,offline_disks FROM v$asm_diskgroup;      -- offline_disks 0

SQL> SELECT * FROM v$asm_operation;                              -- no rows when done

$ asmcmd lsdg

 

Do not drop other disks while a rebalance is running.

Case C: Diskgroup will not mount (ORA-15032 / ORA-15040 / ORA-15042)

Live example: error as seen

SQL> ALTER DISKGROUP DATA MOUNT;

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "3" is missing from group number "1"

 

Step C1: Which disks does ASM actually see

$ kfod disks=all status=true op=disks

$ asmcmd lsdsk --candidate -p

SQL> SHOW PARAMETER asm_diskstring

 

Compare against the expected disk count of the diskgroup. A disk missing here means OS, multipath, permission or asm_diskstring problem, not an ASM problem.

Step C2: Fix the visibility problem

•      Permissions or ownership reset after reboot/patch: chown grid:asmadmin and chmod 660, fix udev rules or AFD labels.

•      asm_diskstring does not match the device path: correct it and re-run kfod.

•      Multipath path down: see Case B Step B2.

Step C3: Mount

SQL> ALTER DISKGROUP DATA MOUNT;

 

Step C4: If a disk is truly lost and redundancy is NORMAL or HIGH

SQL> ALTER DISKGROUP DATA MOUNT FORCE;

-- then repair or replace the lost disk as in Case B Step B4

 

MOUNT FORCE does not help with EXTERNAL redundancy. In that case, the only path is restore from RMAN backup to a healthy diskgroup, or amdu to extract files from the surviving disks.

Step C5: Verify

$ asmcmd lsdg

SQL> ALTER DISKGROUP DATA CHECK ALL NOREPAIR;   -- then read the ASM alert log result

srvctl start database -d PRODDB

 

Case D: ASM disk header corruption (ORA-15196 / ORA-15036)

Live example: error as seen

ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483648] [0] [0 != 1]

ORA-15040: diskgroup is incomplete

WARNING: cache read a corrupt block: group=1(DATA) dsk=3 blk=1 disk=3 (DATA_0003) incarn=...

 

Step D1: Take a metadata and header backup first

$ asmcmd md_backup /tmp/dg_data.mdb -G DATA          # only if the diskgroup is mountable

$ dd if=/dev/mapper/DATA_0003 of=/tmp/DATA_0003.hdr bs=4096 count=1

 

Step D2: Read the primary header

$ kfed read /dev/mapper/DATA_0003 | egrep "kfbh.type|kfdhdb.hdrsts|kfdhdb.dskname|kfdhdb.grpname"

 

kfbh.type:    0 ; 0x002: KFBTYP_INVALID       <-- corrupt (healthy is KFBTYP_DISKHEAD)

kfdhdb.hdrsts: 0

 

Step D3: Confirm the backup header is intact

$ kfed read /dev/mapper/DATA_0003 aun=1 blkn=254 | egrep "kfbh.type|kfdhdb.hdrsts|kfdhdb.dskname"

 

kfbh.type:    1 ; 0x002: KFBTYP_DISKHEAD     <-- good

kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname: DATA_0003

 

Step D4: Repair the header

$ kfed repair /dev/mapper/DATA_0003

$ kfed read /dev/mapper/DATA_0003 | grep kfbh.type      # expect KFBTYP_DISKHEAD

 

Step D5: Mount and verify

SQL> ALTER DISKGROUP DATA MOUNT;

SQL> ALTER DISKGROUP DATA CHECK ALL NOREPAIR;

SQL> SELECT name,header_status,mode_status FROM v$asm_disk WHERE group_number=1;

 

If the backup header is also bad, stop. Use amdu to extract datafiles from the diskgroup, or restore from backup, and raise an Oracle Support SR.

Case E: ORA-26040 data block loaded with NOLOGGING

Live example: error as seen

ORA-01578: ORACLE data block corrupted (file # 9, block # 40210)

ORA-01110: data file 9: '+DATA/PRODDB/DATAFILE/xxgl_data.301.1098333111'

ORA-26040: Data block was loaded using the NOLOGGING option

 

Cause: a direct-path load or NOLOGGING operation (CTAS, INSERT /*+ APPEND */, index create) wrote blocks with no redo. After a restore/recovery or on a standby, those blocks are marked logically corrupt. This is not a hardware problem.

Step E1: Identify object and confirm NOLOGGING origin

SELECT owner,segment_name,segment_type FROM dba_extents

WHERE file_id=9 AND 40210 BETWEEN block_id AND block_id+blocks-1;

 

SELECT file#,unrecoverable_change#,unrecoverable_time FROM v$datafile WHERE file#=9;

SELECT force_logging FROM v$database;

 

Step E2: Fix

Redo does not exist for those blocks, so RMAN cannot repair them. Re-create the data from its source:

•      Index: ALTER INDEX owner.idx REBUILD;

•      Table: truncate and reload, or re-run the original load job, or recreate from source/export.

Step E3: Prevent

SQL> ALTER DATABASE FORCE LOGGING;

SQL> SELECT force_logging FROM v$database;     -- YES

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;       -- after any unavoidable NOLOGGING operation

 

Case F: ORA-00600 [kcbz_check_objd_typ]

Live example: error as seen

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_2], [0], [0], [1], [], [], [], [], [], [], [], []

Incident 12345 created, trace: .../diag/rdbms/proddb/PRODDB/trace/PRODDB_ora_28811.trc

 

Meaning: the block read does not match the object that owns it. Causes: in-memory corruption, lost write by storage, or a real on-disk corruption. Treat it as corruption until proven otherwise.

Step F1: Get the incident details

adrci> show incident

adrci> show incident -mode detail -p "incident_id=12345"

adrci> ips pack incident 12345            # package for Oracle Support

 

Step F2: Find the file and block in the trace

grep -iE "file#|rdba|dba:|obj#|dataobj" PRODDB_ora_28811.trc | head -20

 

-- convert rdba to file/block

SELECT dbms_utility.data_block_address_file(TO_NUMBER('01c1f2d1','xxxxxxxx')) fno,

       dbms_utility.data_block_address_block(TO_NUMBER('01c1f2d1','xxxxxxxx')) bno FROM dual;

 

Step F3: Validate the object and the file

RMAN> VALIDATE DATAFILE <fno> CHECK LOGICAL;

SQL> ANALYZE TABLE apps.xx_gl_lines VALIDATE STRUCTURE CASCADE ONLINE;

$ dbv file=<datafile> blocksize=8192       # for non-ASM copies; for ASM use RMAN VALIDATE

 

Step F4: Fix by outcome

•      RMAN reports the block in v$database_block_corruption: use Case A Step A4.

•      Clean validate but error repeats: bounce the instance to clear the buffer cache, check bug notes in My Oracle Support for the exact first argument and your RU level, and open an SR with the ips package.

•      Repeats across blocks/files: suspect storage lost writes. Engage the storage team and enable the checks below.

ALTER SYSTEM SET db_block_checksum=FULL SCOPE=BOTH;

ALTER SYSTEM SET db_block_checking=MEDIUM SCOPE=BOTH;

ALTER SYSTEM SET db_lost_write_protect=TYPICAL SCOPE=BOTH;

 

Note: db_block_checking adds CPU overhead; test impact before leaving it on in production.

Case G: ORA-15080 / ORA-15081 / ORA-27072 I/O errors

Live example: error as seen

ORA-15080: synchronous I/O operation failed to read block 0 of disk 3 in disk group DATA

ORA-15081: failed to submit an I/O operation to a disk

ORA-27072: File I/O error

Additional information: 4

 

Step G1: Prove it is the OS or storage

dmesg -T | grep -iE "i/o error|sd[a-z]+|blk_update_request|Medium Error"

journalctl -k --since "2 hours ago" | grep -i error

multipath -ll

dd if=/dev/mapper/DATA_0003 of=/dev/null bs=1M count=100 iflag=direct    # read test as grid

 

Step G2: Fix

•      Dead path or flapping HBA: storage/SAN team fixes zoning, cable, HBA; then multipath -r.

•      Permission change after reboot or patch: restore grid:asmadmin 660 on the device.

•      Failing LUN or disk: have storage replace it, then follow Case B Step B4 to add it and rebalance.

Step G3: Bring ASM and DB back to a clean state

SQL> ALTER DISKGROUP DATA ONLINE ALL;

SQL> SELECT * FROM v$asm_operation;

RMAN> VALIDATE CHECK LOGICAL DATABASE;

SQL> SELECT * FROM v$database_block_corruption;     -- expect no rows

 

Final validation and prevention checklist

•      asmcmd lsdg: all diskgroups MOUNTED, offline_disks 0, adequate free space for rebalance.

•      v$asm_disk: read_errs and write_errs 0, header_status MEMBER.

•      v$database_block_corruption: no rows. RMAN VALIDATE CHECK LOGICAL DATABASE clean.

•      Take a fresh level 0 backup and run RMAN CROSSCHECK; confirm archivelogs are continuous.

•      Set db_block_checksum=TYPICAL or FULL, db_block_checking=MEDIUM (after load test), FORCE LOGGING on.

•      Schedule regular asmcmd md_backup and a weekly RMAN VALIDATE CHECK LOGICAL.

•      Use Data Guard (Active Data Guard auto block repair) where possible.

•      Raise the root cause (SAN, HBA, multipath, firmware) with the storage team and record the RCA.

Do not do

•      Do not run kfed repair or ALTER DISKGROUP ... REPAIR before taking a backup and a metadata dump.

•      Do not set _allow_resetlogs_corruption or _corrupted_rollback_segments without Oracle Support.

•      Do not drop disks while a rebalance is running.

•      Do not run DBMS_REPAIR skip and CTAS on a table and discard the original before comparing row counts.