Published on January 10, 2019 by Matt Edwards
I once had a situation where a Unix admin took 4 LUNs on a linux system and added them to a filesystem, as it was running out of space.
Unfortunately, those LUNs were actively used by ASM – and 4 databases were destroyed due to the data being wiped out. That was a VERY long weekend for me as you can imagine. I’d rather not repeat that.
There are now 3 ways to manage your ASM disks, ASMLIB, UDEV, and now the ASM Filter Driver. ASMLIB will most likely be deprecated at some point, and while UDEV is solid – Oracle has built in some great features in the ASM Filter Driver, so I want to focus on using that to manage the storage devices.
One of the key benefits of the ASM Filter Driver is that you cannot accidentally overwrite the data. Oracle installs a kernel module that only allows writes from oracle processes. Other writes are rejected by the kernel. This would have prevented that disaster weekend where we had to restore several large databases.
Here’s a demonstration on setting up the ASM Filter Driver, and how a non oracle process write is rejected by the kernel.
Once you unzip the 12.2 installation, we need to set the storage devices to be managed by the ASM Filter Driver. For this demo, I will be using /dev/sdb for the ASM disk, on Oracle Enterprise Linux 7.5 with unbreakable kernel 4.1.12-124.20.7.
[root@localhost bin]# ./asmcmd afd_label DATA1 /dev/sdb --init
[root@localhost bin]# ./asmcmd afd_lslbl /dev/sdb
--------------------------------------------------------------------------------
Label Duplicate Path
================================================================================
DATA1 /dev/sdb
From this point forward, setup the GRID installation as you would normally, and make sure the “Use ASM Filter Driver” box is checked. It should be by default.
Once you’re running, you can verify that filtering is enabled:
[oracle@localhost disks]$ $ORACLE_HOME/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'localhost.localdomain'
SQL> SELECT SYS_CONTEXT('SYS_ASMFD_PROPERTIES', 'AFD_STATE') FROM DUAL;
SYS_CONTEXT('SYS_ASMFD_PROPERTIES','AFD_STATE')
--------------------------------------------------------------------------------
CONFIGURED
[oracle@localhost disks]$ cd /dev/oracleafd/disks
[oracle@localhost disks]$ ls -al
total 4
drwxrwxr-x 2 oracle oracle 60 Nov 3 02:58 .
drwxrwxr-x 3 oracle oracle 80 Nov 3 02:58 ..
-rwxrwx--- 1 oracle oracle 9 Nov 3 02:58 DATA1
[oracle@localhost disks]$ cat DATA1
/dev/sdb
Now what I’m going to do is overwrite & destroy the contents of the /dev/sdb disk, which is used by ASM.
[oracle@localhost disks]$ sudo su - root
[root@localhost bin]# dd if=./afdboot of=/dev/sdb
159418+1 records in
159418+1 records out
81622419 bytes (82 MB) copied, 4.00743 s, 20.4 MB/s
Well, it looks the writes went through. Or DID they ? Let’s check the kernel messages:
[root@localhost log]# dmesg
[ 1375.685657] F 4296038.287/181103071201 systemd-udevd[547] oracleafd:18:0894:Write IO to ASM managed device: [8] [16]
[ 1375.685661] Buffer I/O error on dev sdb, logical block 0, lost async page write
[ 1864.988174] Buffer I/O error on dev sdb, logical block 0, lost async page write
[ 1864.988180] Buffer I/O error on dev sdb, logical block 1, lost async page write
[ 1864.988183] Buffer I/O error on dev sdb, logical block 2, lost async page write
[ 1864.988186] Buffer I/O error on dev sdb, logical block 3, lost async page write
[ 1864.988189] Buffer I/O error on dev sdb, logical block 4, lost async page write
[ 1864.988191] Buffer I/O error on dev sdb, logical block 5, lost async page write
[ 1864.988194] Buffer I/O error on dev sdb, logical block 6, lost async page write
[ 1864.988196] Buffer I/O error on dev sdb, logical block 7, lost async page write
[ 1864.988199] Buffer I/O error on dev sdb, logical block 8, lost async page write
[ 1864.988201] Buffer I/O error on dev sdb, logical block 9, lost async page write
We see here that the kernel rejected the IO’s to the /dev/sdb disk, as it did not come from an Oracle process. Let’s verify the disk is unharmed by restarting ASM and seeing if it can mount DATA1:
SQL> shutdown immediate;
ASM diskgroups volume disabled
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup;
ASM instance started
Total System Global Area 1140850688 bytes
Fixed Size 8629704 bytes
Variable Size 1107055160 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL>
It came up clean, let’s also check the files inside the diskgroup using ASMCMD:
[oracle@localhost ~]$ asmcmd
ASMCMD> ls
DATA/
ASMCMD> cd DATA
ASMCMD> ls
ASM/
orapwasm
ASMCMD> cd ASM
ASMCMD> ls
ASMPARAMETERFILE/
PASSWORD/
ASMCMD> cd ASMPARAMETERFILE
ASMCMD> ls
REGISTRY.253.991191939
All looks good ! the writes were completely rejected and ASM is still intact. NOW, let’s disable the filtering and
repeat this exercise to see what happens:
ASMCMD> afd_filter -d --all
ASMCMD> afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'localhost.localdomain'
ASMCMD> afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
DATA1 DISABLED /dev/sdb
[root@localhost bin]# dd if=./afdboot of=/dev/sdb
159418+1 records in
159418+1 records out
81622419 bytes (82 MB) copied, 3.78481 s, 21.6 MB/s
“dmesg” shows no rejects on the writes from the above “dd” command. Let’s see how ASM responds to a restart:
SQL> shutdown immediate;
ASM diskgroups volume disabled
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup;
ORA-00099: warning: no parameter file specified for ASM instance
ASM instance started
Total System Global Area 1140850688 bytes
Fixed Size 8629704 bytes
Variable Size 1107055160 bytes
ASM Cache 25165824 bytes
ORA-15110: no diskgroups mounted
SQL>
The disk is destroyed, and ASM cannot start as there is no parameter it can use anymore. Let’s look at the contents using ASMCMD:
[oracle@localhost ~]$ asmcmd
ASMCMD> ls
ASMCMD>
Nothing shows. The diskgroup cannot be mounted as we destroyed it using “dd”, and thus there is no info to show In ASMCMD.
I hope this gives a good overview of the ASM Filter Driver and the benefits it brings. Post a comment below or get in contact with us if you would like to learn more.
Share on
