Oracle OUI will not detect ASM disks on EqualLogic PS6100

Storage

Storage
Gain insight and information on Dell Storage products from engineers, peers and experts in the industry.

Oracle OUI will not detect ASM disks on EqualLogic PS6100

This question has suggested answer(s)
RHEL 5.5 w/ ASMLIB

Oracle 11g w/ Dell EqualLogic PS6100 iSCSI SAN
I've also posted this in the Oracle Forums - hoping someone here has played with ASM and the ehcmd modules.

We've created the ASM disks and can see them before starting the OUI...

[root@dr-dbnode2 logs]# kpartx -l /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3
eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3p1 : 0 585954747 /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3 63

[root@dr-dbnode2 logs]# kpartx -l /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2
eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2p1 : 0 585954747 /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2 63

[root@dr-dbnode2 logs]# kpartx -l /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1
eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1p1 : 0 585954747 /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1 63

[root@dr-dbnode2 etc]# oracleasm listdisks
ASMDATA1
ASMDATA2
ASMDATA3

[root@dr-dbnode2 etc]# oracleasm querydisk -v -d -p ASMDATA1
Disk "ASMDATA1" is a valid ASM disk on device /dev/dm-2[253,2]
/dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1p1: LABEL="ASMDATA1" TYPE="oracleasm"
/dev/sdj1: LABEL="ASMDATA1" TYPE="oracleasm"

[root@dr-dbnode2 etc]# oracleasm querydisk -v -d -p ASMDATA2
Disk "ASMDATA2" is a valid ASM disk on device /dev/dm-8[253,8]
/dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2p1: LABEL="ASMDATA2" TYPE="oracleasm"
/dev/sdf1: LABEL="ASMDATA2" TYPE="oracleasm"

[root@dr-dbnode2 etc]# oracleasm querydisk -v -d -p ASMDATA3
Disk "ASMDATA3" is a valid ASM disk on device /dev/dm-4[253,4]
/dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3p1: LABEL="ASMDATA3" TYPE="oracleasm"
/dev/sdg1: LABEL="ASMDATA3" TYPE="oracleasm"

[grid@dr-dbnode2 ext]$ kfod asm_diskstring='/dev/oracleasm/disks/*' nohdr=true verbose=true disks=all status=true op=disks
286110 CANDIDATE /dev/oracleasm/disks/ASMDATA1 grid asmadmin
286110 CANDIDATE /dev/oracleasm/disks/ASMDATA2 grid asmadmin
286110 CANDIDATE /dev/oracleasm/disks/ASMDATA3 grid asmadmin

When we run the OUI we get to the 'Add Disks' screen and there are no disks listed at all...

So we tried to run with a -silent install and -force with the disks in our rsp file and the action log shows the following...

[root@dr-dbnode2 logs]# cat installActions2012-01-12_05-06-51PM.log

INFO: Waiting for completion of background operations
INFO: Completed background operations
INFO: Executing action at state CreateASMDiskGroup
INFO: Completed executing action at state <CreateASMDiskGroup>
INFO: Waiting for completion of background operations
INFO: Completed background operations
INFO: Moved to state <CreateASMDiskGroup>
INFO: Waiting for completion of background operations
INFO: Completed background operations
INFO: Validating state <CreateASMDiskGroup>
INFO: inventory location is/opt/oracle/app/oraInventory
INFO: inventory location is/opt/oracle/app/oraInventory
INFO: inventory location is/opt/oracle/app/oraInventory
INFO: Is CRS Installed false
INFO: while getting CSS versionjava.lang.Exception: Unable to find the specified Oracle Home.
WARNING: KFOD not found at
INFO: inventory location is/opt/oracle/app/oraInventory
WARNING: KFOD not found at
FINE: oracle.install.commons.util.Resource:Can't find resource for bundle oracle.install.asm.resource.ASMErrorResID, key oracle.install.asm.bean.ASMInputValidationErrorCode.hint
SEVERE: [FATAL] [INS-30508] Invalid ASM disks.
CAUSE: The disks [/dev/oracleasm/disks/ASMDATA1, /dev/oracleasm/disks/ASMDATA2, /dev/oracleasm/disks/ASMDATA3] were not valid.
ACTION: Please choose or enter valid ASM disks.
INFO: Advice is ABORT
INFO: Adding ExitStatus INVALID_USER_INPUT to the exit status set
INFO: Completed validating state <CreateASMDiskGroup>
INFO: Terminating all background operations
INFO: Terminated all background operations
FINE: oracle.install.commons.util.Resource:Can't find resource for bundle oracle.install.commons.util.resource.ErrorCodeResourceBundle, key oracle.install.commons.util.exception.CommonErrorCode.hint
INFO: Finding the most appropriate exit status for the current application
INFO: Exit Status is -2
INFO: Shutdown Oracle Grid Infrastructure
INFO: Unloading Setup Driver

Been stuck here for a couple days...

My current direction is looking at the "WARNING: KFOD not found at" message...

kfod does exist in the /tmp/OraInstall2012-01-12_04-40-31PM/ext/bin/kfod folder while the installation is running...so I'm not sure what to make of it.

Let me know if anyone can shed some light here...

Thanks in advance,

Colby
All Replies
  • colby,

    Not sure if you looked at this document, but it might be worth a review to ensure you covered the best practices first: www.equallogic.com/.../DownloadAsset.aspx

    Also, this link has a boatload of info for Oracle: www.equallogic.com/.../default.aspx

    -Joe

    Follow me on Twitter: @joesatdell

     

  • Hi Joe,

    We've been through a significant amount of the reading on those links.  The articles were helpful to us for setting up the EqualLogic array to connect from the OS perspective.

    It's the Oracle Universal Installer (OUI) that will not 'discover' the ASM disks.  It appears that the ASM disks are being created and seen from the cmd line.  I was hoping that someone could shed some light on the reason the OUI doesn't see the ASM disks.  I think this is an Oracle issue, but I was hoping someone at Dell had come across and solved this during their testing of various Oracle installations.

    Basically I haven't been able to find anyone specifically seeing what we're seeing so I put out the feeler's as far as I could reach them.

    Thanks again,

    Colby

  • It appears that you are using EqualLogic HITKit

    did you implement the work around which Dell recommended? like adding a file in /etc/udev/rules.d directory?

    if so give me the contents of

    1. /etc/sysconfig/oracleasm

    2. ls -l /dev/dm*

    3. ls -l /dev/oracleasm/disks

    3. ls -l /dev/mapper/eql*

  • adendukuri, thanks for the response.

    I did not follow a specific rules.d work around.  I googled for a related work around but did not find it.  Can you point me to the work around you're referring to?

    Yes, we are using the HIT Kit - installed as per the guide:

         Host Integration Tools for Linux - Installation and User Guide - Version 1.0 - January 2011

    The installation does create a rules.d file:      99-equallogic.rules

    The info requested to follow...

  • The info you asked for is below:

    [root@dr-dbnode2 ~]# cat /etc/sysconfig/oracleasm

    #

    # This is a configuration file for automatic loading of the Oracle

    # Automatic Storage Management library kernel driver.  It is generated

    # By running /etc/init.d/oracleasm configure.  Please use that method

    # to modify this file

    #

    # ORACLEASM_ENABELED: 'true' means to load the driver on boot.

    ORACLEASM_ENABLED=true

    # ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

    ORACLEASM_UID=grid

    # ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

    ORACLEASM_GID=asmadmin

    # ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

    ORACLEASM_SCANBOOT=true

    # ORACLEASM_SCANORDER: Matching patterns to order disk scanning

    ORACLEASM_SCANORDER=""

    # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

    ORACLEASM_SCANEXCLUDE=""

    [root@dr-dbnode2 ~]# ls -l /dev/dm*

    ls: /dev/dm*: No such file or directory

    [root@dr-dbnode2 ~]# ls -l /dev/oracleasm/disks/

    total 0

    brw-rw---- 1 grid asmadmin 8, 113 Jan 17 13:30 ASMDATA1

    brw-rw---- 1 grid asmadmin 8,  65 Jan 17 13:30 ASMDATA2

    brw-rw---- 1 grid asmadmin 8, 145 Jan 17 13:30 ASMDATA3

    [root@dr-dbnode2 ~]# ls -l /dev/mapper/eql*

    brw-rw---- 1 root disk 253, 12 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1

    brw-rw---- 1 root disk 253, 13 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-074e2e258-08670a9557f4f0dd-oracleasmdisk1p1

    brw-rw---- 1 root disk 253,  6 Jan 17 13:14 /dev/mapper/eql-8-cb2b76-076e2e258-53670a9556a4f075-oraclefra1

    brw-rw---- 1 root disk 253,  7 Jan 17 13:14 /dev/mapper/eql-8-cb2b76-076e2e258-53670a9556a4f075-oraclefra1p1

    brw-rw---- 1 root disk 253,  3 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2

    brw-rw---- 1 root disk 253,  4 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0d8e2e258-55270a955854f0dd-oralceasmdisk2p1

    brw-rw---- 1 root disk 253,  8 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3

    brw-rw---- 1 root disk 253,  9 Jan 17 13:34 /dev/mapper/eql-8-cb2b76-0fde2e258-3ac70a955884f0dd-oracleasmdisk3p1

  • [root@dr-dbnode2 ~]# cat /etc/udev/rules.d/99-equallogic.rules

    #-----------------------------------------------------------------------------

    #  Copyright (c) 2010 by Dell, Inc.

    #

    # All rights reserved.  This software may not be copied, disclosed,

    # transferred, or used except in accordance with a license granted

    # by Dell, Inc.  This software embodies proprietary information

    # and trade secrets of Dell, Inc.

    #

    #-----------------------------------------------------------------------------

    #

    # These udev rules should serve as a guide for setting up system-wide UDEV

    # rules for Dell EqualLogic storage devices.

    #

    # Rule #1: Set the SCSI subsystem device timout to 60s for every Dell

    # EqualLogic device detected.  It is not recommended to change this from

    # the value set here.

    ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

    RUN+="/bin/sh -c 'echo 60 > /sys/$DEVPATH/device/timeout'"

    # Example Rule #1: The recommended read-ahead size is between 1024 and 4096.

    #

    # Higher values are better for sequential access by prefetching this amount (in

    # kb), at the cost of random access performance.

    #

    # Uncomment the following two lines to set the read ahead size to 1024kb for

    # all Equallogic devices:

    ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

    RUN+="/bin/sh -c 'echo 1024 > /sys/$DEVPATH/queue/read_ahead_kb'"

    # Example Rule #2: per-WWID matching

    #

    # You can override values on a per-volume basis, based on the SCSI WWID value.

    #

    # Cut and paste this rule as many times as necessary, replacing the example

    # WWID (in the RESULT=="..." line) with the actual WWID of the SCSI device you

    # wish to match.

    #

    # You can also replace the setting being altered, for example to change the

    # disk IO scheduler instead, change 'read_ahead_kb' to 'scheduler'.

    #ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",

    #PROGRAM="/sbin/scsi_id --whitelist --device=$tempnode",

    #RESULT=="360c090629cc917ce72cbb40000003000",

    #RUN+="/bin/sh -c 'echo 2048 > /sys/$DEVPATH/queue/read_ahead_kb'"

  • We managed to figure it out...

    It turns out that a previously failed -silent run of the OUI had left some Oracle files hanging about.  The OUI not showing the ASM disks because they were associated to the failed installation.  After removing 'ALL' the files from the failed installation and a reboot we were able to get through the full install successfully.

    I also added the following to the end of the /etc/udev/rules.d/99-equallogic.rules to ensure the mode and ownership is correct.

    This properly sets things on the /dev/sd* block devices but doesn't change the /dev/mapper/eql*...

    I'm thinking that we'll need to get the /dev/mapper/eql* devices set properly as well...but the install succeeded with root:disk as the owner...we'll find out after a reboot...?

    ------------------------------------------------

    [root@dr-dbnode2 ~]# cat /etc/udev/rules.d/99-equallogic.rules

    <...same contents as above with the following appended...>

    ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

    RUN+="/bin/sh -c 'chmod 660 /dev/%k'"

    ACTION=="add",SUBSYSTEM=="block",ENV{ID_VENDOR}=="EQLOGIC",ENV{ID_FS_USAGE}=="",\

    RUN+="/bin/sh -c 'chown grid:asmadmin /dev/%k'"

    ------------------------------------------------

    adendukuri - Is this the work around that you were mentioning?

    Thanks for the response,

    cs

  • No, That is not the work around which I am talking about

    1. the best practice is to have /etc/sysconfig/oracleasm file edit and should have

    ORACLEASM_SCANORDER="dm-"

    # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

    ORACLEASM_SCANEXCLUDE="sd"

    2. after failed install you may have to write zeroes to the cluster related ASM disks using dd command

    3. it is important that you need to have dm devices and to create them equallogic engineering team came with workaround that create dm devices. This is a must have work around in place. Else you will have problem in bringing cluster nodes except the one which comes up first. this is Oracle undocumented requirement. This is must because this dm device maintains persistant name across all the nodes otherwise not possible with "sd" devices.

    4. that rules.d file creates such dm devices which will have different major (253 which is constant) and minor ( variable number) for each of device.

    I am positive that if you reboot the cluster nodes you may not able bring up one of the node even though you built the cluster first time successfully.

    hope this helps

  • Hello adendukuri,

    I think I see what you're saying, we do not have any /dev/dm-X devices created.  I have been searching for some details of the work around but have no success finding anything other than this post.  I'm still digging, but do you happen to have any further information on the details?

    Looks like I'll be reading up on udev...<grin>

    For the current installation we're running a standalone Oracle instance so it's been fine after reboots.  This installation is destined  to be a DR site setup.  However, we are shooting for consistency across our sites so it will matter from our primary Oracle RAC setup.

    Thanks again,

    Colby

    P.S. Furhter to our conversation - it appears that my configuration may not yet be supported - see below.  I missed reading this statement in any documentation we had before I started our installation.

    DELL-Adam M 08-17-2011 4:04 PM - "DISCLAIMER: An Oracle RAC configuration with ASM using the EqualLogic HITKit is currently not officially supported by Dell. There are engineering efforts ongoing to provide its availability as a supported configuration, but is unavailable as such at this time."

    en.community.dell.com/.../getting-more-iops-on-your-oracle-workloads-running-on-red-hat-enterprise-linux-and-dell-equallogic-with-eql-hitkit.aspx