Equallogic iscsi error event when running Commvault simpana FS backup

Storage

Storage
Gain insight and information on Dell Storage products from engineers, peers and experts in the industry.

Equallogic iscsi error event when running Commvault simpana FS backup

  • Hi there,

    First of, let me explain the situation:

    We have an equallogic storage system and are using commvault simpana as backup. when doing a File system backup during the night, there are also snapshot taken at the same time. When doing this the Equallogic generates the following error event:


    http://10.1.1.22/



    -----------------------------------------

    ERROR event from storage array IFSSTOR01

    subsystem: MgmtExec

    event: 7.4.3

    time: Tue May 25 21:47:03 2010



    iSCSI login to target '10.1.1.22:3260, iqn.2001-05.com.equallogic:0-8a0906-71f984f06-3ec8ab688e04bfc2-ifsprod-data-2010-05-25-21:38:07.14123' from initiator '10.1.1.226:54248, iqn.1991-05.com.microsoft:dsgifso1.dsgor.dsg' failed for the following reason:

    Requested target not found.

    -----------------------------------------

    Now the iDA FS backup uses VSS. You can configure this to NOT use VSS when doing a backup. When we disable the use of VSS on the iDA FS, then the Equallogic System does not generate this error event anymore. Now it seems that the Comvmault iDA FS backup uses VSS it makes a vss snapshot that is seen as a iscsi target to which the server want to conntect. This fails as the iscsi target does not exists.

    Question as to why does it do this?? and how to skip/fix this as it does not seem to effect the server and/or the backup. (in other words, server and backup seems all fine)

    Anyone that can help??
  • Hi,
    This seems to be a rather old post, but as I'm experiencing the exact same problem (using Symantec Backup exec), I was wondering if you ever found a solution for this?
    Strangly enough, we only get this error on 2008R2 machines (approx 10 out of 50 servers that perform daily backups)
  • bump
  • Gatmopawiro,

    Hi, I’m Joe with Dell EqualLogic.

    In your example it appears that when Commvault requested snapshot “0-8a0906-71f984f06-3ec8ab688e04bfc2-ifsprod-data-2010-05-25-21:38:07.14123”,the array returned the message that it no longer existed on the array (BTW, this is an old snapshot from 05/25/2010 for the volume ifsprod-data).

    The snapshot access is requested by “Commvault Simpana”, and the message on the array is just information, stating that the Snapshot no longer exists on the array (the snapshot could have been deleted, or was overwritten by new snapshots, possibly due to not enough snapshot reserve space for that volume).

    So the array is just pointing out that someone is trying to request a snapshot that no longer exists. You should first confirm that it is not there, if it is, then please open a support case with us.

    If you find that snapshot is no longer on the array, you will need to instruct Commvault to stop looking for it, sorry but I don’t know how to do that. You can try and post the question on Commvault’s forum at: http://www.commvaultsupport.com/cgi-bin/yabb2/YaBB.pl, or contact Commvault directly.

    Regards,
    Joe

    Follow me on Twitter: @joesatdell

     

  • Hi Joe,

    I don't think is a Commvault problem, as we have exactly the same issue, but we use Smantec Backup Exec. We see this exact problem on our 2008R2 servers only... On a daily basis on all those servers...
  • gatmopawiro,

    Also check the iSCSI persistant connections in the initiator, it could be that this initiator ('10.1.1.226:54248, iqn.1991-05.com.microsoft:dsgifso1.dsgor.dsg) didn't properly clear out the snapshot connection after it was completed.

    Regards,
    Joe

    Follow me on Twitter: @joesatdell

     

  • Wow, I am not sure how we missed this post for so long. Yeah Joe, that snapshot is old because the question is old. ;)

    The real question that must be answered first: Is the snapshot in question really there on the array? And if it is there, is it online? The array is very literal, it will log into a volume or snapshot if they are present, online and the initiator has access granted through the ACL.

    I suspect that the snapshot is either getting deleted (check the event log in the array) or not getting set online.
    Or like Joe said, it may have been accidently added to the persistent targets list in the iSCSI initiator.

    Aaron
  • Hi Aaron & Joe

    As mentioned: I'm having the same issue, using Symantec backup Exec.

    We get similar messages on a daily basis for all servers running 2008R2...

    -----------------------------------------
    ERROR event from storage array OUDEQL2
    subsystem: MgmtExec
    event: 7.4.3
    time: Fri Apr 8 05:16:33 2011

    iSCSI login to target '192.168.40.1:3260, iqn.2001-05.com.equallogic:0-8a0906-97d491c09-13a0002b89f4d9e7-filesrvr-vol3-2011-04-08-04:57:01.59565' from initiator '192.168.40.126:54657, iqn.1991-05.com.microsoft:filesrvr1.azo.lan' failed for the following reason:
    Requested target not found.
    -----------------------------------------

  • tdemeyer,

    Sorry if the answer wasn't clear. Typically the two reasons are A) Not enough snapshot reserve allocated for the volumes that the snapshot is located on, or B) A stuck or persistent iSCSI reservation for a snapshot that no longer exists.

    First ensure that the snapshot is no longer there (review the volumes snapshot list); and B) that it did exist at one time (review the event log).

    To address the issue of not enough snapshot reserve; A) look in your event log, you should see the event for the snapshot being created, then at some point in the log after that, before the backup actually connects to the snapshot, the snapshot will be deleted. This is an indication of not enough snapshot reserve. You will need to increase this reserve for every volume you see this message, in your example it’s the volume called "filesrvr-vol3". Start by doubling or tripling the reserve, and as you become more familiar with your reserve space, you can gradually decrease it to gain back free space for new volumes.

    For a stuck iSCSI connection, review the iSCSI initiator connections and persistent connection on the host attempting to connect to the snapshot. If the snapshot is listed, (and you confirmed that the snapshot is no longer on the array) just disconnect the target using the iSCSI Initiator on the host, and remove the persistent connection if it exists.

    Finally, if the snapshot is there, but you still see the message, we would need to look at the array DISGs, for this, you would need to open a support case with us.

    Regards,
    Joe

    Follow me on Twitter: @joesatdell

     

  • I opened a case with Dell Equallogic support on this and they said that the error isn't anything to worry about, but is just a warning.  From the way it was explained to me is that the backup software initiated a command to create a snapshot, the snapshot is created, backup performed then snapshot is removed.  The windows server tries to reconnect to that volume but it doesn't exist thus the error.  I'm getting the error with two of our exchange servers.

  • I get this error every night and wake up to 99+ messages in my inbox stating "Requested target not found". When I log into my server and go to the iScsi Initiator there is a target in reconnecting status. All I have to do is disconnect it and it's fine but it is becoming extremely annoying to have to do this everyday. I use CommVault Simpana 9.0.0 R2. I know the backup software is issuing a command to create the snapshot and then removes it when it's done but why the reconnecting thing with Windows. The thing that is even more frustrating is I opened a ticket with CommVault and they said it's Equallogic's problem and from what I'm seeing here is Equallogic is saying it's the backup software's problem. Does anyone know of any tweaks that can remedy this issue? I have checked and have plenty of snapshot reserve.

  • BTW - This only happens with my Exchange Database backups. The file system iData Agent does just fine.

  • Adding this registry key fixed the error message for us.

    [HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\MSExchangeDBAgent]
    "nUseVSSSoftwareProvider"=dword:00000001
    
    
    Basically, what was happening is that commvault opened 4 connections but after the backup only closed 3 of the connections.  The Equallogic deleted the snapshot, then windows tried reconnecting the 1 connection it didn't disconnect.  The above key changes it so it uses vss instead of the snapshot.
  • Thanks for the info. Before I make these changes, did you notice and performance issues with how log the backups take to complete after making the change? or any other issues?