MD3200 Lockout - Keyboard input not working - PowerVault Storage Forum - Storage - Dell Community

MD3200 Lockout - Keyboard input not working

Storage

Storage
Information and ideas on Dell storage solutions, including DAS, NAS, SAN and backup.

MD3200 Lockout - Keyboard input not working

  • I have an issue with our MD3200. We just got a replacement controller and I cannot clear the lockdown. I have a serial cable connected and Putty to view the serial connection. The end of the boot sequence reads:

    08/29/17-21:08:40 (tRAID): NOTE: SYMbol available
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** WARNING!!!!!! **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** This controller is not running the correct firmware **
    08/29/17-21:08:40 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
    08/29/17-21:08:40 (tRAID): WARN: ** In order to prevent any configuration loss, the **
    08/29/17-21:08:40 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
    08/29/17-21:08:40 (tRAID): WARN: ** firmware release. **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:41 (tRAID): SOD: sodComplete Notification Complete
    08/29/17-21:08:41 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 43 seconds
    08/29/17-21:08:41 (tRAID): sodMain complete
    08/29/17-21:08:40 (tRAID): WARN: SOD lockdown sequence complete - suspending...

    After that, nothing. anything I type will appear in the console but it ignores the command. I tried typing LemClearLockdown and it just skips to the next line without completing. I can type nonsense and it jumps to the next line as well. When I press ctrl + break I get:

    Press within 5 seconds: <S> for Service Interface, <BREAK> for baud rate

    Once again, when I enter S it does nothing but skip to the next line. Does anyone have any ideas on how to fix this?

    Thanks!

  • We have an MD3200 with a new controller. I cannot clear the lockdown because it never seems to finish booting. it gets as far as 

    08/29/17-21:08:40 (tRAID): NOTE: SYMbol available
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** WARNING!!!!!! **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** This controller is not running the correct firmware **
    08/29/17-21:08:40 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
    08/29/17-21:08:40 (tRAID): WARN: ** In order to prevent any configuration loss, the **
    08/29/17-21:08:40 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
    08/29/17-21:08:40 (tRAID): WARN: ** firmware release. **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: ** **
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:40 (tRAID): WARN: *********************************************************
    08/29/17-21:08:41 (tRAID): SOD: sodComplete Notification Complete
    08/29/17-21:08:41 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 43 seconds
    08/29/17-21:08:41 (tRAID): sodMain complete
    08/29/17-21:08:40 (tRAID): WARN: SOD lockdown sequence complete - suspending...

    Then, each time I type a command, it just jumps to the next line. I cannot enter any commands. 

    When I press ctrl + break I get the message 

    Press within 5 seconds: <S> for Service Interface, <BREAK> for baud rate

    but entering S does nothing. Can anyone suggest something I haven't tried yet?

    Thanks!

  • Hello Omega876,

    Is this a single controller MD3200 or is it a dual controller MD3200? What is the current version of firmware on your MD3200? It sounds like the firmware on the replacement is not the same as the firmware on your MD3200.

    Please let us know if you have any other questions.

    DELL-Sam L
    Dell | Social Outreach Services - Enterprise
    Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)

  • Sam,

    This is a dual controller MD3200 but I only have one controller plugged in. In my previous tests with different controllers (I have 3, one came with the unit, two are replacements) I am able to control it with console commands and even add it to MDSM with no issues (other than it tells me I am missing the other controller).

    So for this test, I just wanted to test the controller alone without the influence of the other working controller, if that makes sense. 

    How do I check the firmware of the MD3200?

    Thanks.

  • Just an update,

    I read somewhere that the controllers, if not on the same firmware, will try and update one to the other. I attempted to connect both controllers and ended up corrupting my working controller. So now I have one controller that freezes during boot and one controller that is locked out and unable to accept shell commands. 

  • Hello Omega876,

    Yes you are correct, when a replacement controller is inserted, it will try to update to the current version of firmware that is running on active controller. When replacing a controller it is best to offline the controller in MDSM before removal.

    You should be able to test the non-working controller in your MD3200, and see if it will copy the configuration to it. You should be able to insert just the non-working controller by itself & it should copy the configuration & firmware from your HDD.

    Please let us know if you have any other questions.

    DELL-Sam L
    Dell | Social Outreach Services - Enterprise
    Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)

  • Hi Sam,

    Since we bought the unit used, I'm not sure if the controllers were set to offline. What I know, is when the MD3200 arrived, I couldn't manage the controllers at all from MDSM. I was able to interact with the Shell via a console cable, and enter commands, but I couldn't add both controllers to MDSM no matter what I tried. The controller we suspected was bad, we sent back. The replacement controller, when booted by itself shows the following. As you can see, none of my commands or typing are registered at the bottom.

    =<###>=-
    Instantiating /ram as rawFs, device = 0x1
    Formatting /ram for DOSFS
    Instantiating /ram as rawFs, device = 0x1
    Formatting...Retrieved old volume params with %38 confidence:
    Volume Parameters: FAT type: FAT32, sectors per cluster 0
    0 FAT copies, 0 clusters, 0 sectors per FAT
    Sectors reserved 0, hidden 0, FAT sectors 0
    Root dir entries 0, sysId (null) , serial number 10000
    Label:" " ...
    Disk with 1024 sectors of 512 bytes will be formatted with:
    Volume Parameters: FAT type: FAT12, sectors per cluster 1
    2 FAT copies, 1010 clusters, 3 sectors per FAT
    Sectors reserved 1, hidden 0, FAT sectors 6
    Root dir entries 112, sysId VXDOS12 , serial number 10000
    Label:" " ...

    RTC Error: Real-time clock device is not working
    OK.

    Adding 13818 symbols for standalone.


    Reset, Power-Up Diagnostics - Loop 1 of 1
    3600 Processor DRAM
    01 Data lines Passed
    02 Address lines Passed
    3300 NVSRAM
    01 Data lines Passed
    4410 Ethernet 82574 1
    01 Register read Passed
    02 Register address lines Passed
    6D40 Bobcat
    02 Flash Test Passed
    3700 PLB SRAM
    01 Data lines Passed
    02 Address lines Passed
    6D50 LSISAS2008 IOC 1
    01 Register Read Test Passed
    02 Register Address Lines Test Passed
    03 Register Data Lines Test Passed
    3900 Real-Time Clock
    01 RT Clock Tick Passed
    Diagnostic Manager exited normally.
    09/05/17-19:41:22 (tNetCfgInit): NOTE: eth0: LinkUp event
    09/05/17-19:41:22 (tNetCfgInit): NOTE: Acquiring network parameters for interface gei0 using DHCP
    09/05/17-19:41:30 (ipdhcpc): NOTE: netCfgDhcpReplyCallback :: received OFFER on interface gei0, unit 0
    09/05/17-19:41:31 (ipdhcpc): NOTE: DHCP server: 10.1.14.3
    09/05/17-19:41:31 (ipdhcpc): WARN: **WARNING** The DHCP Server did not assign a permanent IP for gei0.
    09/05/17-19:41:31 (ipdhcpc): WARN: Network access to this controller may eventually fail.
    09/05/17-19:41:31 (ipdhcpc): NOTE: Client IP routers: 10.1.14.3
    09/05/17-19:41:31 (ipdhcpc): NOTE: Client DNS name servers: 10.1.14.3
    09/05/17-19:41:31 (ipdhcpc): NOTE: DNS domain name: routerf3bc84.com
    09/05/17-19:41:31 (ipdhcpc): NOTE: Assigned IP address: 10.1.14.108
    09/05/17-19:41:31 (ipdhcpc): NOTE: Assigned subnet mask: 255.255.255.0
    09/05/17-19:41:31 (tNetReset): NOTE: Network Ready


    Current date: 09/05/17 time: 10:05:53

    Send <BREAK> for Service Interface or baud rate change
    09/05/17-19:41:34 (tRAID): NOTE: Set Powerup State
    09/05/17-19:41:34 (tRAID): SOD Sequence is Normal, 0 on controller A
    09/05/17-19:41:35 (tRAID): NOTE: Turning on tray summary fault LED
    09/05/17-19:41:35 (tRAID): NOTE: Installed Protocols: <MTPs: SAS USB > <ITPs: RDMA ISER IPOIB > <STPs: FCP RmtDMA iSCSI SAS >
    09/05/17-19:41:35 (tRAID): NOTE: Required Protocols: <MTPs: SAS > <ITPs: UNK > <STPs: SAS >
    09/05/17-19:41:36 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
    09/05/17-19:41:36 (tRAID): NOTE: RCBBitmapManager total RPA size = 1828716544
    09/05/17-19:41:36 (tRAID): NOTE: HealthCheck: AltFwStableThresh set from 60 to 120
    09/05/17-19:41:36 (tRAID): NOTE: init: ioc: 0, PLVersion: 06-000-05-00
    09/05/17-19:41:37 (tRAID): NOTE: Initiating Host channel: ioc:1 bringup
    09/05/17-19:41:37 (tRAID): NOTE: IOC HW detected, index:0 type:host chan:0 chipID:72 rev:3
    09/05/17-19:41:38 (IOSched): NOTE: SAS Expander Added: expDevHandle:x11 enclHandle:x2 numPhys:25 port:2 ioc:0 channel:0
    09/05/17-19:41:38 (tSasExpChk): NOTE: Local Expander Firmware Version: 00.00.80.04
    09/05/17-19:41:39 (tRAID): NOTE: IOC Firmware Version: 06-00-05-00
    09/05/17-19:41:40 (tRAID): WARN: SendIocInit -- Using SHELL Bus for HIC
    09/05/17-19:41:43 (tRAID): SOD: Instantiation Phase Complete
    09/05/17-19:41:42 (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels 0-0
    09/05/17-19:41:42 (tRAID): NOTE: LockMgr Role is Master
    09/05/17-19:41:46 (tSasDiscCom): NOTE: SAS: Initial Discovery Complete Time: 30 seconds since last power on/reset, 10 seconds since sas instantiated
    09/05/17-19:41:46 (tRAID): NOTE: WWN baseName 0006782b-cb4199af (valid==>SigMatch)
    09/05/17-19:41:46 (tRAID): NOTE: spmEarlyData: No data available
    09/05/17-19:41:47 (tRAID): SOD: Pre-Initialization Phase Complete
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10000 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10001 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10002 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10003 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10004 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10005 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10006 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10007 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (ssmSpin): WARN: Drive 0x10008 incompatible DSM LockedOutStatus=6 Reason=2
    09/05/17-19:42:16 (tRAID): WARN: dbm::RWFileSystem::initialize: Exception caught, ConstructorIOException: -16
    09/05/17-19:42:16 (tRAID): ERROR: In PersistenceManager::initialize: catch DbmNoFileSystemException: recType: 84
    09/05/17-19:42:16 (tRAID): ERROR: ADM Load Reservations failed with error (5) Exception
    09/05/17-19:42:16 (tRAID): WARN: CmgrLockdownException: UP REVISION SSTOR - LOCKDOWN Type=3
    09/05/17-19:42:16 (tRAID): WARN: sodLockdownCheck: sodSequence update: CtlrLockdown
    09/05/17-19:42:16 (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
    09/05/17-19:42:16 (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
    09/05/17-19:42:16 (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
    09/05/17-19:42:17 (tRAID): SOD: Code Synchronization Initialization Phase Complete
    09/05/17-19:42:16 (tRAID): WARN: Lockdown case-REV UP drives. NVPS did not update from Sstor.
    09/05/17-19:42:17 (tRAID): SOD: Initialization Phase Complete
    ==============================================
    Title: Disk Array Controller
    Copyright 2008-2011 LSI Logic Corporation, All Rights Reserved.

    Name: RC
    Version: 07.80.41.60
    Date: 08/30/2011
    Time: 18:16:51 CDT
    Models: 2660
    Manager: devmgr.v1080api06.Manager
    ==============================================

    09/05/17-19:42:16 (tRAID): NOTE: SYMbol available
    09/05/17-19:42:16 (tRAID): WARN: *********************************************************
    09/05/17-19:42:16 (tRAID): WARN: *********************************************************
    09/05/17-19:42:16 (tRAID): WARN: ** **
    09/05/17-19:42:16 (tRAID): WARN: ** WARNING!!!!!! **
    09/05/17-19:42:16 (tRAID): WARN: ** **
    09/05/17-19:42:16 (tRAID): WARN: ** This controller is not running the correct firmware **
    09/05/17-19:42:16 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
    09/05/17-19:42:16 (tRAID): WARN: ** In order to prevent any configuration loss, the **
    09/05/17-19:42:16 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
    09/05/17-19:42:16 (tRAID): WARN: ** firmware release. **
    09/05/17-19:42:16 (tRAID): WARN: ** **
    09/05/17-19:42:16 (tRAID): WARN: ** **
    09/05/17-19:42:16 (tRAID): WARN: *********************************************************
    09/05/17-19:42:16 (tRAID): WARN: *********************************************************
    09/05/17-19:42:17 (tRAID): SOD: sodComplete Notification Complete
    09/05/17-19:42:17 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 43 seconds
    09/05/17-19:42:17 (tRAID): sodMain complete
    09/05/17-19:42:17 (tRAID): WARN: SOD lockdown sequence complete - suspending...

    lem
    clear
    lemClearLockdown

  • Hello Omega876,

    So based on what you posted I can see that this the primary controller & it is not booting due to not being able to read the firmware. I am guessing that you are getting the same message on the other controller as well?

    Have you tried to use just one controller & see if it boots fully? If you have and you are getting the same errors then what is needed is to clear the lockdown state.

    In order to clear a lock down state you have to be in secure shell of the controller. Once in secure shell you can run the lemclearlockdown. You will need to know the password for secure shell to access the full menu. Dell doesn’t provide the secure shell password to customers.

    Please let us know if you have any other questions.

    DELL-Sam L
    Dell | Social Outreach Services - Enterprise
    Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)

  • Sam,

    I am only having this issue on this controller. i was able to login to the secure Shell on controller 0 and clear the lockdown. I was also able to clear the config. Now controller 0 is working fine and I was able to add controller 0 MDSM.

    The problem is still with this controller 1. This controller I was also able to login to the secure shell with the username and password. I ran the command lemClearLockdown and the unit begins to clear the lockdown. The issue is that no matter how many times I run this command it stays in lockdown because of the firmware message as you mentioned earlier.

    Name: RC
    Version: 07.80.41.60
    Date: 08/30/2011
    Time: 18:16:51 CDT
    Models: 2660
    Manager: devmgr.v1080api06.Manager
    ==============================================

    09/06/17-19:34:01 (tRAID): NOTE: SYMbol available
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************
    09/06/17-19:34:01 (tRAID): WARN: ** **
    09/06/17-19:34:01 (tRAID): WARN: ** WARNING!!!!!! **
    09/06/17-19:34:01 (tRAID): WARN: ** **
    09/06/17-19:34:01 (tRAID): WARN: ** This controller is not running the correct firmware **
    09/06/17-19:34:01 (tRAID): WARN: ** release. The controller will enter lockdown mode. **
    09/06/17-19:34:01 (tRAID): WARN: ** In order to prevent any configuration loss, the **
    09/06/17-19:34:01 (tRAID): WARN: ** controller must be loaded with x0.00.xx.xx **
    09/06/17-19:34:01 (tRAID): WARN: ** firmware release. **
    09/06/17-19:34:01 (tRAID): WARN: ** **
    09/06/17-19:34:01 (tRAID): WARN: ** **
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************
    09/06/17-19:34:02 (tRAID): SOD: sodComplete Notification Complete
    09/06/17-19:34:02 (tRAID): sodMain CtlrLockdown sequence finished, elapsed time = 19 seconds
    09/06/17-19:34:02 (tRAID): sodMain complete
    09/06/17-19:34:01 (tRAID): WARN: SOD lockdown sequence complete - suspending...
    Name: RC
    Version: 07.80.41.60
    Date: 08/30/2011
    Time: 18:16:51 CDT
    Models: 2660
    Manager: devmgr.v1080api06.Manager
    ==============================================

    09/06/17-19:34:01 (tRAID): NOTE: SYMbol available
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************
    09/06/17-19:34:01 (tRAID): WARN: *********************************************************

    My question: Is there a way to update the firmware manually without the controller being added to MDSM? If not, what are my options for fixing this?

    Thanks!

  • Hello Omega876,

    It looks like your controller could be set to simplex mode. What you will need to do is to run the following command from controller in slot 0 to put your MD3200 in duplex mode. From SMCLI run the following

    SMcli.exe -n ArrayName -c “set storageArray redundancyMode=duplex;

    Once the command finishes running then you will want to hot add the controller in slot 1 & it should bring it online.

    If you can’t run the command from SMCLI then you can run the following from the serial port on controller 0.

    setSimplexMode_MT 1

    Please let us know if you have any other questions.

    DELL-Sam L
    Dell | Social Outreach Services - Enterprise
    Download the Dell Quick Resource Locator app today to access PowerEdge support content on your mobile device! (iOS, Android, Windows)