SSH Sessions repeatedly hang on 3348.

Network Switches

Network Switches
Information and ideas on Dell PowerConnect network switch solutions.

SSH Sessions repeatedly hang on 3348.

  • There appears to be a bug in the 1.0.0.106 firmware. I've repeated it on more than one identical switch.

    Sometimes when doing a "show spanning-tree" the session will drop/hang open on the switch.

    After this has happened once, and you've reconnected, many other commands will make it freeze (usually immediately) just pressing '?' will hang it.

    Of course after having 5 sessions hung, you can't connect again.


    excerpts from a log I made follow:

    s1-c8# show span


    Spanning tree enabled mode MSTP
    Default port cost method: short
    ... etc...

    /***Typed the same command several more times***/

    s1-c8# show span


    Spanning tree enabled mode MSTP
    Default port cost method: short


    Gathering information ..........Read from remote host s1-c8: Connection reset by peer
    Connection to s1-c8 closed.

    nick@gorf:~$ ssh s1-c8

    01-Jan-2000 01:14:28 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED
    01-Jan-2000 01:14:28 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED

    s1-c8> en
    Password:********
    01-Jan-2000 01:15:01 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED
    01-Jan-2000 01:15:01 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED

    s1-c8# show span /*** HUNG ***/



    *** same thing a couple more times ***

    nick@gorf:~$ ssh s1-c8


    01-Jan-2000 01:15:59 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED
    01-Jan-2000 01:15:59 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED
    01-Jan-2000 01:15:59 %AAA-I-CONNECT: User CLI session for user unKnown over ssh , source 204.174.223.110
    destination 10.100.32.71 ACCEPTED

    s1-c8> /*** This time I just pressed ? and it hung ***/

    nick@gorf:~$ ssh s1-c8
    Connection to s1-c8 closed.
    nick@gorf:~$ ssh s1-c8
    Connection to s1-c8 closed.
    nick@gorf:~$

    /*** Now there's no more free sessions ***/
  • I'm not familiar with version 1.0.0.106 on a 3348.  Do you mean 3448 or really 3348?  If you are using 3348 then that FW looks to be very old.  You might consider upgrading to the latest version 1.2.0.6 (go to support.dell.com).  Also there is a newer version of the 3448 FW too.  The latest version for 3448 is 1.0.0.112.
     
    I have not seen this problem before and neither has my colleagues.  You might try upgrading to the latest FW to see if it helps.
     
    Cuong.
  • Sorry, you're right. It is a 3448.
  • Still does the same thing with 112
  • Does it happens only on an SSH session or does it happens also when you simply telnet into the switch?  Also does it happens only when you have all five SSH sessions or can it happens when only one SSH session is active.  How easily is it to reproduce?  Do you have to do "show spanning" repeatedly over many times to reproduce it?

    Any info that might help me reproduce the problem is appreciated.

  • It happened to me while using SSH, though I haven't tested it with either the serial console or telnet.

    Reproducing it is quite easy.

    It doesn't seem to be consistent in the number of times you have to show spanning-tree. The first time it happened was while configuring some spanning tree settings, so in that case, it was only run a few times between other commands. When I first tried to reproduce it it took several tries (maybe 5-6), though sometimes it may take 20 or 30. You should be able to reproduce it easily by just repeating the command many times.

    If you don't have any luck with that I can send you my config, though I don't know that that will be necessary.
  • I have tried with both SSH and regular telnet sessions.  I have executed the show spanning tree operation more then 50 times on each type of sessions.  I also tried with multiple telnet & SSH sessions active.  I'm using SSH on cygwin and on Linux.  No luck reproducing it.

    Can you give me more information please.  When you said it locked up on you, does it hangs only the given SSH session so that you can access it with another SSH session?  If so maybe something to do with your particular SSH tool.  What are you using?  Is it SSH that comes with Linux or something on windows?

    Does the console show any message when a session "locks up" on you?  Any other observation that might help?

    Cuong.

  • I think I figured out the problem.  It seems that some SSH clients on windows will seem to lock up when use with the switch.  It maybe that the performance on these clients are so slow that they look as if they lock up when executing commands that result in a lot of data being returned (such as "show spanning").

    I have downloaded and tried a number of different SSH clients on Windows and I find that there are really three types:

    • SSH clients based on OpenSSH which requires cygwin - this is what I used normally and these works fine.
    • SSH clients based on Putty.  These seems to have some performance problems when connected to the switch.  There are a number of these types of clients all based on Putty including the one in "TTermpro" (look here for some information on these types of clients: http://www.openssh.com/windows.html).
    • Alternatively you can also use SSHClient which is not based on OpenSSH or Putty and seems to have the best performance of all SSH clients available on Windows.  See here for information on this client: http://ecs.rutgers.edu/ssh_windows.php3.  The latest version of this client is SSHSecureShellClient version 3.2.9 found here (ftp://sunsite.unc.edu/pub/packages/security/ssh/).

    Also when these slower clients are killed they do not release the session which cause the problems you notice where the sessions are locked open.  In fact if you are really patience these clients do eventually response just take a VERY long time.

    I suggest you use the OpenSSH based clients on windows or the SSHSecureShellClient for better performance.

    Cuong.

  • I have been using OpenSSH in linux on more than one seperate machine.

    "OpenSSH_3.8.1p1 Debian-8.sarge.4, OpenSSL 0.9.7g 11 Apr 2005"


    The first time the problem occurs the session disconnects "Connection reset by peer" as in the log in my original message.

    Subsequent attempts to do show span (or many other commands) will hang the session. (no response within several minutes) the session seems to have hung completely and stays open on the switch even after killing my client and connecting again.

    Here's the config, incase that's helpful.

    As I mentioned before, I've tried it on 2 switches with the this config, and both had the same problem.




    nick@gorf:~$ ssh s1-c8


    s1-c8> en
    Password:********

    s1-c8# show running-config
    spanning-tree mode mstp
    interface range ethernet g(3-4)
    switchport mode trunk
    exit
    vlan database
    vlan 30,50,70
    exit
    interface range ethernet e(1-48)
    switchport access vlan 50
    exit
    interface range ethernet g(3-4)
    switchport trunk allowed vlan add 50
    exit
    interface range ethernet g(3-4)
    switchport trunk allowed vlan add 70
    exit
    interface vlan 30
    name colo
    exit
    interface vlan 50
    name colo2
    exit
    More: , Quit: q, One line: interface vlan 70
    name management
    exit
    interface vlan 70
    ip address 10.100.32.71 255.255.255.0
    exit
    ip default-gateway 10.100.32.254
    hostname s1-c8
    line console
    autobaud
    exit
    line console
    speed 115200
    exit
    aaa authentication login default none
    line ssh
    password 00000000000000000000000000000000 encrypted
    exit
    enable password level 15 00000000000000000000000000000000 encrypted
    username \ password 00000000000000000000000000000000 encrypted
    ip ssh server
    snmp-server community Dell_Network_Manager rw view DefaultSuper
    More: , Quit: q, One line:





    Default settings:
    Service tag: GYWYK71

    SW version 1.0.0.106 (date 01-Jun-2005 time 15:26:21)

    Fast Ethernet Ports
    ==========================
    no shutdown
    speed 100
    duplex full
    negotiation
    flow-control off
    mdix auto
    no back-pressure

    Gigabit Ethernet Ports
    More: , Quit: q, One line: =============================
    no shutdown
    speed 1000
    duplex full
    negotiation
    flow-control off
    mdix auto
    no back-pressure

    interface vlan 1
    interface port-channel 1 - 8

    spanning-tree
    spanning-tree mode STP

    qos basic
    gorf%
  • Ok.  Let me try this config on my switch and I will let you know.  However, I have been unable to reproduce your problem on my current system using various different SSH implementation on Linux including OpenSSH.  Maybe its related to your configuration.  Anyway, I will load your config and try it.

    Also, have you tried it with just telnet just to eliminate the SSH issue?

    Cuong.

  • I've loaded a configuration very similar to yours.  The difference is that I'm using VLAN1 for management and I adjusted the IP address to fit my network, and of course I changed the login password.  Otherwise the configuration is very similar to what you have.  I have executed the "show span" on this switch for more then 50 times using both SSHClient on windows and OpenSSH on Linux.  I have not had any problem.  I also tried using telnet with similar result.

    When you run this, does it seems slow on any other command?  Meaning that do you experience a general slowness that might be attributable to having to encrypt/decrypt using SSH?  Or do you normally see everything working with good performance and experiencing a problem only on the "show span" command?  Also does it "lock up" when it is "gathering information..." to display the spanning tree or in the middle of dumping the data to the screen?

    Can you attach a console to the serial port of the switch while executing your command to see if there is any messages printed to the console that might help to diagnose this problem.

    At this point I'm really at a lost.  I cannot reproduce the problem on my network and I'm not sure what you are seeing.  I can't figure out whether the problem you are experiencing is SSH related or something else.  If you could reproduce it using only telnet then it might help to eliminate SSH as the possibility.

    Cuong.

  • if you look back at my original post, you can see the answers to most of your questions.

    As far as me doing testing on the console or with telnet, I think you should be able to handle that given the information at hand.

    I'm not sure what you could be doing so differently from me that would prevent you from reproducing the problem as both myself and a co-worker of mine independantly ran into this, both using our own computers, with nothing common between them other than perhaps a similar config.

    Thanks.
    -Nick
  • I tried duplicating the issue using a 3448 that provides link to all of our lab systems (different lab than Cuoung) and was also unable to find a problem. I issued the "show span" (using both "terminal datadump" and "no terminal datadump" 100 times and experienced no lockups. I also tried hopping through more than one SSH tunnel by using PuTTY to connect from my Win2k3 server to my RHEL4 U1 server and then SSHing into the switch, but saw no difference.

    3448 fw - 1.0.0.112
    openssh-3.9p1-8 (RHEL4 U1)
    3448 uptime - 24 days

    Is the problem intermittant? If you reboot the 3448 does it appear to work for an unknown amount of time? What is the uptime for one of the switches on which you've seen this problem?

     

  • The switch was a little more stubborn this time, but still broke.

    logs are at http://arx.ca/dellswitch/

    The order things were done, just to clarify.

    ssh.log, where I try to config the switch, but my editor won't let me open my terminal prog in its record mode.

    minicom.log So, I open the terminal in another window, do the basic config STUFF (jeeze your filters are picky. Maybe they should just turn **** to stars rather than making me fix it) to be able to login via ssh. (This is a brand new switch, never used)

    ssh2.log I was getting tired of waiting for "gathering information....." all the time, since the switch was being more stubborn than usual, so I opened another session. good timing I suppose as it broke almost immediately. (note that I'm pretty sure multiple sessions are NOT required to break it)

    It took about 15 minutes for me to break it this time, though it's usually not that stubborn.

    all 3 files were left open, so you can see the connections (and disconnections) on the minicom window while the ssh stuff is breaking.

    don't worry about all the ^H and stuff... I can't type. Clarified some of the more confusing parts, but I'm pretty lazy, and have more important things to do right now.

    l8rs

    -Nick

    Message Edited by nickw- on 10-14-2005 11:51 AM