Hi Sumit,
Here is the sshd server side output with logging set to DEBUG3 after re-running client
and server over custom port 22222:
[root@testclient2 ~]# /usr/sbin/sshd -ddd -p 22222
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 920
debug2: parse_server_config: config /etc/ssh/sshd_config len 920
debug3: /etc/ssh/sshd_config:19 setting AddressFamily inet
debug3: /etc/ssh/sshd_config:29 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:31 setting HostKey /etc/ssh/ssh_host_ecdsa_key
debug3: /etc/ssh/sshd_config:32 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: /etc/ssh/sshd_config:44 setting SyslogFacility AUTHPRIV
debug3: /etc/ssh/sshd_config:45 setting LogLevel DEBUG3
debug3: /etc/ssh/sshd_config:60 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: /etc/ssh/sshd_config:80 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:94 setting GSSAPIAuthentication yes
debug3: /etc/ssh/sshd_config:95 setting GSSAPICleanupCredentials yes
debug3: /etc/ssh/sshd_config:116 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:124 setting UsePrivilegeSeparation sandbox
debug3: /etc/ssh/sshd_config:141 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME
LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:142 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS
LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:143 setting AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
debug3: /etc/ssh/sshd_config:144 setting AcceptEnv XMODIFIERS
debug3: /etc/ssh/sshd_config:147 setting Subsystem sftp /usr/libexec/openssh/sftp-server
debug3: /etc/ssh/sshd_config:155 setting KerberosAuthentication yes
debug3: /etc/ssh/sshd_config:156 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:157 setting UsePAM yes
debug3: /etc/ssh/sshd_config:158 setting AuthorizedKeysCommand
/usr/bin/sss_ssh_authorizedkeys
debug3: /etc/ssh/sshd_config:159 setting ChallengeResponseAuthentication yes
debug3: /etc/ssh/sshd_config:160 setting AuthorizedKeysCommandUser nobody
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:rdY76f1jebuS5sQfPVqorId1fy+yuIUMcGIVl5KZdXg
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:04F5INVrYOenS32GxvfsU8o8z70Fta/J7alUuSG+KpE
debug1: private host key #2: ssh-ed25519
SHA256:nC7e/CMtGFYlpWX/3W7/DdeDlL+v6rBHVAVQYpK/YOU
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='22222'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22222 on 0.0.0.0.
Server listening on 0.0.0.0 port 22222.
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 920
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 129.57.228.15 port 54834 on 129.57.228.121 port 22222
debug1: Client protocol version 2.0; client software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 8491
debug3: preauth child monitor started
debug1: SELinux support enabled [preauth]
debug3: ssh_selinux_change_context: setting context from
'unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' to
'unconfined_u:unconfined_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
debug3: ssh_selinux_change_context: setcon
unconfined_u:unconfined_r:sshd_net_t:s0-s0:c0.c1023 from
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 failed with Invalid argument
[preauth]
debug3: privsep user:group 74:74 [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: list_hostkey_types:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
[preauth]
debug2: host key algorithms:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
[preauth]
debug2: ciphers stoc:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
[preauth]
debug2: MACs ctos:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[preauth]
debug2: MACs stoc:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[preauth]
debug2: compression ctos: none,zlib(a)openssh.com [preauth]
debug2: compression stoc: none,zlib(a)openssh.com [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
[preauth]
debug2: host key algorithms:
ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01(a)openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
[preauth]
debug2: ciphers ctos:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm(a)openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
[preauth]
debug2: ciphers stoc:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm(a)openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
[preauth]
debug2: MACs ctos:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[preauth]
debug2: MACs stoc:
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[preauth]
debug2: compression ctos: none,zlib(a)openssh.com,zlib [preauth]
debug2: compression stoc: none,zlib(a)openssh.com,zlib [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305(a)openssh.com MAC: <implicit>
compression: none [preauth]
debug1: kex: server->client cipher: chacha20-poly1305(a)openssh.com MAC: <implicit>
compression: none [preauth]
debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
debug3: mm_request_send entering: type 120 [preauth]
debug3: mm_request_receive_expect entering: type 121 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 120
debug3: mm_request_send entering: type 121
debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
debug3: mm_request_send entering: type 120 [preauth]
debug3: mm_request_receive_expect entering: type 121 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 120
debug3: mm_request_send entering: type 121
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug3: receive packet: type 30 [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0x5590367586d0(100)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 31 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: send packet: type 7 [preauth]
debug3: receive packet: type 21 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user testuser service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug3: Trying to reverse map address 129.57.228.15.
debug2: parse_server_config: config reprocess config len 920
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for testuser [preauth]
debug3: mm_start_pam entering [preauth]
debug3: mm_request_send entering: type 100 [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug3: mm_inform_authrole entering [preauth]
debug3: mm_request_send entering: type 80 [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: userauth_finish: failure partial=0 next
methods="publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive"
[preauth]
debug3: send packet: type 51 [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 100
debug1: PAM: initializing for "testuser"
debug1: PAM: setting PAM_RHOST to "devclient1.acc.jlab.org"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 100 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 80
debug3: mm_answer_authrole: role=
debug2: monitor_read: 80 used once, disabling now
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user testuser service ssh-connection method gssapi-with-mic
[preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method gssapi-with-mic [preauth]
debug3: mm_request_send entering: type 42 [preauth]
debug3: mm_request_receive_expect entering: type 43 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 42
debug1: Unspecified GSS failure. Minor code may provide more information
No key table entry found matching host/testclient2@
debug3: mm_request_send entering: type 43
debug3: userauth_finish: failure partial=0 next
methods="publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive"
[preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user testuser service ssh-connection method gssapi-with-mic
[preauth]
debug1: attempt 2 failures 0 [preauth]
debug2: input_userauth_request: try method gssapi-with-mic [preauth]
debug3: userauth_finish: failure partial=0 next
methods="publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive"
[preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user testuser service ssh-connection method
keyboard-interactive [preauth]
debug1: attempt 3 failures 0 [preauth]
debug2: input_userauth_request: try method keyboard-interactive [preauth]
debug1: keyboard-interactive devs [preauth]
debug1: auth2_challenge: user=testuser devs= [preauth]
debug1: kbdint_alloc: devices 'pam' [preauth]
debug2: auth2_challenge_start: devices pam [preauth]
debug2: kbdint_next_device: devices <empty> [preauth]
debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
debug3: mm_sshpam_init_ctx [preauth]
debug3: mm_request_send entering: type 104 [preauth]
debug3: mm_sshpam_init_ctx: waiting for MONITOR_ANS_PAM_INIT_CTX [preauth]
debug3: mm_request_receive_expect entering: type 105 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 104
debug3: mm_answer_pam_init_ctx
debug3: PAM: sshpam_init_ctx entering
debug3: mm_request_send entering: type 105
debug2: monitor_read: 104 used once, disabling now
debug3: mm_sshpam_query [preauth]
debug3: mm_request_send entering: type 106 [preauth]
debug3: mm_sshpam_query: waiting for MONITOR_ANS_PAM_QUERY [preauth]
debug3: mm_request_receive_expect entering: type 107 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 106
debug3: mm_answer_pam_query
debug3: PAM: sshpam_query entering
debug3: ssh_msg_recv entering
debug3: PAM: sshpam_thread_conv entering, 1 messages
debug3: ssh_msg_send: type 1
debug3: ssh_msg_recv entering
debug3: mm_request_send entering: type 107
debug3: mm_sshpam_query: pam_query returned 0 [preauth]
debug3: send packet: type 60 [preauth]
Postponed keyboard-interactive for testuser from 129.57.228.15 port 54834 ssh2 [preauth]
The important part of logging from the server side is probably the lines:
debug1: Unspecified GSS failure. Minor code may provide more information
No key table entry found matching host/testclient2@
debug3: mm_request_send entering: type 43
debug3: userauth_finish: failure partial=0 next
methods="publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive"
[preauth]
debug3: send packet: type 51 [preauth]
Compare this with the client output where the important part of logging from the client
side is probably the gssapi-with-mic section showing packet responses from server of type
51 (FAILURE). The message "we did not send a packet, disable method" is
probably noteworthy too:
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password