SSH Keys problem?

[lucid]

I've been working on this stupid thing for the better part of a day. I can't seem to connect to my server if I use key authentication only. The error I'm getting is simply:

Code: [Select]

Permission denied (publickey).

Now, here's the details. I'm using a user on my client made specifically for SSH. This user's name is shell. The user on the server is also named shell. Hopefully that doesn't confuse you. Here's the server's sshd_config:

Code: [Select]

# $OpenBSD: sshd_config,v 1.93 2014/01/10 05:59:19 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

AllowUsers shell
Port 12222
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# The default requires explicit activation of protocol 1
#Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Ciphers and keying
#RekeyLimit default none

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile ~/.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
PermitEmptyPasswords no

# Change to no to disable s/key passwords
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no # pam does that
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
UsePrivilegeSeparation sandbox # Default for new installations.
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS yes
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
Banner /etc/issue

# override default of no subsystems
Subsystem sftp /usr/lib/ssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server

And the client's ssh_config:

Code: [Select]

# $OpenBSD: ssh_config,v 1.27 2013/05/16 02:00:34 dtucker Exp $

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

# Host *
User shell
#   ForwardAgent no
#   ForwardX11 no
#   RhostsRSAAuthentication no
   PubKeyAuthentication yes
   RSAAuthentication yes
#   PasswordAuthentication no
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h

Now, I've copied the client's public key to the authorized_keys file on the server using ssh-copy-id. I've also tried using scp. Makes no difference. I have both the pub and private key on the client stored in the ssh user's(shell) .ssh directory. I've tried changing the permissions of the server's authorized_keys file to everything from 644 up to 777. Makes no difference. I've also tried explicitly defining the path to the client's private key when I connect using:

Code: [Select]

ssh -i /home/shell/.ssh/id_rsa -p 12222 shell@ipaddress
Still, I get the same error. Here's some more details:

ssh -v -p 12222 shell@ipaddress

Code: [Select]

OpenSSH_6.3, OpenSSL 1.0.1g 7 Apr 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 192.168.0.7 [192.168.0.7] port 13409.
debug1: Connection established.
debug1: identity file /home/shell/.ssh/id_rsa type 1
debug1: identity file /home/shell/.ssh/id_rsa-cert type -1
debug1: identity file /home/shell/.ssh/id_rsa type 1
debug1: identity file /home/shell/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:5e:41:a6:fa:f5:ac:8f:ab:23:ea:aa:ca:71:4a:65
debug1: Host '[192.168.0.7]:13409' is known and matches the ECDSA host key.
debug1: Found key in /home/shell/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
                    .---,.--.                       
                   ////;///\\\.                     
                  ////;////\\\\\                     
                 |||/``_..`.\\\\\
                 \\\-_..-,  \\\\\'
                  \\`-|_"    \\\\'                   
                   \\\ ,    ///|\\                   
                   _||\  _.'//////                   
                .-'///'"" |//////__                 
               /  //'.`--..////'.' ".               
     .-=-.    /_...--._""--//""._    .               
    +     \   J  ;.    "-..---..  J   \             
   /       \  |  lo)    . .     ".F    \             
  /         \ F  `.    .   .    (o:_.-=='           
 /           \F  J;"--"    :.    .'  |             
f        ;    \ J|:::  .:; ::;--"|   |         
        /      \FF::  ::::. ::;/ |   |         
      ,J.       Y:'  :::::'   /  |   |         
     /  F        .   ''_    ./   |.-""-.....---"""-.
   ,'   J.        .        .:    /,;:\\\:.          Y
  /      Into       .       ;_..-(/ |:\\\\"'         J
 /       J         .  ..  ;'       `' `'`'  .'      '
/        :/)      .: .%' The              _/       /
         `:_--,   ' ,.";'             _.='/       / 
          `:_"_'-, ///Tunnel     _..-""  /       /   
         .-'`---''-""-.___..---"""      J       /   
      .-'              ,',' |           F      /     
      ;---------------','   |          j'     /     
      |---------------|    ,'          F     /       
      |               |  ,'           j'    /       
      |_______________|,'             F____/           


debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/shell/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /home/shell/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
As you can see, that sexy lady is my banner. Try not to beat it too hard. I honestly don't see what went wrong based on the verbosity. Oh, and one more thing. I generated the key pair like so:

Code: [Select]

ssh-keygen -t rsa -b 4096 -C "$(whoami)@$(hostname)-$(date -I)"

I generated it using the ssh user. That's about all of the information I can muster. Hopefully I'm just being stupid.

EDIT: I also just tried clearing the authorized_keys file and starting over with a new public key. This time, I used ecdsa and didn't specify any size. Once again copied id_ecdsa.pub to the authorized_keys file on the server and double checked. Still getting the same exact problem. I also removed all lines specifying a particular user in any of the configs. Yes, I restarted the server after making changes. No dice.

This is supposed to be easy. I've done this plenty of times in the past without any trouble.

[voodoo]

Your pristine config files make me feel like this is a permissions issue.  Can you post the output of the following?

Code: [Select]

ls -al ~/.ssh

[lucid]

Client:

Code: [Select]

drwx------ 2 shell sshd 4096 Jun  9 21:53 .
drwx--x--x 3 shell sshd 4096 Jun  9 21:18 ..
-rw------- 1 shell sshd  314 Jun  9 21:47 id_ecdsa
-rw-r--r-- 1 shell sshd  188 Jun  9 21:47 id_ecdsa.pub
-rwx------ 1 shell sshd  181 Jun  9 19:20 known_hosts

Server:

Code: [Select]

drwxr-xr-x  2 shell shell 4096 Jun  9 21:53 .
drwxrwxrwx 41 shell shell 4096 Jun 10 12:24 ..
-rwxrwxrwx  1 shell shell  188 Jun  9 21:53 authorized_keys
-rw-r--r--  1 shell shell  181 Jun  9 21:11 known_hosts
As you can see the last thing I tried was to grant 777 permissions to authorized_keys on the server, but as mentioned before, I've tried everything from 644 up to 755.

[Architect]

Setting 777? Why on Earth would anybody do this?

Change .ssh/ permission to 700 on both boxes
Change .ssh/authorized_keys* to 640 on both boxes
Chown ssh_config's to shell on both boxes

[lucid]

Setting 777? Why on Earth would anybody do this?

Just testing. I would never actually leave it this way. Thanks, I will try what you said.

EDIT: Ok, well, I only chmodded authorized_keys on the server only because the client has no authorized_keys file. Considering that no one will be connecting to the client, I don't see why I would need one. I changed the owner of /etc/ssh/ssh_config to the user on both boxes(both named shell), and changed both .ssh directories to 700. Still getting the same error. I also chowned sshd_config as well. No luck.

I don't get it. I feel like I've literally fixed everything in existence that could possibly cause a problem with SSH.

[Architect]

Well that would be the correct (secure) configuration for the keys and whatnot, but I don't see why it's denying access either.

[ReX]

Hmm, is it possible to see the banlist? Maybe your server/shell as banned your ip/isp range,
I know on some servers/shells after a few times trying to get on with pw or ssh keys, The firewall automaticly does the clever business.


Hopefully its a firewall issue, As i see nothing wrong with your settings.


ReX

[lucid]

Well that would be the correct (secure) configuration for the keys and whatnot, but I don't see why it's denying access either.

It's literally a mystery. The only thing that would make any sense is that it has something to do with the fact that I created a new user(shell), on the client specifically for SSH only. On my client(Slackware box) I normally run as root because I like to live on the edge. So I've created users specifically for certain tasks, like SSH.

On the server(Arch), I don't generally run as root, although I used to. I have a user(also called shell), that I use for every day tasks the way one is supposed to with linux.

Although, after chowning the configs I don't see how having created a new user for SSH would be a problem. I could start over(for the 1,000 time) and clear the authorized_keys file on the server, and generate a new key pair and copy the public key over once again, but I don't think it'll make a difference.

EDIT: I'm looking through the verbose output when trying to connect, and for the most part everything looks fine. I see that it successfully finds the key, it matches and the signature is correct, the keys are sent and received, but then right after it shows my server's banner, I see some output which is questionable, yet I don't really know what it means:

Code: [Select]

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering ECDSA public key: /home/shell/.ssh/id_ecdsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
Then that's when it fails.

[lucid]

Hmm, is it possible to see the banlist? Maybe your server/shell as banned your ip/isp range,
I know on some servers/shells after a few times trying to get on with pw or ssh keys, The firewall automaticly does the clever business.


Hopefully its a firewall issue, As i see nothing wrong with your settings.


ReX

I have no firewall enabled. I disabled iptables the moment this problem started. Looking at the logs using:

Code: [Select]

journalctl -u sshd | tail -100
Because Arch is a weird hipster distro with no auth.log or anything like that, I don't even see anything pertaining to my recent failures. In fact, the last log was Jan 3 .

[ReX]

Hi lucid,
Forgot to refresh my page to see your reply, But never the less.


I've posted you a private message on the IRC server, hopefully one of those methods will help you with your problem, If any of them work, Let me know and ill post it up here or you can.


Atleast if anybody else suffers from the same problem, The fix is there


ReX

[lucid]

Ok, I tried yet again. Let's start from the ground up.

On client:

~/.ssh perms: 700
/etc/ssh/sshd_config owner: shell
/etc/ssh/ssh_config owner: shell

On Server:

~/.ssh perms: 700
~/.ssh/authorized_keys perms: 640
/etc/ssh/sshd_config: shell
/etc/ssh/ssh_config owner: shell

I've created a brand new rsa key with absolute defaults. All I did this time was run ssh-keygen. No special options. Then I cleared the authorized_keys file and started over. I copied the default public key to the authorized keys file on the server. Double-checked, and they match. PubKeyAuth and RSAKeyAuth are enabled on the server. The ports match up. It is indeed using .ssh/authorized_keys as it's file. Once done, I disabled password auth and restarted the server. I attempted to connect, but I get the same. damn. mother. fucking. error. No firewalls are enabled. I can successfully  connect and sign in to the server if password authentication is enabled.

Does the known_hosts file have anything to do with this? When I first connected it gave me this:

Code: [Select]

The authenticity of host '[192.168.0.7]:13409 ([192.168.0.7]:13409)' can't be established.
ECDSA key fingerprint is 75:5e:41:a6:fa:f5:ac:8f:ab:23:ea:aa:ca:71:4a:65.
Are you sure you want to continue connecting (yes/no)?
I of course said yes, and then it connected me(when password auth was enabled). Once I did that and connected for the first time, I find this in my known_hosts file on both client and server:

Code: [Select]

[192.168.0.7]:12222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6Pjif7KYLW1YIZFRjrgRmuuUrs0yCz8wx+uGQMSF4X0rxsJ/PHEod5o7b9kvlzDdfe0hEIP9GHvjriRINcgnQ=
This matches nothing else I have. None of my keys, and nothing in the authorized_keys file. In my experience, connecting to a new server for the first time always does this. It's never caused any issues before. So it's a complete shot in the dark, but I can't think of literally anything else.

EDIT: I keep going back to this in the verbose output:

Code: [Select]

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/shell/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/shell/.ssh/id_dsa
debug3: no such identity: /home/shell/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/shell/.ssh/id_ecdsa
debug3: no such identity: /home/shell/.ssh/id_ecdsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
It tries all three options if I specify no identity file in /etc/ssh/ssh_config. Otherwise it does this:

Code: [Select]

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/shell/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
It tries the private key, but then it just keeps trying and then fails. I've triple-checked the client's ssh_config file and whether or not I specify an IdentityFile(~/.ssh/id_rsa), it does one of the two above. I've also chowned the file to shell about 10 times. This is impossible.

I don't normally think this way but it actually seems like it just doesn't work, for literally no reason at all. I've done everything and redone it again like 10 times, and as you all say, everything seems correct. It just doesn't work for no reason.

[Architect]

Well shit. Maybe it's broke because science..

Seriously though, I have no other explanation than it could be a simple config error in ssh[d]_config.

[proxx]

Hi Sis, joining the party(better late than never)
Can you post the full paths of the authorized_keys file and the key  path of the client?
Do both machines assume the same username ?
I suspect you did this as root but I have no way of knowing.
Maybe I am overstating the obvious but since its not entirely clear from the conversation above; The authorized_keys file should be in the home folder of the user you want to login as.
For now thats the best I can come up with since its simply saying your key sucks.
You can try nazi-level logging on the daemon which will actually tell you what the fuck is going on since this is obviously not advertised to the client, ssh -v will only get you so far.

[techb]

This might be stupid, but if your still on Arch have you done a -Syu in a while? It fixes ALOT of issues I've ever had with anything. Except when they decided to change up the install locations of shit like /bin and all.

Might be a crap suggestion, but I know I've gone at least two months without an update and reboot.

[lucid]

Full path to keys

Code: [Select]

/home/shell/.ssh/id_rsa*

Also, there's no authorized_keys on the client, but on the server it's the same because the users have the same name

Code: [Select]

/home/shell/.ssh/authorized_keys
And I actually just updated. I'm really starting to think it's a problem with Arch. Thinking about it, I believe I've had the same problem on the same Arch box and never solved it due to ADD.