I’m having a really weird problem where when I connect to an SSH server (that has a pretty standard default configuration, including a default MaxAuthTries of 6) using Net::SSH, the connection fails after 3 authentication attempts with a disconnected: Too many authentication failures (2) (Net::SSH::Disconnect) exception while the OpenSSH command line client connects fine.

The problem is that my SSH Agent already knows about 4 identities and offers them automatically – annoyingly before the identity that is configured in the SSH client configuration file for that server – and so the OpenSSH client always succeeds to connect and the Net::SSH client always fails to connect. I can get the Net::SSH client to connect by disabling the SSH Agent.

My setup:

$ ssh-add -l
2048 SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU /home/odeda/.ssh/id_rsa (RSA)
3072 SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao odeda@vesho (RSA)
2048 SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM odeda@jior (RSA)
1024 SHA256:EKBzRjDE6twnRLlIT3IHEfu08tVXCSK1henMHB+s+p8  (DSA)
$ grep example.com ~/.ssh/config -A1
Host *.example.com
   IdentityFile ~/.ssh/my_example_identity.pem

Running Net::SSH

$ bundle exec ruby -rnet/ssh -e 'Net::SSH.start("server.example.com","ubuntu",{
  host_name:"3.456.789.12",
  verify_host_key: :never,
  user_known_hosts_file: "/dev/null",
  non_interactive: true}) do |ssh|
    puts "Connected";
    puts ssh.exec! "echo Hello";
    ssh.loop(3);
  end'
~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:198:in `block in poll_message':
    disconnected: Too many authentication failures (2) (Net::SSH::Disconnect)
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:190:in `loop'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:190:in `poll_message'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:175:in `next_message'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:103:in `block in next_message'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:102:in `loop'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:102:in `next_message'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:50:in `authenticate_with_alg'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:100:in `block in authenticate_with'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:97:in `each'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:97:in `authenticate_with'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:19:in `block in authenticate'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:132:in `block in each_identity'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:124:in `each'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:124:in `each_identity'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:18:in `authenticate'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:88:in `block in authenticate'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:72:in `each'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:72:in `authenticate'
        from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh.rb:262:in `start'
        from -e:1:in `<main>'

Running OpenSSH client

$ ssh -oHostname=3.456.789.12 -l ubuntu server.example.com -- echo Hello
Hello

Running Net::SSH without the SSH agent

$ SSH_AUTH_SOCK= bundle exec ruby -rnet/ssh -e 'Net::SSH.start("server.example.com","ubuntu",{
  host_name:"3.456.789.12",
  verify_host_key: :never,
  user_known_hosts_file: "/dev/null",
  non_interactive: true}) do |ssh|
    puts "Connected";
    puts ssh.exec! "echo Hello";
    ssh.loop(3);
  end'
Connected
Hello

Server logs

It gets weirder if we look at the server logs:

This is the OpenSSH client connection:

Jul 03 07:50:44 172-24-51-231.us-east-2.example.com sshd[42437]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
  userauth_pubkey: key type ssh-dss not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
  Accepted publickey for ubuntu from 213.546.879.21 port 47034 ssh2: RSA SHA256:6Azd3HzkKpBLJ+zsolTOpksL4XX8aNdGWtprUwRvsoY
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
  pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)

And this is the Net::SSH connection:

Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU failed, status 22
Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
  userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao failed, status 22
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
  userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
  AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM failed, status 22
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
  userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
  error: maximum authentication attempts exceeded for ubuntu from 213.546.879.21 port 56414 ssh2 [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
  Disconnecting authenticating user ubuntu 213.546.879.21 port 56414: Too many authentication failures [preauth]

So we can see that the 4th key I have in my agent – which is a DSA key that is no longer supported by default on modern servers – is sent by the OpenSSH client and rejected because of PubkeyAcceptedAlgorithms and then the 5th public key (from the config file) is accepted. In the case of the Net::SSH client, the 3 RSA keys from the agent – same keys as before – are rejected by PubkeyAcceptedAlgorithms and the fourth isn’t sent – instead we get error: maximum authentication attempts exceeded after only 3 tries.

I tried to find a configuration option in Net::SSH (or OpenSSH client configuration) that maybe controls some kind of feature (that I don’t know exists) where the client tells the server how many authentication attempts to allow – but of course I couldn’t find such because it makes no sense.

I of course tries to set the Net::SSH option number_of_password_prompts to something larger than the default of 3 and of course it didn’t do anything because that option is for password prompts, which we are not doing at all.

Any idea will be appreciated, TIA!