LinuxQuestions.org - Cannot enable SSH on Google Cloud downloaded centos vmdk on VirtualBox

- Linux - Virtualization and Cloud (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/)

- - Cannot enable SSH on Google Cloud downloaded centos vmdk on VirtualBox (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/cannot-enable-ssh-on-google-cloud-downloaded-centos-vmdk-on-virtualbox-4175682963/)

Cannot enable SSH on Google Cloud downloaded centos vmdk on VirtualBox

Hi,

I am facing an strange issue. Recently, I have downloaded a Google Cloud based CentOS vmdk from my G-Cloud and tried using it on VirtualBox. The plan is, I want to shift that VM to another cloud, Oracle Cloud.

The instance has an application running, based on NGINX, MySQL and CodeIgniter v2, as well as postfix is in use.

All is good on the Google Cloud, but when the same instance is running in VirtualBox, the application is working fine, but I cannot enable SSH at all. Service is enabled, but SSH cannot start. I tried running "sshd -t" but no error returned.

I am attaching some required screenshot, from the grub parameter of the boot kernel to sshd config file, netstat and ssh service status. uploaded on my organization OneDrive folder, link given below.

Please help how I can enable the SSH on the downloaded VM, if I can enable the SSH on downloaded VM, then I can shift the VM to another cloud. I tried uploading the VM to Oracle Cloud, and SSH does not work, but no issue with SSH on original Google Cloud instance.

https://rapplesentertainment-my.shar...cK2WQ?e=E2sYw0

What happens if you use other options, including illegal options?
What is the output of

Code:

ls -l /usr/sbin/sshd

file /usr/sbin/sshd

ldd /usr/sbin/sshd

strace /usr/sbin/sshd

I suspect some library problem or architecture incompatibility.

Have you tried reinstalling sshd and the openssh libraries?

By the way, you can use a serial console with Virtualbox, which would allow you to copy and paste to/from your VM's command line.

Quote:

Originally Posted by berndbausch (Post 6171312)

What happens if you use other options, including illegal options?
What is the output of

Code:

ls -l /usr/sbin/sshd

file /usr/sbin/sshd

ldd /usr/sbin/sshd

strace /usr/sbin/sshd

Hi,

Yes I have tried completely uninstalling the openssh server using yum uninstall and yum remove command, also tried "ssh-keygen -A" by regenerating all keys.

For all the output, allow me some moment, I will get back with the required output. I think as you mentioned, some library or architecture problem, since the VM is based on Google Cloud, but need to fix it.

Quote:

Originally Posted by berndbausch (Post 6171312)

What happens if you use other options, including illegal options?
What is the output of

Code:

ls -l /usr/sbin/sshd

file /usr/sbin/sshd

ldd /usr/sbin/sshd

strace /usr/sbin/sshd

Hi,

Yes I have tried completely uninstalling the openssh server using yum uninstall and yum remove command, also tried "ssh-keygen -A" by regenerating all keys.

For the required output, please check the screenshots here attached as well as the complete output of strace.

Help me enabling the sshd please.

Strange issue, when I uninstall postfix, I was able to turn on sshd once, but cannot do that anymore now. I need the postfix for my application deployed on the VM.

Any help here please?

Sorry for not reacting earlier.

Good news: My suspicion that sshd couldnkt run because of an incompatible architecture in the binary itself or a library it's using seems to be unfounded. The system call trace shows that sshd works.

Bad news: I can't see anything else, since the system call trace does not include the child process launched by sshd. To see those traces, use strace -f. My bad for not thinking of this.

Another thing I would try is running sshd with a very verbose option (sshd -vvv). Perhaps the debug messages thus printed will reveal the problem.

Quote:

Originally Posted by berndbausch (Post 6174315)

Hello, I have lots of funny news for you, don't know what is happening here. please see the attached screenshot - maybe some shock for you.

Also for output - I have provided the output for "sshd -ddd" and "strace -f".

running "/usr/sbin/sshd -ddd" enable a listening to sshd, but status shows sshd is inactive/dead. Also netstat -tupln shows sshd is listening, but netstat -tpln does NOT show port 22 is listening. And I can ssh to the VM from my mac/local machine, even checking the sshd status from the mac also shows sshd is dead.

When you run sshd from the command line, you just launch a process, not the sshd service. Therefore systemctl status reports that the service is dead. However, sshd is running, and you can log in.

Your netstat output is rather unexpected. Unfortunately you don't share it.

Why the regular start of sshd doesn't work, I will try to find by looking at the trace files you attached. That will take some time.

Quote:

Originally Posted by berndbausch (Post 6174605)

Thanks, I will wait for the trace output result.

Quote:

Originally Posted by berndbausch (Post 6174605)

Sorry about the netstat output - i messed up the process run of sshd, and messed up the netstat output. Actually both netstat -tpln and -tupln show no sshd/port 22.

Anyway, I also checked the strace -f output, and I see some DenyWrite and files not found issue. Can you please check and let me know how to solve this? My curious question is why and how this sshd is working on Google Cloud, but not on local VMware/Virtualbox or even in other cloud.

Quote:

Originally Posted by berndbausch (Post 6174605)

Hi, any update here please?

Quote:

Originally Posted by islammanjurul (Post 6175565)

Actually both netstat -tpln and -tupln show no sshd/port 22.

If it is not listening then it is probably not running. What is the output of the configuration test?

Code:

sudo /usr/sbin/sshd -T | sort

If the configuration file is ok, then going in parallel with your strace approach, can you try starting a one-off instance of sshd using the -d option(s)?

Code:

sudo /usr/sbin/sshd -d -E /tmp/one-off-sshd.log

That should put a report in the file /tmp/one-off-sshd.log about what was tried but failed.

Quote:

Originally Posted by Turbocapitalist (Post 6177439)

If it is not listening then it is probably not running. What is the output of the configuration test?

Code:

sudo /usr/sbin/sshd -T | sort

If the configuration file is ok, then going in parallel with your strace approach, can you try starting a one-off instance of sshd using the -d option(s)?

Code:

sudo /usr/sbin/sshd -d -E /tmp/one-off-sshd.log

That should put a report in the file /tmp/one-off-sshd.log about what was tried but failed.

here are the outputs:

Code:

sudo /usr/sbin/sshd -T | sort

Code:

[test@pulsuat-2c689991 ~]$ sudo /usr/sbin/sshd -T | sort

[sudo] password for test: 

acceptenv LANG

acceptenv LANGUAGE

acceptenv LC_ADDRESS

acceptenv LC_ALL

acceptenv LC_COLLATE

acceptenv LC_CTYPE

acceptenv LC_IDENTIFICATION

acceptenv LC_MEASUREMENT

acceptenv LC_MESSAGES

acceptenv LC_MONETARY

acceptenv LC_NAME

acceptenv LC_NUMERIC

acceptenv LC_PAPER

acceptenv LC_TELEPHONE

acceptenv LC_TIME

acceptenv XMODIFIERS

addressfamily any

allowagentforwarding yes

allowstreamlocalforwarding yes

allowtcpforwarding yes

authenticationmethods any

authorizedkeyscommand none

authorizedkeyscommanduser none

authorizedkeysfile .ssh/authorized_keys

authorizedprincipalscommand none

authorizedprincipalscommanduser none

authorizedprincipalsfile none

banner none

challengeresponseauthentication yes

chrootdirectory none

ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc

clientalivecountmax 3

clientaliveinterval 180

compression yes

disableforwarding no

exposeauthenticationmethods never

fingerprinthash SHA256

forcecommand none

gatewayports no

gssapiauthentication yes

gssapicleanupcredentials no

gssapienablek5users no

gssapikexalgorithms gss-gex-sha1-,gss-group1-sha1-,gss-group14-sha1-

gssapikeyexchange no

gssapistorecredentialsonrekey no

gssapistrictacceptorcheck yes

hostbasedacceptedkeytypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss

hostbasedauthentication no

hostbasedusesnamefrompacketonly no

hostkeyagent none

hostkeyalgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss

hostkey /etc/ssh/ssh_host_dsa_key

hostkey /etc/ssh/ssh_host_ecdsa_key

hostkey /etc/ssh/ssh_host_ed25519_key

hostkey /etc/ssh/ssh_host_rsa_key

ignorerhosts yes

ignoreuserknownhosts no

ipqos lowdelay throughput

kbdinteractiveauthentication yes

kerberosauthentication no

kerberosorlocalpasswd yes

kerberosticketcleanup yes

kerberosusekuserok yes

kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

listenaddress 0.0.0.0:2222

logingracetime 120

loglevel INFO

macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

maxauthtries 6

maxsessions 10

maxstartups 10:30:100

passwordauthentication yes

permitemptypasswords no

permitopen any

permitrootlogin no

permittty yes

permittunnel no

permituserenvironment no

permituserrc yes

pidfile /var/run/sshd.pid

port 2222

printlastlog yes

printmotd yes

pubkeyacceptedkeytypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss

pubkeyauthentication yes

rekeylimit 0 0

revokedkeys none

showpatchlevel no

streamlocalbindmask 0177

streamlocalbindunlink no

strictmodes yes

subsystem sftp /usr/libexec/openssh/sftp-server

syslogfacility AUTHPRIV

tcpkeepalive yes

trustedusercakeys none

usedns yes

usepam yes

useprivilegeseparation sandbox

versionaddendum none

x11displayoffset 10

x11forwarding yes

x11maxdisplays 1000

x11uselocalhost yes

xauthlocation /usr/bin/xauth

Code:

sudo /usr/sbin/sshd -d -E /tmp/one-off-sshd.log

Code:

debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips  26 Jan 2017

debug1: private host key #0: ssh-rsa SHA256:Rq7VVwy1U/udxLzHY+pzCG4FAUcwE+OJ3elCwUjUPXU

debug1: private host key #1: ssh-dss SHA256:ZXBhQiZMNDMWiuzNORdR57Ij5viSZX3I+ggu9f4479k

debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:k6YJ21Qpgv7+9tRvC7KSPsevAQfjC95bWHGXAVm4NdI

debug1: private host key #3: ssh-ed25519 SHA256:sN7uZ+2IcPZlicQ10PlC8p7BJ2zCIqnAl7XTHiTEs5g

debug1: rexec_argv[0]='/usr/sbin/sshd'

debug1: rexec_argv[1]='-d'

debug1: rexec_argv[2]='-E'

debug1: rexec_argv[3]='/tmp/one-off-sshd.log'

debug1: Set /proc/self/oom_score_adj from 0 to -1000

debug1: Bind to port 2222 on ::.

Server listening on :: port 2222.

debug1: Bind to port 22 on 0.0.0.0.

Server listening on 0.0.0.0 port 2222.

Kindly help me here, all outputs required to troubleshoot are here, strace -f output at top message as attachment.

The configuration looks normal. Have you changed 'usedns' though?

The log output shows that sshd runs and appears to be listening on all interfaces.

Two more questions while waiting for the strace:

Can you SSH to it from the same machine which it is on using the local host address?

Can you SSH to it from the same machine which it is on using an external address?

Quote:

Originally Posted by Turbocapitalist (Post 6177668)

As already mentioned on a previous comment on this thread, when i run /usr/sbin/sshd -ddd , it creates a process, and then I am able to ssh from my local host to the virtualbox VM. After a reboot, the ssh is gone, again need to run sshd -ddd.

about usedns - can you elaborate please? no change in resolv.conf file actually. everything is default as per the centos OS.