[SOLVED] Using lftp to connect to a Bitwise SSH server.

Bob4K · 04-16-2017, 06:49 AM

Hi,

I'm trying to use lftp with a script that would automatically log in an SSH server and download in multipart mode all the files that are situated on 2 given directories. The names and the number of the files is unknown. I'm running lftp on Windows, but since it's originally a Linux utility and uses the same syntax, I hope to find some useful advice here.

The server is a Bitwise SSH server.

Let's say that the server MyPC is foo.dynu.com, the port is 9999, the username MyUser and the password MyPass. The Server is configured to only accept the use of a key pair, logging with only the username and password is rejected.

If I use CrossFT to log in, the Bitwiser server log shows (note that the user name, password, IPs and ports have been changed to random values):

Code:

desc="Connection accepted.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Client version string received.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters clientVersion="SSH-2.0-JSCH-0.1.53"/>
  </event>
 
desc="Key exchange algorithms.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters kexAlg="diffie-hellman-group14-sha1" cipherAlgIn="aes128-ctr" cipherAlgOut="aes128-ctr" macAlgIn="hmac-md5" macAlgOut="hmac-md5" comprAlgIn="none" comprAlgOut="none"/>
  </event>
 
desc="User authentication failed with partial success.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="TestOnly" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
    <parameters failureReason="NoSignature"/>
    <help message="The public key supplied has been verified; requesting signature."/>
  </event>
 
desc="User authentication succeeded.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="WithSignature" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
  </event>
 
desc="Session channel opened.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
  </event>
 
desc="SFTP protocol version negotiated.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
    <parameters versionRequested="3" versionSelected="3"/>
  </event>
 
desc="Virtual filesystem: get file status.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
desc="Getting file status succeeded.">
      <parameters path="d:\caps" flags="0x8000002F" partialFailure="false"/>
    </sfs>
  </event>

I tried the following script with lftp:

Code:

debug 10
set sftp:connect-program "ssh -a -x -i D:/m1/.ssh/id_rsaNoPass.ppk"
open MyUser,MyPass@foo.dynu.com:9999
cd /d/caps
cat *

and I get on the terminal:

Code:

nt{lftp}% lftp test
D:/m1/soft/l/lftp/lftp-4.7.5.win64-openssl/bin/lftp.exe -f test
Password:
---- Resolving host address...
buffer: EOF on FD 5
---- 1 address found: 222.22.222.222
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
buffer: EOF on FD 5
<--- !!!T!!!!!!,BssLoginTimeout: user authentication timeout!!!!¦¦¦¦-¦?w¦%¦¦S¦¦¦N¦J?G¦¦
**** Peer closed connection
---- Closing control socket
---- dns cache hit
---- attempt number 2 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
cd `/d/caps' [FEAT negotiation...]

with the following log:

Code:

desc="Connection accepted.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters disconnectReason="EofReceived"/>
    <help message="The client has disconnected the session by sending EOF."/>
  </event>

So, there are a few problems:

- lftp asks for a password. But the private key does not need a passphrase, and the user password is provided in the script. So, what is this password asked for?

- the server acknowledges the connection, so it's not a firewall problem. But then, it stalls, there is no key pair authentication.

What am I missing?

wpeckham · 04-16-2017, 12:15 PM

Quote:

Originally Posted by Bob4K

Hi,

I'm trying to use lftp with a script that would automatically log in an SSH server and download in multipart mode all the files that are situated on 2 given directories. The names and the number of the files is unknown. I'm running lftp on Windows, but since it's originally a Linux utility and uses the same syntax, I hope to find some useful advice here.

The server is a Bitwise SSH server.

Let's say that the server MyPC is foo.dynu.com, the port is 9999, the username MyUser and the password MyPass. The Server is configured to only accept the use of a key pair, logging with only the username and password is rejected.

If I use CrossFT to log in, the Bitwiser server log shows (note that the user name, password, IPs and ports have been changed to random values):

Code:

desc="Connection accepted.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Client version string received.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters clientVersion="SSH-2.0-JSCH-0.1.53"/>
  </event>
 
desc="Key exchange algorithms.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <parameters kexAlg="diffie-hellman-group14-sha1" cipherAlgIn="aes128-ctr" cipherAlgOut="aes128-ctr" macAlgIn="hmac-md5" macAlgOut="hmac-md5" comprAlgIn="none" comprAlgOut="none"/>
  </event>
 
desc="User authentication failed with partial success.">
    <session id="1289" remoteAddress="11.111.11.1:11111"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="TestOnly" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
    <parameters failureReason="NoSignature"/>
    <help message="The public key supplied has been verified; requesting signature."/>
  </event>
 
desc="User authentication succeeded.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <authentication attemptNr="1" userName="MyUser" method="publickey" flags="WithSignature" keyAlgorithm="ssh-rsa" keyMd5="60:7c:34:3c:c6:80:23:93:50:1e:2a:20:e1:73:53:a7"/>
  </event>
 
desc="Session channel opened.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
  </event>
 
desc="SFTP protocol version negotiated.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
    <parameters versionRequested="3" versionSelected="3"/>
  </event>
 
desc="Virtual filesystem: get file status.">
    <session id="1289" remoteAddress="11.111.11.1:11111" windowsAccount="MyPC\MyUser"/>
    <channel type="session" id="1"/>
desc="Getting file status succeeded.">
      <parameters path="d:\caps" flags="0x8000002F" partialFailure="false"/>
    </sfs>
  </event>

I tried the following script with lftp:

Code:

debug 10
set sftp:connect-program "ssh -a -x -i D:/m1/.ssh/id_rsaNoPass.ppk"
open MyUser,MyPass@foo.dynu.com:9999
cd /d/caps
cat *

and I get on the terminal:

Code:

nt{lftp}% lftp test
D:/m1/soft/l/lftp/lftp-4.7.5.win64-openssl/bin/lftp.exe -f test
Password:
---- Resolving host address...
buffer: EOF on FD 5
---- 1 address found: 222.22.222.222
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
buffer: EOF on FD 5
<--- !!!T!!!!!!,BssLoginTimeout: user authentication timeout!!!!¦¦¦¦-¦?w¦%¦¦S¦¦¦N¦J?G¦¦
**** Peer closed connection
---- Closing control socket
---- dns cache hit
---- attempt number 2 (max_retries=1000)
---- Connecting to foo.dynu.com (222.22.222.222) port 9999
<--- SSH-2.0-5.21 FlowSsh: Bitvise SSH Server (WinSSHD) 6.03: free only for personal non-commercial use
cd `/d/caps' [FEAT negotiation...]

with the following log:

Code:

desc="Connection accepted.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1290" remoteAddress="00.000.00.0:00000"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1291" remoteAddress="00.000.00.0:52014"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1292" remoteAddress="00.000.00.0:52043"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1293" remoteAddress="00.000.00.0:52088"/>
    <parameters disconnectReason="SshError"/>
description="BssLoginTimeout: user authentication timeout"/>
  </event>
 
desc="Connection accepted.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters addressRule="AnyIP" listenAddress="192.168.1.26:9999"/>
  </event>
 
desc="Session disconnected normally.">
    <session id="1294" remoteAddress="00.000.00.0:52149"/>
    <parameters disconnectReason="EofReceived"/>
    <help message="The client has disconnected the session by sending EOF."/>
  </event>

So, there are a few problems:

- lftp asks for a password. But the private key does not need a passphrase, and the user password is provided in the script. So, what is this password asked for?

- the server acknowledges the connection, so it's not a firewall problem. But then, it stalls, there is no key pair authentication.

What am I missing?

Not sure what you are missing, but when keys are involved I pass the logon name and password on the command line. It is critical to know if anyone else can log in and see the process list, as this exposes the name and password in the process list.

In these cases, by the way, sftp behavior and lftp behavior are quite different. Lftp may delay authentication on the remote server until it needs a folder or file list, or otherwise actually needs to transfer data. SFTP does the authentication up front.

Try this manually first, before turning it over to a script, so you get a chance to observe and troubleshoot during the dialog.

Bob4K · 04-16-2017, 12:26 PM

Thanks for the answer.

Unfortunately, this turns out to be Windows ssh problem.

My script had a few errors, it should be:

Code:

debug 10
set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i d:/m1/.ssh/id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999

It turned out that I needed the full path to shh.

But still, I get:

Code:

---- attempt number 1 (max_retries=1000)
---- Running connect program (D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP -s -l MyUser -p 9999 rrg.dynu.co
m sftp)
---> sending a packet, length=5, type=1(INIT), id=0
<--- OpenSSH_7.4p1, OpenSSL 1.0.2j  26 Sep 2016
<--- debug1: Connecting to foo.dynu.com [222.22.222.222] port 9999.
<--- debug1: Connection established.
<--- Could not create directory '/home/m1/.ssh'.
<--- debug1: key_load_public: No such file or directory

So, ssh is unable to load "d:/m1/.ssh/id_rsaNP". I tried something like "/cygwin/d/m1/.ssh/id_rsaNP", but that does not work either.

This forum is likely not the right place to address this Windows/Cygwin port to lftp/ssh, any idea where I could find some help?

wpeckham · 04-16-2017, 02:08 PM

I use primarily windows tools on windows (putty and winscp), and use cygwin for xwindows integration etc.
Everything I have done in cygwin has pretty much worked as if it were running under linux, but there are path restrictions and differences. I would look into why it thinks it should create that folder, and why it cannot.

Meanwhile, I might find another way to do the transfer. This is only ONE way, and there are many more tools than lftp!

Bob4K · 04-16-2017, 02:09 PM

I tried an older version, 4.4.15 32 bits instead if the current (Windows) 4.4.17 64 bits, and it works nicely.

In case anybody is interested, here is a working script on Windows:

Code:

set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999
cd /d/caps
set mirror:use-pget-n 8
mirror /d/caps /cygdrive/d/caps

Bob4K · 04-16-2017, 02:14 PM

Quote:

Originally Posted by wpeckham

Meanwhile, I might find another way to do the transfer. This is only ONE way, and there are many more tools than lftp!

Windows clients that can do multi-part transfers are not very good. The best is CrossFTP, who makes a bunch of small files, and then rejoin them, which is a waste fo time.

lftp creates a large file from the get go, and then fills it. That's the main reason I wanted to try it.

wpeckham · 04-16-2017, 02:17 PM

Quote:

Originally Posted by Bob4K

I tried an older version, 4.4.15 32 bits instead if the current (Windows) 4.4.17 64 bits, and it works nicely.

In case anybody is interested, here is a working script on Windows:

Code:

set sftp:connect-program 'D:/m1/soft/L/lftp/lftp/bin/ssh.exe -v -a -x -i id_rsaNP'
open sftp://MyUser:dummy@foo.dynu.com:9999
cd /d/caps
set mirror:use-pget-n 8
mirror /d/caps /cygdrive/d/caps

Ahaa! Do you think that you tripped over a bit-width code issue? I had not heard of one, but this is interesting. I may try it myself to see if I can replicate the behavior.
meanwile, you might want to mark the thread solved so others will be able to find this solutions!

Bob4K · 04-16-2017, 02:26 PM

Quote:

Originally Posted by wpeckham

Ahaa! Do you think that you tripped over a bit-width code issue? I had not heard of one, but this is interesting. I may try it myself to see if I can replicate the behavior.

I tested the 64 bit 4.4.15 version, and it works too, so it's a bug in 4.7.7.

Quote:

meanwile, you might want to mark the thread solved so others will be able to find this solutions!

Done.

That said, CrossFTP appears to be better overall: I reach 16 MBs with it, while it peaks at 10 with lftp. So, despite a better way to handle multi-part, lftp takes more time to download large files.

Well, at least I had fun making it work...

Bob4K · 04-16-2017, 02:55 PM

Interesting find: if I download to an SSD rather than a regular HDD, the speed jumps to 21 MBs.

So, I'll be replacing CrossFTP with lftp, with more speed and the added bonus of a scripted automated download.

wpeckham · 04-17-2017, 06:33 AM

Wonderful! I am not sure I ever saw anyone advance lftp for speed. It is a great tool for transfer automation, and handles MANY protocols, but it may not be as fast as a dedicated tool that does or checks less. I never tested performance for lftp.

Very glad you solved this!

Bob4K · 04-17-2017, 11:49 AM

Quote:

Originally Posted by wpeckham

Wonderful! I am not sure I ever saw anyone advance lftp for speed. It is a great tool for transfer automation, and handles MANY protocols, but it may not be as fast as a dedicated tool that does or checks less. I never tested performance for lftp.

Very glad you solved this!

Every SFTP client I tried on Windows runs at max at 1.5MBs (that's lftp speed) by connection, despite the fact that I have a Gb FTTH connection. Filezilla maxes out at .5MBs.

So, segmented downloads are the only way to get speed, and I found only a few clients that support that:

- Bitkinex (I think), but it's unusable on a high res monitor due to its use of tiny fonts.

- SmartFTP. But it uses an incredibly bloated UI, and does something to the Bitwise server that often leaves it on a loop at 100% cpu. Also, it requires to install a Microsoft patch on Windows 7 that prevents the Windows Resource Monitor to work.

- CrossFTP, which is ok but annoying in the way it creates multiple small files before joining them.

lftp with 32 connections works the best, but 32 simultaneous accesses is too much for a regular hard drive, so it needs an SSD.