Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
Thanks,
On Mon, 2 Nov 2020, Jakub Jelen wrote:
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
This is excellent!
Indeed, most users don't care about the underlying SSH protocol flavour other than expecting it to be secure. They just want "scp" to work.
Your solution does both. It is a very good example of listening to what users want and supported users without requiring them to learn a new command like sftp. I wish more developers had this focus!
I will try it out over the next few days. My personal usecases would be that it still keeps working with rsync's method of invoking scp and that we would have sftp enable by default for sshd by default.
Thanks!
Paul
On Monday, November 2, 2020 2:44:39 PM WET Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
Hi Jakub, if I am not sure if I understood what you said, you intend to deprecate the scp protocol/inner working but not the scp binary. Is that correct?
I like the ease of the use of the scp, even if sometimes I would prefer it to have the same options as the usual cp. :-)
Best regards,
Am 02.11.20 um 15:44 schrieb Jakub Jelen:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
if it is compatible with what powerusers i.e. sysadmins do with it, it should be fine. I have such things as Compression, Cipher, Ports and 2 sets of login credentials in mind, to directly copy from a to b without parking it first on the pc running the scp.
On the server side it has to honor things like CHROOT directives.
best regards, Marius
On 11/2/20 4:05 PM, Marius Schwarz wrote:
Am 02.11.20 um 15:44 schrieb Jakub Jelen:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
if it is compatible with what powerusers i.e. sysadmins do with it, it should be fine. I have such things as Compression, Cipher, Ports and 2 sets of login credentials in mind, to directly copy from a to b without parking it first on the pc running the scp.
Compression, Ciphers and Ports are configuration options for ssh protocol, which stays unchanged.
Copying remote to remote (without the -3 option) should work after updating scp also on the remote source machine.
On the server side it has to honor things like CHROOT directives.
This is also a property/configuration of ssh server. Using chroot with sftp is much easier than with scp as you really do not need any binaries in chroot and you can use internal-sftp server.
Regards,
On Mon, Nov 02, 2020 at 03:44:39PM +0100, Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
I like it! scp-the-tool (including command/filename completion!) is just so much nicer to work with than sftp-the-tool.
But sftp is also slightly slower than scp, at least on large sustained transfers.
My non-scientific test transfered a 13GB file at 111.1MB/s vs 112.0MB/s between two (very fast) endpoints connected via GigE.
(I need to see how well it handles smaller transfers, this is where SCP gets pretty crappy..)
- Solomon
Am 02.11.20 um 16:13 schrieb Solomon Peachy:
On Mon, Nov 02, 2020 at 03:44:39PM +0100, Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
I like it! scp-the-tool (including command/filename completion!) is just so much nicer to work with than sftp-the-tool.
Well, thats a nice feature requirement, almost forgot about it.
But sftp is also slightly slower than scp, at least on large sustained transfers.
Hope not, slow data transfer is already an issue with Gb/s :).
Best regards, Marius
Le lundi 02 novembre 2020 à 16:16 +0100, Marius Schwarz a écrit :
Am 02.11.20 um 16:13 schrieb Solomon Peachy:
On Mon, Nov 02, 2020 at 03:44:39PM +0100, Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
I like it! scp-the-tool (including command/filename completion!) is just so much nicer to work with than sftp-the-tool.
Well, thats a nice feature requirement, almost forgot about it.
Well lftp-the-tool is even better. And they all do the same thing from a functional POW, only the underlying protocol changes. This is sad, the user wouldn’t care less about protocol differences (as he would not care less about TLS negociation, as long as the resulting recipe is steong)
User attachment to different tools is a legacy of other OSes where one protocol is built in and the others not.
Regards,
On Monday, November 2, 2020 3:44:39 PM CET Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
How is the "compatibility scpd to support old clients" going to differ from the current implementation?
libcurl implements its own SCP client over libssh. Will this implementation continue to work after OpenSSH gets updated on servers?
Applications often allow users to pass arbitrary URLs to libcurl. So one can, for example, use scp:// URLs to specify a kickstart for Anaconda. The fact that scp utility will be reimplemented over SFTP does not help much in this case. Each build of libcurl that supports scp:// supports sftp:// as well. But libcurl will not transmit scp:// requests over sftp:// in case SCP is not supported by the remote server any more.
Kamil
On Mon, 2020-11-02 at 16:36 +0100, Kamil Dudka wrote:
On Monday, November 2, 2020 3:44:39 PM CET Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
How is the "compatibility scpd to support old clients" going to differ from the current implementation?
libcurl implements its own SCP client over libssh. Will this implementation continue to work after OpenSSH gets updated on servers?
Applications often allow users to pass arbitrary URLs to libcurl. So one can, for example, use scp:// URLs to specify a kickstart for Anaconda. The fact that scp utility will be reimplemented over SFTP does not help much in this case. Each build of libcurl that supports scp:// supports sftp:// as well. But libcurl will not transmit scp:// requests over sftp:// in case SCP is not supported by the remote server any more.
Sounds like a RFE for libcurl to slowly move scp:// to be using the sftp protocol instead ?
Or they could simply deprecate it, and then users will have to change their config to say sftp://
For something like libcurl the latter is probably more appropriate anyway.
Simo.
On Monday, November 2, 2020 4:47:47 PM CET Simo Sorce wrote:
On Mon, 2020-11-02 at 16:36 +0100, Kamil Dudka wrote:
How is the "compatibility scpd to support old clients" going to differ from the current implementation?
libcurl implements its own SCP client over libssh. Will this implementation continue to work after OpenSSH gets updated on servers?
Applications often allow users to pass arbitrary URLs to libcurl. So one can, for example, use scp:// URLs to specify a kickstart for Anaconda. The fact that scp utility will be reimplemented over SFTP does not help much in this case. Each build of libcurl that supports scp:// supports sftp:// as well. But libcurl will not transmit scp:// requests over sftp:// in case SCP is not supported by the remote server any more.
Sounds like a RFE for libcurl to slowly move scp:// to be using the sftp protocol instead ?
As far as I know, (lib)curl has never silently transformed a protocol scheme explicitly specified with URL. This can be discussed upstream but I do not feel like starting the discussion myself.
Or they could simply deprecate it, and then users will have to change their config to say sftp://
For something like libcurl the latter is probably more appropriate anyway.
Yes, I believe this is the curl way to handle it. Nothing is being changed for curl now as I understand it. So there is no need to take an immediate action. Anyway, I will notify curl upstream about the plan so they are not surprised later on.
Kamil
Simo.
-- Simo Sorce RHEL Crypto Team Red Hat, Inc
* Kamil Dudka:
As far as I know, (lib)curl has never silently transformed a protocol scheme explicitly specified with URL. This can be discussed upstream but I do not feel like starting the discussion myself.
Curl does it for https://, so it should be fine for scp://.
Thanks, Florian
On Monday, November 2, 2020 5:57:29 PM CET Florian Weimer wrote:
- Kamil Dudka:
As far as I know, (lib)curl has never silently transformed a protocol scheme explicitly specified with URL. This can be discussed upstream but I do not feel like starting the discussion myself.
Curl does it for https://, so it should be fine for scp://.
Could you please be more specific? What exactly does curl do for https:// ?
Kamil
Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
* Kamil Dudka:
On Monday, November 2, 2020 5:57:29 PM CET Florian Weimer wrote:
- Kamil Dudka:
As far as I know, (lib)curl has never silently transformed a protocol scheme explicitly specified with URL. This can be discussed upstream but I do not feel like starting the discussion myself.
Curl does it for https://, so it should be fine for scp://.
Could you please be more specific? What exactly does curl do for https:// ?
It automatically replaces the underlying HTTP transport with HTTP/2 if the server replaces it. These protocols are about as similar as SCP and SFTP, I would say.
Thanks, Florian
On Monday, November 2, 2020 6:31:33 PM CET Florian Weimer wrote:
- Kamil Dudka:
On Monday, November 2, 2020 5:57:29 PM CET Florian Weimer wrote:
- Kamil Dudka:
As far as I know, (lib)curl has never silently transformed a protocol scheme explicitly specified with URL. This can be discussed upstream but I do not feel like starting the discussion myself.
Curl does it for https://, so it should be fine for scp://.
Could you please be more specific? What exactly does curl do for https:// ?
It automatically replaces the underlying HTTP transport with HTTP/2 if the server replaces it. These protocols are about as similar as SCP and SFTP, I would say.
Thank you for clarifying it!
The key difference is that the HTTP protocol upgrade to HTTP/2 (or HTTP/3 via Alt-Svc header) are covered by IETF specifications. There is also no protocol scheme to specify HTTP/2 explicitly in URL, unlike SCP vs. SFTP.
Kamil
Thanks, Florian
On 11/2/20 4:36 PM, Kamil Dudka wrote:
On Monday, November 2, 2020 3:44:39 PM CET Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
How is the "compatibility scpd to support old clients" going to differ from the current implementation?
I can think of a solution that in the end, there will be just the server parts of the current scp and the client code branches will be gone or support sftp only. But this can change as we are not there yet.
libcurl implements its own SCP client over libssh. Will this implementation continue to work after OpenSSH gets updated on servers?
With the above update, everything will work as before -- it affects only the client scp binary.
Applications often allow users to pass arbitrary URLs to libcurl. So one can, for example, use scp:// URLs to specify a kickstart for Anaconda. The fact that scp utility will be reimplemented over SFTP does not help much in this case. Each build of libcurl that supports scp:// supports sftp:// as well. But libcurl will not transmit scp:// requests over sftp:// in case SCP is not supported by the remote server any more.
As Simo wrote, I think it is something that will have to happen sooner or later inside of libcurl or libssh or in users configurations. But again, the above change should not have any effect on this.
Regards,
On Mon, Nov 02, 2020 at 03:44:39PM +0100, Jakub Jelen wrote:
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
Has that testing included performance measurements, both on high bandwidth low-latency transfers and low bandwidth high-latency transfers? At least in the past SFTP used to be worse than SCP on high-latency connections.
Jakub
Jakub Jelen jjelen@redhat.com wrote:
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
Does it work with connection sharing (ControlPath, ControlMaster, ControlPersist config options)?
David
On 11/2/20 10:29 PM, David Howells wrote:
Jakub Jelen jjelen@redhat.com wrote:
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
Does it work with connection sharing (ControlPath, ControlMaster, ControlPersist config options)?
It should keep working the same way as it worked before with scp.
Regards,