pkg/sftp Much Slower than Linux SCP, Why? - go

Question

With standard scp (version: 1:7.6p1-4ubuntu0.3) on Ubuntu 18.04, transferring a 110GB file to another host takes roughly 6-8 minutes.
When using Go's pkg/sftp, it takes double the amount of time:
Example:
package main
import (
"fmt"
"github.com/melbahja/goph"
"golang.org/x/crypto/ssh"
"io"
"log"
"os"
"time"
)
func main() {
auth, err := goph.Key("your/key", "")
if err != nil {
log.Fatalln(err)
}
client, err := goph.NewConn(&goph.Config{
User: "someUser",
Addr: "someHostname",
Port: 22,
Auth: auth,
//Timeout: time.Duration(timeout) * time.Second,
Callback: ssh.InsecureIgnoreHostKey(),
})
if err != nil {
log.Fatalln(err)
}
file := "/some/file"
start := time.Now()
local, err := os.Open(file)
if err != nil {
return
}
defer local.Close()
ftp, err := client.NewSftp()
// VARIATION 1 => ftp, err := client.NewSftp(sftp.MaxPacketUnchecked(1 << 16))
if err != nil {
return
}
defer ftp.Close()
remote, err := ftp.Create(file)
if err != nil {
return
}
defer remote.Close()
/*
VARIATION 1 => buffer := make([]byte, 1 << 16)
VARIATION 1 => _, err = io.CopyBuffer(remote, local, buffer)
*/
_, err = io.Copy(remote, local)
if err != nil {
log.Fatalln(err)
}
duration := time.Since(start)
fmt.Println(duration)
}
Note: I even attempted to increase the size of the read buffer (and tcp max packet size) with the commented out lines (see VARIATION 1), and it made no difference.
Any ideas as to why, and how to speed up the Go equivalent?

Answer 1

I've found that pkg/sftp supports concurrent upload and the concurrent upload speed is quite similar to linux sftp command.
Here the client initialization:
sftpConn, err := sftp.NewClient(sshConn,
sftp.UseConcurrentReads(true),
sftp.UseConcurrentWrites(true),
sftp.MaxConcurrentRequestsPerFile(64),
// Big max packet size can improve throughput.
sftp.MaxPacketUnchecked(128*agentio.KiB),
// Beware of customizing max packet size for download!
// On download, big max packet size can cause "connection lost" error.
)
Here is how you can use it, pay attention to ReadFrom:
func sftpUpload(c *sftp.Client, r io.Reader, path string) error {
fp, err := c.Create(path)
if err != nil {
return fmt.Errorf("create destination file: %w", err)
}
defer fp.Close()
_, err = fp.ReadFrom(r)
return err
}
To use concurrent upload the reader's underline struct has to be bytes.Reader, io.LimitedReader or satisfy one of the following interfaces:
Len() int
Size() int64
Stat() (os.FileInfo, error)
It's required to determine amount of bytes to upload.
If ReadFrom can't determine reader's size than it will fallback to single thread upload.
Check the ReadFrom source code:
func (f *File) ReadFrom(r io.Reader) (int64, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.c.useConcurrentWrites {
var remain int64
switch r := r.(type) {
case interface{ Len() int }:
remain = int64(r.Len())
case interface{ Size() int64 }:
remain = r.Size()
case *io.LimitedReader:
remain = r.N
case interface{ Stat() (os.FileInfo, error) }:
info, err := r.Stat()
if err == nil {
remain = info.Size()
}
}
Also, if you pass the reader to ReadFrom which producing a lot of small peaces of data than the throughput will be quite low even with concurrency.