Snowball Edge - aws-sdk-go package in Golang - Can't Connect to S3 - go

Question

I'm using the aws-sdk-go package in Golang to connect to Amazon S3 to provide a cloud-based storage pool. I have this working well. I would like to be able to support bulk high-speed transfers using Snowball, so I got a Snowball Edge to test this in my lab. I have not figured out how to get this working, and the documentation for Snowball Edge doesn't seem complete. This configuration may be impacted by having ordered a Snowball Edge and not just a Snowball.
The reason that I'm finding the Edge more problematic, is that a normal Snowball requires an application called snowballAdapter to be running, which looks like it handles some port mapping issues. But, this application seems to be incompatible with the Edge device, as it reports that it doesn't work with a "Snowball Edge Manifest file".
I looked at the ports that are available on the real AWS S3 and nmap reports:
nmap -v -sT -Pn s3.us-east-1.amazonaws.com
...
Scanning s3.us-east-1.amazonaws.com (52.216.161.53) [1000 ports]
Discovered open port 443/tcp on 52.216.161.53
Discovered open port 80/tcp on 52.216.161.53
Whereas on Snowball Edge, the ports are:
nmap -v -sT -Pn 192.168.1.4
....
Scanning 192.168.1.4 [1000 ports]
Discovered open port 8080/tcp on 192.168.1.4
Discovered open port 22/tcp on 192.168.1.4
Discovered open port 9091/tcp on 192.168.1.4
Discovered open port 8443/tcp on 192.168.1.4
....
PORT STATE SERVICE
22/tcp open ssh
8080/tcp open http-proxy
8443/tcp open https-alt
9091/tcp open xmltec-xmlmail
So, it seems to me that the issue may be that I have to make the aws package use port 8443 for Snowball Edge instead of 443 for the real S3. The code for connecting to S3 is pretty straight forward:
creds := credentials.NewStaticCredentials(s3Config.S3AccessKey, s3Config.S3SecretAccessKey, s3Config.S3Token)
_, err := creds.Get()
if err != nil {
return nil, nil, err
}
if len(baseFolder) > 0 {
baseFolder = baseFolder + "/"
}
cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds)
svc := s3.New(session.New(), cfg)
params := &s3.ListObjectsInput{
Bucket: aws.String(s3Config.S3BucketName),
Prefix: aws.String(baseFolder),
Delimiter: aws.String("/"),
}
resp, err := svc.ListObjects(params)
So, the question is, how do I change the code to point to the Snowball Edge? I've tried mapping to the Snowball Edge in /etc/hosts from the Amazon S3 endpoint. I understand why this didn't work after discovering that the ports are different. I've played around with adding different forms of "WithEndpoint("...host...") with no success. Or, am I on the completely wrong track, and should be able to get the snowballAdapter to work with a Snowball Edge?
By the way, all the snowballEdge commands work as expected, so the device seems to be working fine, e.g.:
./snowballEdge list-access-keys
{
"AccessKeyIds" : [ "..." ]
}
./snowballEdge get-secret-access-key --access-key-id ....
[snowballEdge]
aws_access_key_id = ...
aws_secret_access_key = ...
And, I've used the correct keys associated with the device, and it does have the S3 Service configured in it:
./snowballEdge list-services
{
"ServiceIds" : [ "s3" ]
}

Answer 1

Snowball Edge is a very different beast than is AWS S3. In addition to the access key and secret access key, it requires that an endpoint and additional credentials in the form of a certificate. The real AWS S3 has a valid certificate, supported by the certificate authorities, but Snowball Edge has a self-signed certificate.
The configuration can be created with this:
cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds).WithEndpoint(s3Config.S3Endpoint).WithHTTPClient(httpClient)
svc = s3.New(session.New(), cfg)
The endpoint looks like this (you see Snowball Edge responding on port 8443 in the question text above):
https://192.168.1.4:8443
The certificate is required to be formatted like you'd have in a file (including all the new line characters. It looks like this (again, the string you pass in has to include the new line characters after every line):
-----BEGIN CERTIFICATE-----
MIIC7zCCAdegAwIBAgIJBJZB/gkBP0B5MA0GCSqGSIb3DQEBCwUAMBYxFDASBgNV
BAMMCzE3Mi4yMC4xLjE3NB4XDTE3MDQyODIwMTMwOFoXDTE5MDQxODIwMTMwOFow
FjEUMBIGA1UEAwwLMTcyLjIwLjEuMTcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCrLWlTfrSj9R1of5Z98EHYIEEPBgnWxnlrvA+ryAzPmiXbYomI4Tpl
PsuIA+7hGXG10H0zwlz0n22EUv4pE79toYcd3czOJUAHEuSelhtP7u91vM4GguFx
A00gosu04RFUD+BYNeaLTQfd7vdmQB3bY3KEbn7Dfrs/1MYFhKb8J77mgCuUbAPu
PNvwLoV+hBL+ndgs+bIu4MtXjUJDiigRZkacpMQaduDMqEq6seoc+JwrKNBjRBRu
3l/fcQoWf+g902oZJaXXnVGqqb7o2YAQFehUAbmCJfuKFSl5tu0B+3KvQQni7lK+
SV8WItdrPumS98BBlt6NpzgC5fTwCmapAgMBAAGjQDA+MAwGA1UdEwQFMAMBAf8w
HQYDVR0OBBYEFMAvKzKgKI+izqPX6DJjJz/0fELtMA8GA1UdEQQIMAaHBKwUAREw
DQYJKoZIhvcNAQELBQADggEBAGwyzmI+9psQu9/N/oClN7Lej7e4E8cC8vymVfPz
fdW45IMNVEYHxHbu9+JzLOtLxfuDmD6B6fEYVoPubb6tsnacuwOSMZhTvhhy9nv2
f+2Pslgj/kYTeMePbHOPTyQ4sd1BE7ALdNiL/hd08ZNhqObagixNYw9eYeHEStBy
tOADKcY9gOxek1k9t+96nATgSy0WIytwra0uEgyipKQ2gXKpgg15SI4nDxQLLEgG
lb3FtRk+PfJxQ4zbHZe/cRNflcGwVCefycLQOA2Sdr8pgHW7gvETu9i9ywF0UV6f
b9wsPcDmg3EaxBa+wrLlYSzaPhI+rZYh6bpnTn311QIFZ+s=
-----END CERTIFICATE-----
Found the Snowball Edge to be pretty fast storage using S3. We were able to achieve about 1,100 MBytes/second of copy rate.