go-dockerclient `UploadToContainer` tar examples - go

Unlike the python docker client which I've been using there seems to be very few examples for how to use go-dockerclient. I'm trying to work out how to upload a tar archive to a docker container using UploadToContainer which is documented here: https://godoc.org/github.com/fsouza/go-dockerclient#Client.UploadToContainer
I've built a .tar archive in memory which looks like this:
import (
"archive/tar"
"bytes"
"fmt"
"log"
)
func main() {
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new tar archive.
tw := tar.NewWriter(buf)
// Add some files to the archive.
var files = []struct {
Name, Body string
}{
{"file1.txt", "This archive contains some text files."},
{"file2.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
{"file3.txt", "Get animal handling license."},
}
for _, file := range files {
hdr := &tar.Header{
Name: file.Name,
Mode: 0600,
Size: int64(len(file.Body)),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
if _, err := tw.Write([]byte(file.Body)); err != nil {
log.Fatalln(err)
}
}
// Make sure to check the error on Close.
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
fmt.Print(buf)
I've also successfully created a docker-client like this:
import (
"fmt"
"github.com/fsouza/go-dockerclient"
"os"
)
func main() {
endpoint := "https://localhost:52376"
path := os.Getenv("DOCKER_CERT_PATH")
ca := fmt.Sprintf("%s/ca.pem", path)
cert := fmt.Sprintf("%s/cert.pem", path)
key := fmt.Sprintf("%s/key.pem", path)
client, _ := docker.NewTLSClient(endpoint, cert, key, ca)
I have a running container called nginx-ssl and I need to upload the tar archive to the container but can't work out whats needed to write the tar to my container.
This what I do using python where data is an in memory tar object thats been pre-created.
cli.put_archive(container='nginx-ssl', path="/keys", data=newtar)
UPDATE
++++++
I just tried this:
input := bytes.NewBufferString("test")
uploadOpts := docker.UploadToContainerOptions{
InputStream: input,
Path: "/test-test",
}
err := client.UploadToContainer("20e0c347cd41", uploadOpts)
if err != nil {
print(err)
}
20e0c347cd41 is the "CONTAINER ID"
which returns this error:
(0xd60088,0xc820278100)
Then I tried this:
//input := bytes.NewBufferString(buf)
uploadOpts := docker.UploadToContainerOptions{
InputStream: buf,
Path: "/test-test.txt",
}
err := client.UploadToContainer("nginx-ssl", uploadOpts)
if err != nil {
print(err)
}
where InputStream: buf is the output from my tar archive shown above. This throws the following error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x2df8]
goroutine 1 [running]:
panic(0x3d3420, 0xc82000a0d0)
but nothing gets uploaded.
¶

In case you still need it, that is how I have managed to upload a file (named test.txt) onto the container(in the /data dir) using go-dockerclient.
content := "File content"
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
hdr := &tar.Header{
Name: "test.txt",
Mode: 0644,
Size: int64(len(content)),
}
err := tw.WriteHeader(hdr)
if err != nil {
fmt.Println(err.Error())
}
_, err = tw.Write([]byte(content))
if err != nil {
fmt.Println(err.Error())
}
in := sT{bytes.NewBufferString(string(buf.Bytes()))}
opts := docker.UploadToContainerOptions{Path: "data", InputStream: in}
cont, err := client.CreateContainer(options)
if err != nil {
log.Printf(err.Error())
return
}
err = client.UploadToContainer(container.ID, opts)
and sT is defined as:
type sT struct {
*bytes.Buffer
}
You might still improve the code (like use a better aproach for the buffer), but for my tests this is ok.
The result is /data/test.txt containing the text "File content"
HTH

Related

Golang: Facing error while creating .tar.gz file having large name

I am trying to create a .tar.gz file from folder that contains multiple files / folders. Once the .tar.gz file gets created, while extracting, the files are not not properly extracted. Mostly I think its because of large names or path exceeding some n characters, because same thing works when the filename/path is small. I referred this https://github.com/golang/go/issues/17630 and tried to add below code but it did not help.
header.Uid = 0
header.Gid = 0
I am using simple code seen below to create .tar.gz. The approach is, I create a temp folder, do some processing on the files and from that temp path, I create the .tar.gz file hence in the path below I am using pre-defined temp folder path.
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
fp "path/filepath"
)
func main() {
// Create output file
out, err := os.Create("output.tar.gz")
if err != nil {
log.Fatalln("Error writing archive:", err)
}
defer out.Close()
// Create the archive and write the output to the "out" Writer
tmpDir := "C:/Users/USERNAME~1/AppData/Local/Temp/temp-241232063"
err = createArchive1(tmpDir, out)
if err != nil {
log.Fatalln("Error creating archive:", err)
}
fmt.Println("Archive created successfully")
}
func createArchive1(path string, targetFile *os.File) error {
gw := gzip.NewWriter(targetFile)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// walk through every file in the folder
err := fp.Walk(path, func(filePath string, info os.FileInfo, err error) error {
// ensure the src actually exists before trying to tar it
if _, err := os.Stat(filePath); err != nil {
return err
}
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
// generate tar header
header, err := tar.FileInfoHeader(info, info.Name())
header.Uid = 0
header.Gid = 0
if err != nil {
return err
}
header.Name = filePath //strings.TrimPrefix(filePath, fmt.Sprintf("%s/", fp.Dir(path))) //info.Name()
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
if _, err := io.Copy(tw, file); err != nil {
return err
}
return nil
})
return err
}
Please let me know what wrong I am doing.

Editing a zip file in memory

I am trying to edit a zip file in memory in Go and return the zipped file through a HTTP response
The goal is to add a few files to a path in the zip file example
I add a log.txt file in my path/to/file route in the zipped folder
All this should be done without saving the file or editing the original file.
I have implemented a simple version of real-time stream compression, which can correctly compress a single file. If you want it to run efficiently, you need a lot of optimization.
This is only for reference. If you need more information, you should set more useful HTTP header information before compression so that the client can correctly process the response data.
package main
import (
"archive/zip"
"io"
"net/http"
"os"
"github.com/gin-gonic/gin"
)
func main() {
engine := gin.Default()
engine.GET("/log.zip", func(c *gin.Context) {
f, err := os.Open("./log.txt")
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
info, err := f.Stat()
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
z := zip.NewWriter(c.Writer)
head, err := zip.FileInfoHeader(info)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer z.Close()
w, err := z.CreateHeader(head)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
_, err = io.Copy(w, f)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
})
engine.Run("127.0.0.1:8080")
}
So after hours of tireless work i figured out my approach was bad or maybe not possible with the level of my knowledge so here is a not so optimal solution but it works and fill ur file is not large it should be okay for you.
So you have a file template.zip and u want to add extra files, my initial approach was to copy the whole file into memory and edit it from their but i was having complications.
My next approach was to recreate the file in memory, file by file and to do that i need to know every file in the directory i used the code below to get all my files into a list
root := "template"
err = filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}append(files,path)}
now i have all my files and i can create a buffer to hold all this files
buf := new(bytes.Buffer)
// Create a new zip archive.
zipWriter := zip.NewWriter(buf)
now with the zip archive i can write all my old files to it while at the same time copying the contents
for _, file := range files {
zipFile, err := zipWriter.Create(file)
if err != nil {
fmt.Println(err)
}
content, err := ioutil.ReadFile(file)
if err != nil {
log.Fatal(err)
}
// Convert []byte to string and print to screen
// text := string(content)
_, err = zipFile.Write(content)
if err != nil {
fmt.Println(err)
}
}
At this point, we have our file in buf.bytes()
The remaining cold adds the new files and sends the response back to the client
for _, appCode := range appPageCodeText {
f, err := zipWriter.Create(filepath.fileextension)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(appCode.Content))
}
err = zipWriter.Close()
if err != nil {
fmt.Println(err)
}
w.Header().Set("Content-Disposition", "attachment; filename="+"template.zip")
w.Header().Set("Content-Type", "application/zip")
w.Write(buf.Bytes()) //'Copy' the file to the client

Build Docker Image From Go Code

I'm trying to build a Docker image using the Docker API and Docker Go libraries (https://github.com/docker/engine-api/). Code example:
package main
import (
"fmt"
"github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"golang.org/x/net/context"
)
func main() {
defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"}
cli, err := client.NewClient("unix:///var/run/docker.sock", "v1.22", nil, defaultHeaders)
if err != nil {
panic(err)
}
fmt.Print(cli.ClientVersion())
opt := types.ImageBuildOptions{
CPUSetCPUs: "2",
CPUSetMems: "12",
CPUShares: 20,
CPUQuota: 10,
CPUPeriod: 30,
Memory: 256,
MemorySwap: 512,
ShmSize: 10,
CgroupParent: "cgroup_parent",
Dockerfile: "dockerSrc/docker-debug-container/Dockerfile",
}
_, err = cli.ImageBuild(context.Background(), nil, opt)
if err == nil || err.Error() != "Error response from daemon: Server error" {
fmt.Printf("expected a Server Error, got %v", err)
}
}
The error is always same:
Error response from daemon: Cannot locate specified Dockerfile: dockerSrc/docker-debug-container/Dockerfile
or
Error response from daemon: Cannot locate specified Dockerfile: Dockerfile
Things I've checked:
The folder exists in build path
I tried both relative and absolute path
There are no softlinks in the path
I tried the same folder for binary and Dockerfile
docker build <path> works
and bunch of other stuff
My other option was to use RemoteContext which looks like it works, but only for fully self contained dockerfiles, and not the ones with "local file presence".
Update:
Tried passing tar as buffer, but got the same result with the following:
dockerBuildContext, err := os.Open("<path to>/docker-debug- container/docker-debug-container.tar")
defer dockerBuildContext.Close()
opt := types.ImageBuildOptions{
Context: dockerBuildContext,
CPUSetCPUs: "2",
CPUSetMems: "12",
CPUShares: 20,
CPUQuota: 10,
CPUPeriod: 30,
Memory: 256,
MemorySwap: 512,
ShmSize: 10,
CgroupParent: "cgroup_parent",
// Dockerfile: "Dockerfile",
}
_, err = cli.ImageBuild(context.Background(), nil, opt)
The following works for me;
package main
import (
"archive/tar"
"bytes"
"context"
"io"
"io/ioutil"
"log"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
defer tw.Close()
dockerFile := "myDockerfile"
dockerFileReader, err := os.Open("/path/to/dockerfile")
if err != nil {
log.Fatal(err, " :unable to open Dockerfile")
}
readDockerFile, err := ioutil.ReadAll(dockerFileReader)
if err != nil {
log.Fatal(err, " :unable to read dockerfile")
}
tarHeader := &tar.Header{
Name: dockerFile,
Size: int64(len(readDockerFile)),
}
err = tw.WriteHeader(tarHeader)
if err != nil {
log.Fatal(err, " :unable to write tar header")
}
_, err = tw.Write(readDockerFile)
if err != nil {
log.Fatal(err, " :unable to write tar body")
}
dockerFileTarReader := bytes.NewReader(buf.Bytes())
imageBuildResponse, err := cli.ImageBuild(
ctx,
dockerFileTarReader,
types.ImageBuildOptions{
Context: dockerFileTarReader,
Dockerfile: dockerFile,
Remove: true})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
#Mangirdas: staring at a screen long enough DOES help - at least in my case. I have been stuck with the same issue for some time now.
You were right to use the tar file (your second example). If you look at the API doc here https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/build-image-from-a-dockerfile you can see that it expects a tar.
What really helped me was looking at other implementations of the client, perl and ruby in my case. Both create a tar on the fly when being asked to build an image from a directory.
Anyway, you only need to put your dockerBuildContext somewhere else (see the cli.ImageBuild())
dockerBuildContext, err := os.Open("/Path/to/your/docker/tarfile.tar")
defer dockerBuildContext.Close()
buildOptions := types.ImageBuildOptions{
Dockerfile: "Dockerfile", // optional, is the default
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, buildOptions)
if err != nil {
log.Fatal(err)
}
defer buildResponse.Body.Close()
I am not there with naming the images properly yet, but at least I can create them...
Hope this helps.
Cheers
The Docker package has a function for creating a TAR from a file path. It's whats used by the CLI. It's not in the client package so it need to be installed separately:
import (
"github.com/mitchellh/go-homedir"
"github.com/docker/docker/pkg/archive"
)
func GetContext(filePath string) io.Reader {
// Use homedir.Expand to resolve paths like '~/repos/myrepo'
filePath, _ := homedir.Expand(filePath)
ctx, _ := archive.TarWithOptions(filePath, &archive.TarOptions{})
return ctx
}
cli.ImageBuild(context.Background(), GetContext("~/repos/myrepo"), types.ImageBuildOptions{...})
I agree with Marcus Havranek's answer, that method has worked for me. Just want to add how to add a name to an image, since that seemed like an open question:
buildOptions := types.ImageBuildOptions{
Tags: []string{"imagename"},
}
Hope this helps!
Combining a few of the answers, and adding how to correctly parse the returned JSON using DisplayJSONMessagesToStream.
package main
import (
"os"
"log"
"github.com/docker/docker/api/types"
"github.com/docker/docker/pkg/archive"
"github.com/docker/docker/pkg/jsonmessage"
"github.com/docker/docker/pkg/term"
"golang.org/x/net/context"
)
// Build a dockerfile if it exists
func Build(dockerFilePath, buildContextPath string, tags []string) {
ctx := context.Background()
cli := getCLI()
buildOpts := types.ImageBuildOptions{
Dockerfile: dockerFilePath,
Tags: tags,
}
buildCtx, _ := archive.TarWithOptions(buildContextPath, &archive.TarOptions{})
resp, err := cli.ImageBuild(ctx, buildCtx, buildOpts)
if err != nil {
log.Fatalf("build error - %s", err)
}
defer resp.Body.Close()
termFd, isTerm := term.GetFdInfo(os.Stderr)
jsonmessage.DisplayJSONMessagesStream(resp.Body, os.Stderr, termFd, isTerm, nil)
}
I've left our a few convenience functions like getCLI but I'm sure you have your own equivalents.
I encounter same problem. Finally find out the tar file should be docker build context even with Dockerfile.
Here is my code,
package main
import (
"log"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"golang.org/x/net/context"
)
func main() {
dockerBuildContext, err := os.Open("/Users/elsvent/workspace/Go/src/test/test.tar")
defer dockerBuildContext.Close()
buildOptions := types.ImageBuildOptions{
SuppressOutput: true,
PullParent: true,
Tags: []string{"xxx"},
Dockerfile: "test/Dockerfile",
}
defaultHeaders := map[string]string{"Content-Type": "application/tar"}
cli, _ := client.NewClient("unix:///var/run/docker.sock", "v1.24", nil, defaultHeaders)
buildResp, err := cli.ImageBuild(context.Background(), dockerBuildContext, buildOptions)
if err != nil {
log.Fatal(err)
}
defer buildResp.Body.Close()
}
opt := types.ImageBuildOptions{
Dockerfile: "Dockerfile",
}
filePath, _ = homedir.Expand(".")
buildCtx, _ := archive.TarWithOptions(filePath, &archive.TarOptions{})
x, err := cli.ImageBuild(context.Background(), buildCtx, opt)
io.Copy(os.Stdout, x.Body)

How to create a compressed tar archives using compress/gzip and archive/tar?

I'm attempting to create a compressed tar archive using the Go standard library, specifically compress/gzip and archive/tar. I can successfully create a tar archive, but when I try to compress said archive, the resulting tarball can't be decompressed. On OSX, I get "Error 1 - Operation Not Permitted"
To run this code, you'll need a file named foo.txt in the same directory.
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"io/ioutil"
"log"
"os"
)
func main() {
var b bytes.Buffer
// Create a new zip archive.
w := tar.NewWriter(gzip.NewWriter(&b))
fi, err := os.Stat("foo.txt")
if err != nil {
log.Fatal(err)
}
header, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatal(err)
}
err = w.WriteHeader(header)
if err != nil {
log.Fatal(err)
}
contents, err := ioutil.ReadFile("foo.txt")
if err != nil {
log.Fatal(err)
}
_, err = w.Write(contents)
if err != nil {
log.Fatal(err)
}
err = w.Close()
// Make sure to check the error on Close.
err = ioutil.WriteFile("foo.tar.gz", b.Bytes(), 0666)
if err != nil {
log.Fatal(err)
}
}
You need to close the underlying gzip writer so that it you are guaranteed all bytes are flushed out to the file. Like so:
// gzip writer
gz := gzip.NewWriter(f)
// Create a new tar archive.
w := tar.NewWriter(gz)
// add things to the tar archive
// ...
// make sure the gzip writer flushes any pending bytes
if err = gz.Close(); err != nil {
log.Fatal(err)
}

Go file downloader

I have the following code which is suppose to download file by splitting it into multiple parts. But right now it only works on images, when I try downloading other files like tar files the output is an invalid file.
UPDATED:
Used os.WriteAt instead of os.Write and removed os.O_APPEND file mode.
package main
import (
"errors"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"strconv"
)
var file_url string
var workers int
var filename string
func init() {
flag.StringVar(&file_url, "url", "", "URL of the file to download")
flag.StringVar(&filename, "filename", "", "Name of downloaded file")
flag.IntVar(&workers, "workers", 2, "Number of download workers")
}
func get_headers(url string) (map[string]string, error) {
headers := make(map[string]string)
resp, err := http.Head(url)
if err != nil {
return headers, err
}
if resp.StatusCode != 200 {
return headers, errors.New(resp.Status)
}
for key, val := range resp.Header {
headers[key] = val[0]
}
return headers, err
}
func download_chunk(url string, out string, start int, stop int) {
client := new(http.Client)
req, _ := http.NewRequest("GET", url, nil)
req.Header.Add("Range", fmt.Sprintf("bytes=%d-%d", start, stop))
resp, _ := client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
return
}
file, err := os.OpenFile(out, os.O_WRONLY, 0600)
if err != nil {
if file, err = os.Create(out); err != nil {
log.Fatalln(err)
return
}
}
defer file.Close()
if _, err := file.WriteAt(body, int64(start)); err != nil {
log.Fatalln(err)
return
}
fmt.Println(fmt.Sprintf("Range %d-%d: %d", start, stop, resp.ContentLength))
}
func main() {
flag.Parse()
headers, err := get_headers(file_url)
if err != nil {
fmt.Println(err)
} else {
length, _ := strconv.Atoi(headers["Content-Length"])
bytes_chunk := length / workers
fmt.Println("file length: ", length)
for i := 0; i < workers; i++ {
start := i * bytes_chunk
stop := start + (bytes_chunk - 1)
go download_chunk(file_url, filename, start, stop)
}
var input string
fmt.Scanln(&input)
}
}
Basically, it just reads the length of the file, divides it with the number of workers then each file downloads using HTTP's Range header, after downloading it seeks to a position in the file where that chunk is written.
If you really ignore many errors like seen above then your code is not supposed to work reliably for any file type.
However, I guess I can see on problem in your code. I think that mixing O_APPEND and seek is probably a mistake (Seek should be ignored with this mode). I suggest to use (*os.File).WriteAt instead.
IIRC, O_APPEND forces any write to happen at the [current] end of file. However, your download_chunk function instances for file parts can be executing in unpredictable order, thus "reordering" the file parts. The result is then a corrupted file.
1.the sequence of the go routine is not sure。
eg. the execute result maybe as follows:
...
file length:20902
Range 10451-20901:10451
Range 0-10450:10451
...
so the chunks can't just append.
2.when write chunk datas must have a sys.Mutex
(my english is poor,please forget it)

Resources